Revisiting A Classic

I finished another programming project and I think it’s my strongest yet, thanks to me finally getting serious about testing! I called my app HuffmanRevisited because it implements the classic Huffman coding algorithm from 1951, and also because I had previously tried to program this algorithm a few months ago.

This time round, I coded it in Java, not JavaScript. It probably required about a solid week of work. And unlike my earlier attempt, I made a finished app. It can:

  • load a text file, encode (compress) it and save the encoded version, or else
  • load a previously compressed file, decode it and save the original content back as a text file

(You can skim my code here if you like.)

A few aspects of my approach helped make coding this an enjoyable task: I tried to specify it rigorously, I wrote a lot of tests, and I tackled a clear, well-known problem.

Before Christmas, I read a book – well, no, I skimmed a book after reading the prologue (which took multiple attempts) – called How To Design Programs. It’s an MIT textbook, you can read it here. I recommend it.

My paraphrase of the book’s prologue is: “precisely specifying what a program does is the hardest part of making it.”

Of course, I had encountered this sentiment in my Object Oriented Software Engineering module in the National College of Ireland. But the MIT textbook avoids the familiar paraphernelia of design patterns, verbose conventions and profuse diagrams. Instead, the book’s prologue challenges you to specify a problem by answering two questions: how is the data represented (in terms of data types that the machine can understand, and at all steps of the program)? and what are the functions (in terms of operations on those types)?

I mean, when I write it out, it’s dull and obvious. Yet, time and again I’ve found myself skimping on this specification step. Because, yes, it is the hardest part and one’s brain tries to escape it.

Even after my epiphany from reading the MIT book, I still evaded it. I specified most but not all of my Huffman encoding app before I started, thinking that the remainder was “simple”. But the simple part is never simple and if you haven’t thought it through, the solution will be incoherent even if it seems to work.

I failed to specify all of HuffmanRevisited, but at least I knew that this failure was to blame when I witnessed complexity mushrooming in front of my eyes as I tried to solve new, small problems that kept cropping up.

BTW, I’ll mention a couple of those little problems to see if you spot a pattern that kind of pleased me:

  • accessing individual bits from an array of bytes
  • packing an unpredictable amount of bits into an array of bytes
  • turning Java’s int and char types into a byte representation (not just casting which truncates them)
  • saving a compact representation of a binary tree containing characters in its leaf nodes (the ‘Huffman tree’)

Yeah… the pattern I spotted is that I’m doing low-level stuff and pushing somewhat against the limitations of Java. This is nice because I’ve been looking for an excuse to study a more low-level language!

The other thing that went really well with this project was testing. I’d written tests in JUnit before, but to be honest I was doing it to fulfil obligations in school assignments. Just like with specifying rigorously, I knew that tests are a great idea but was lazy about writing them.

I totally changed my tune once I had the framework up and running. (I used JUnit 5, Maven and NetBeans 11, and I mention this combination because I had no joy with JUnit 4 or with Ant.) I realised I’ve always done a lot of testing, but amateurishly: printing variables to the console all the time. That works okay… until your program starts to have a decent amount of methods on the call stack (i.e., functions that call functions that call functions that call functions…) and you spend your time trying to remember in which method did your gnomic text originate. Plus, all those print statements mess up your code.

Not to sound too much like a new convert, but using a test framework was a delight after all that. It’s much more organised. (And with my let’s say “wide-ranging” mind, I need all the organisation I can get!) It’s just like what I was doing before, except you only see debug text if there’s a problem, you get to choose any range of values you like to test, you can test as small or as large a section of the program as you like (as long as you’ve made the program itself reasonably articulated), and all of this business lives in separate files. Oh, and you get a cheerful screen of green when you pass your tests!

It’s enough to warm anyone’s heart

Okay, so specification and testing are non-negotiable aspects of real-world software development, but the last aspect I want to discuss can be more of a luxury: a clearly defined problem.

Until I get a start in a programming job, I can’t be sure, but my impression is that even communicating what a problem is, never mind completing a rigorous specification, can be hard in a typical business context.

However, I did this project for self-study so I got to choose exactly what to work on.

(I was helped in this by a comp sci book called The Turing Omnibus that a mentor very kindly lent me! It has a chapter on Huffman coding. The hook of the book, I would say, is that it conversationally introduces topics but doesn’t take you through every nuance. For example, unlike the Wikipedia article on Huffman coding, it mentions neither the need to pad bytes with zeros, nor any scheme for storing a b-tree.)

I was so glad I chose such a an old chestnut of an algorithm to implement! When I was refactoring my way out of that mushrooming complexity I mentioned earlier, the clarity of my app’s intention was a godsend.

Even better was the lack of edge cases. I could be certain my program had worked when it took a text file, compressed it into a smaller file, and then decompressed that encoded version into the exact same data I started with!

That’s intrinsically neater than some other areas I’ve boldly attempted, for example digital audio or vector graphics, where you need good control of sampling and rounding.

When I do go back to such complex topics, I’ll have a crucial bit of extra experience with the exact weapons needed to contain the ambiguity. Testing and full specification.

So, I’ll sign off there. My app could easily absorb some more effort. The next thing to work on would be efficiency. Do some profiling, and just comb through it for wastages. I can also think of some cool ways to present it, but no point hyping work I may not get around to.

Anyway, I’m pleased with it already. Jaysusin’ thing works, and the code structure is fairly sensible.

Thanks for reading. Take care in these hard times!

The header image “Massachusetts Institute of Technology Parking Lot, Memorial Drive, Viewed from Graduate Housing” by MIT-Libraries is licensed under CC BY-NC 2.0. I chose it because David A. Huffman might have seen such a view as he wrote the term paper in which he invented his compression technique.

7 Days

Last Monday I decided I would work on a different coding-related skill each day, for a week. These days I’m familiar enough with my own brain to know that swapping subject areas and deep-diving into topics suit my attentional style. Because motivation can be hard to come by when jobhunting and self-studying during pandemic restrictions, I thought I’d stimulate myself with novelty. It might even help me get psyched up to work on longer-term ambitions, like releasing my ritual Android app Candle Shrine, and also making a portfolio website.

Day 1 – C

On Monday I worked on C programming – an area I haven’t touched since 1999, ha! Yes as a kid I did a summer course which touched on some C. Anyhow, the aim was to set up the compiler and get console input and output. I used a build of Tiny C Compiler nabbed from the Tiny C Games bundle. My project is a simulation of life as experienced by our family cat, Goldie. Actually I’m pretty pleased with this, my sister played it through and enjoyed it. I was inspired by Robert Yang’s concept of “local level design“- a design aesthetic celebrating small-scale social meanings rather than top-down formalism. (This suited me because I don’t yet know enough C to write anything other than this ladder of if statements. Still, it works!)

Day 2 – Linux

The next mini-project was Linux shell commands. I used VirtualBox to dip my toes in -it lets you run any distro on emulated hardware, from a Windows desktop. It nearly locked up on my a couple of times, but in fairness my computer was running two OSes at once so I forgave it. It never fully crashed.

I’d hoped to get into shell scripting, which is the power user technique of saving listings of shell commands (a shell being a console for directly running programs or OS utilities, etc. – like the ol’ Command Prompt in Windows) as text files to be invoked as, effectively, little programs.

But all I had time for was to learn about 20 standard shell commands. However, I really liked this stuff. I can see why there’s a stereotype that devs use Linux. It’s rather satisfying to install stuff, edit files, and set up the file system via typed commands and not all that intimidating either.

Day 3 – Pink Sparks

See it in action here

This one was fun. I made a 3D particle demo – a spinning cube made of flying pink sparks. My focus here was to prove I could make a simple particle system, which indeed wasn’t hard, cribbing off Jonas Wagner’s Chaotic Particles and leveraging the extremely handy Canvas feature in HTML5. I also wanted to do 3D perspective, which was hard. However here I used the simplest possible version, a bare z-divide where x and y coordinates are divided by distance from the viewer. The proper way to do this involves matrices and transforms, but I’m not there yet.

If I get back to this the two things I’ll do are change it to defining line-shaped sources of sparks rather than point-shaped, and make some nicer data than a cube, like say a chunky letter ‘K’. This wouldn’t be particularly hard. Making a display of my initial fits with my interest in what I call ceremonial coding which I believe will be an emerging cultural field in years to come. As life goes online, we’re already finding the need to program and design software for celebrations and community rituals – an example being my graduation from my computer science course, which is being held on Zoom. I am certain that techniques from game design and aesthetics from digital culture will be important to create spiritual meanings and affirmations of identity on computers. My upcoming ritual app for Android phones expresses this conviction.

Again, Robert Yang’s post is very close to the spirit of this: “What if we made small levels or games as gifts, as tokens, as mementos?”

Day 4 – Huffman Coding

I hit a roadblock on Thursday. Huffman coding compresses text, by taking advantage of the fact that certain symbols (for example, ‘z’) occur far less often than others. To make a Java app implementing this was a meaty challenge, requiring binary buffer manipulations, a binary tree, sorting and file I/O. Still should have been achievable – but I let myself down by not rigorously figuring out the data representations at the start. This meant I threw away work, for example figuring out how to flatten the tree into an array and save that to disk, only to twig that the naive representation I’d used created a file far bigger than the original text file.

Though I was working from a textbook with an understandable description of the Huffman coding technique, that was nowhere near enough. I still needed to design my program and I failed to.

So I ran out of motivation as poor design decisions kept bubbling up. This was a stinger and a reminder that no project is too small to require the pencil-and-paper stage. On the plus side, I did implement a tree with saving and loading to disk, plus text analysis.

Hopefully I can reuse these if I come back to this. It’s a fun challenge, particularly the raw binary stuff and the tree flattening (although I don’t know yet if I want to store the tree in my compressed file, I think probably just the symbol table needed to restore the original.)

Day 5 – WebGL Black Triangle

In the spirit of Jay Barnson’s Black Triangles – though I’m sure it’s a million times easier these days!

Now this was pure fun. I used a tutorial to learn how to display graphics on a webpage using WebGL. I… frankly love the feel of OpenGL Shader Language. The idea of using a harshly constrained programming language to express some low-level color or geometry calculations, which is then compiled and run on your graphics card so you can feed it astronomical amounts of data for ultra-fast processing, is so satisfying. (Actually, especially the compilation process, and the narkiness of the parser where for example 1 is different to 1.0… it feels like you’ve loaded a weapon when you’ve successfully compiled at last.) I love graphical magic, but previously have been doing it at several removes, using wrappers on wrappers like Processing. I will definitely be doing more of this.

Day 6 – RESTful API

Another failure! I wanted to make a RESTful API demo as a bullet point on my CV, and host it for free on Heroku. But although I did some good revision on API design, when I got into implementation I totally got tangled up in trying to get libraries to work. Blehh!

I wanted my system to be standards-compliant so I tried using libraries that’d let me use JSON:API instead of raw JSON, which some people say is not good for APIs as it has no standard way to include hyperlinks which are, in fairness, central to the concept of REST (i.e. that instead of the client knowing to use certain hardcoded URLs, each response from the API includes fresh hyperlinks that the client can choose to follow).

But I got stuck when the examples for the library I chose wouldn’t compile because they required a new version of a build tool, Gradle, and despite trying some things off forums my IDE failed repeatedly to automatically install this.

They don’t call it “dependency heaven“!

If I get back to this I’ll use the build tooling I had working already for school projects. Life’s too short!

I wonder if the area of web services – so essential, stolid, bland – might be a natural home for rather pedantic personalities. The type who would make a typology of all things and publish it as a web standard. In any case, wading through some comments and blog posts and Wikipedia pages gave me a stronger understanding of state in web services than I had before.

Day 7 – Pathfinding in JS

Like the previous one, this project spun off into piles of research. But it’s all valuable stuff, I revised a lot of CSS and particularly the grid system, and got deep into JavaScript using this quite good book.

My plan was to implement a classic pathfinding algorithm, Dijkstra’s algorithm. But I wanted to have it so a little monster would chase your cursor around elements on a webpage! Well, as usual, I should’ve thought this through more. The fact is, web page elements are not intended to be processed as 2D shapes. HTML is semantic – web content is structured and manipulated as elements like paragraphs, headings, links that have meaningful relationships in the context of the document itself – with the final presentation of these elements done on the fly according to the user’s needs.

Anyway… my point is, I had to compromise to get this going. My original vision was of words arrayed around the page randomly, at different sizes, with a monster sprite threading his way around them.

My solution doesn’t have the words, just boring blocks, and though I think I could do words at a fixed size, having splashy words in different size could be quite a hassle.

As you can see, the paths go through diagonal choke points, something to fix

Nor did I get around to the monster although I have the sprite for when I do:

Heheheheheh.

But the thing that worked well for this mini-project was: web standards! In particular, I made the excellent decision not to hack CSS positioning from JS, but instead take the time to revise the CSS Grid system. Which, as you might imagine from the name, was actually perfect for this use case. Those numbered cells above are arranged by Grid.

Conclusion

That was fun. I might even do it again!

The Tiniest Example

Here’s a learning technique I just discovered.

I was working on video rendering on Android devices – reading through and mashing together libraries and example code I found. The task as it’s currently structured has 6 main classes interacting:

  • An extractor to present encoded frames of video from a file
  • A decoder to decode each frame
  • An animator to keep this happening regularly and check if it’s time to show a new frame
  • A view which is the interface element the video will be shown in, and which I’m using as a convenient repository for all the other action (this seems to be pretty standard approach)
  • A renderer which talks to OpenGL, the rendering API

I’ve been bodging everything together in one prototyping/sketchpad project. My approach was to keep it just-about-functional for my one test case (displaying three overlaid alpha-blended video elements) as I swapped in and out components and ported bits from Java to Kotlin.

I definitely learnt a lot from that, but today I tried a different tack.

Instead of accomplishing a fairly complete task messily and crudely, I took the single component I wanted to understand and placed it in a blank, fresh project. Then I tested it!

A boring screenshot

I got the class, TimeAnimator, to report all salient variables to GUI elements. Like debug text but organised so I can see the situation at a glance.

I realised I couldn’t make it do what I thought it had been doing, which was fire events at a steady framerate equal to the video framerate (of 30fps).

I shuffled through variants of the class and learnt what they do.

After a bit, I realised none of them did what I wanted. I went back to the library I was studying and finally twigged that the controlling framerate/timing information came from the decoded video data, which was merely being checked (at a frequency much faster than the framerate) by the TimeAnimator, which is not meant to run anywhere near as slow as 30fps.

So far, so trivial. But I might have kept playing with the original code for ages without actually challenging my assumptions, distracted by all its moving parts and the fact that it looked somewhat like the final product I wanted.

I will definitely be reusing this technique of isolating the smallest component that I don’t understand, and “prickin’ it and frickin’ it” as my favourite rapper Guru might say.

Hope to be back here soon to discuss what this video functionality is actually for!

Synth Sins

Warm analogue it ain’t. I knew when I started coding my synth-sequencer, Foldy, a few months ago, that it’d be harshly digital and crude sounding. I was inspired by tracker software as well as by two old PC-only music programs, Drumsynth and Hammerhead (which were the basis of my beat-creating project last year).

I’m releasing it today and calling it version 1.0. It works, but some iffy design decisions mean I won’t keep developing it.

That said, the code quality is a step up from my last release, the experimental art tool MoiréTest. I was able to go back and make big changes in Foldy, without the whole thing crumbling, which is always a good sign.

For the rest of this post I’ll explain what the program does, then what questionable decisions I made and how I would do it again.

(To try it yourself, download Foldy.jar from here and double click on it. If that doesn’t work try the further instructions in the readme.)

Foldy takes in a musical sequence, which you can type into a box in the app window. Notes are numbered as MIDI notes, where A=440 is at 69, and notes range from 0 to 128, and separated by commas. A rest is -1.

(By the way, did you know that, incredibly annoyingly, there is no industry standard for numbering the octaves of MIDI notes? The frequencies are agreed on, but one manufacturer’s C3 is another’s C4… how sad. This doesn’t impact Foldy though, I just work from the frequencies.)

The speed that notes are played is altered using tempo and beat subdivision controls. All the other parameters in the window modify the sound of individual notes. Only one note can play at a time. This kept things a bit simpler though, with the Java Sound API, opening another output line or mixing two together wouldn’t be much harder.

I was going to include a choice of mathematical curves, possibly Bezier curves, for the amplitude envelope, out of a perverse desire to avoid the bog-standard Attack-Decay-Sustain-Release model, which is suited to a keyboard instrument where a note is attacked, held and released. I was thinking this synth could be more percussive, inspired by the basic sample-playback model of drum machines and trackers (a type of sampler software originally made for Amiga computers and associated with the demoscene).

Unfortunately I didn’t finish the Bezier stuff, but in any case it probably wasn’t suitable. (For one thing, Bezier curves can easily have two y values for one x value.) In fact, I didn’t do any extra envelope options, partly because envelopes typically drive filters or modulations, but these are not allowed by my architecture. If there’s an obvious v1.1 feature, extra envelope curves is it.

One feature that did make it in is “wave-folding”. To get more complex waveforms, I cut a sine wave at a certain amplitude, and invert anything above that amplitude. This can be done multiple times to add a lot of harmonics.

Adding harmonics to a sine wave by folding it at 1/2, then 1/4 amplitude

However, this is a restrictive technique with a distinctive grinding, mechanical sound. All we’re doing here is shaping a waveform which is then repeated exactly at the period of the note frequency. The ear instantly picks up the lack of complexity.

I remember when I was a teenager, having the following bright idea: if I can see that the recorded waveform from my bass consists of repeated bumps, can’t I just sample one of those and repeat it/change the speed of it to get any bass note I want?

Why, chap, it’s simply a bunch of bumps (by the way, don’t record bass in stereo like I did here)

This is the basic concept of wavetable synthesis. However, when done as simply as that, it sounds completely artificial, not at all like a bass guitar. The sound of any real instrument has complexities like propagating resonances, changes in pitch, string rattle and other distortions/energy loss.

(E.g. listen to the low note in this sampled bassline – it’s starts really sharp, then reverts to normal. That’s because plucking of a stringed instrument raises the pitch of the note momentarily, especially on an open string – I think this was an open E string on the original sampled recording, just it’s been pitched up here.)

Foldy has no capability for such modulations. I could try put them in, but here we come up against the compromises I made at the start.

Because I was afraid that rounding errors would mount up and give me grief, I decided to keep everything as whole numbers, taking advantage of the fact that digital audio ultimately is whole numbers: a series of amplitudes or “samples” each expressed as, for example a 16bit or “short” integer. (Most studios mix at 24bit these days, but say CD audio only goes up to 16bit precision.)

This informed the basis of the synth. Desired frequencies and tempos are approximated by a wavelength and a subdivision length expressed in whole samples. 44100 samples per second might seem fairly precise, but for musical pitches, it isn’t. So I found a compromise that bounded pitch error to about 20 cents:

Foldy tries to fit multiple wave cycles within a whole number of samples, for example 3 cycles in 401 samples. This gives a bit more precision, because the wavelength is 401/3 = 133.667 samples, in between the 133 and 134 wavelengths that are all I could get otherwise.

I then use these bits of audio, which I call “chunks”, and which could contain a single cycle or a handful of cycles, in the same way I was using single wave cycles originally. So every note would contain hundreds of them. Then I decided I could reuse this division to store amplitude envelopes – I gave each chunk a starting amplitude, and interpolated between these. (Of course, this is redundant at the moment because my overall envelopes are merely a linear interpolation from maximum to zero! But with a curved envelope, the result would be to store the curve within a few dozen or hundred points, with straight lines from point to point.)

Ugh… I don’t even want to write about it anymore. It wasn’t well conceived and caused me a lot of hassle. It precluded any of the more intriguing synthesis techniques I like, such as frequency modulation, because pitch in this system is fixed for each note (and imprecise).

Long story short, when I opened up the source code of Drumsynth recently, I realised that… it just uses floats and gets along fine. For modulation, it simply keeps track of phase as another float. I should’ve done that.

(That said, I think Drumsynth’s sound quality is far from pristine. This isn’t from rounding errors, I’m certain, but from not doing more complex stuff like supersampling. But, that’s out of my ability level right now anyway.)

Using floats, I still would have had trouble with the timing for the sequencer, probably… but that would have led me to the realisation that I was biting off too much!

It’s not a complete loss. I really enjoyed trying to calculate sine waves while sticking to integer arithmetic . I found out about Bhaskara‘s approximation, implemented it, and then found some really nice code using bitshifts to do a Taylor Series approximation of a sine wave. (I wish I had the chops to come up with it myself!)

Reading the source of Drumsynth also completely changed my approach to the GUI code. I originally had all of the classes that make up the synth – Note, Chunk, Sequence and so on – also be GUI elements by inheriting Java Swing component classes. I think I picked this up from some book or tutorial, but it’s obviously not good. It breaks the basic principle of decoupling.

Drumsynth blew my mind with its simplicity. There are no classes as it’s written in C, an imperative language. The synthesis is just one long function! I almost didn’t know you could do that, having spent a year studying Java and OOP. But given that the app is non-realtime (meaning that there is a third of a second pause to calculate the sound before you can hear it)… this is the sensible approach. Logically, it is one long straight task that we’re doing.

So I ripped out the GUI code from my main classes, and stuck it into one class called Control. Drumsynth’s GUI is even more decoupled: it’s written in a different language – a Visual Basic form that calls DLLs to access the synth functions!

(Yes, I know this is pretty out-of-date inspiration – heck Drumsynth even cheekily uses INI files for configuration though they were officially deprecated – but I think the lesson on directness and decoupling stands.)

My overall lessons from this project are:

  • Do normal stuff rather than trying to reinvent things.
  • Find exactly what draws you to a project and make that the focus. E.g. with this I would’ve been better off making something smaller and more conventional but which allowed me to try some unusual FM stuff.
  • Even though I’ve so, so much further to go, I kinda like low-level stuff. I mean, okay, nothing in Java is actually low-level, but still I was dealing with buffers, overflows, even endianness! Those are fun errors to fix.
  • Read other people’s code!

Even more generally, there’s a kind of tricky question here. This project showed me that it’d be a huge amount of work to approach the quality level of some of the audio programming frameworks out there such as JSFX, VST/Synthmaker, or JUCE. If I’m interested in actually programming synths for musical purposes, I should use one of those.

On the other hand, these are all coded in C or C++ (maybe with another abstraction layer such as EEL scripting language in the case of JSFX). If I really want to understand fundamentals, I should learn C.

But, it’s not very likely I’ll get a job doing high performance programming of that sort, considering the competition from those with as much enthusiasm as me for flash graphics or cool audio, but much more chops! I’m at peace with that – I quit music to get out of a profession that is flooded with enthusiasts.

Stuff to mull over.

Side Project Satisfaction

Today I finished up a coding project, which was to make a simple synthesiser written in Java. I noted though, as I dashed off a quick readme file and uploaded the repository for the last time, a distinct sense of anticlimax, even disappointment. The app didn’t work out that well. Instead of writing an account of what I was trying to make and how I set about it, then, this evening I want to reflect on ways to avoid that letdown, specifically with side projects where you have total creative freedom.

This is about accepting and optimising for the way my brain works. Which is: it creates lots and lots of ideas at the start of a creative process. Here are some of the ideas I had for my synth:

  • adding harmonics by wave-folding
  • only using fixed-point arithmetic (inspired by early trackers and my general interest in retrocomputing)
  • using fast sine approximations
  • using sine lookup tables
  • stretched harmonics
  • harmonics detuned to simulate string mass
  • using Bezier curves as volume envelopes
  • doing additive synthesis by mixing together multiple instances of my basic instrument
  • doing additive synthesis by creating custom waveforms containing the desired harmonics
  • generating just intonated and 5-limit tunings
  • generating tunings from two overlapped harmonic series
  • generating 12-tone equal temperament and quarter-tone (24-tone) tunings mathematically
  • using the stuttering/echoing sound of buffer underruns as a musical effect
  • adding frequency modulation synthesis
  • having the synth be about a limited subset of sounds like bells, marimbas and bass
  • avoiding certain assumptions from the MIDI standard, e.g. removing note duration in favour of focusing on note onsets only (an idea influenced by my study of African-derived musics which have this emphasis)
  • using the synth in some kind of persistent, low-key online game as a way for players to leave melodies for each other to find
  • using rules to generate different melodies from the same basic info (say an array of integers), for example with different tunings or in different modes and scales
  • generating scales, chords and perhaps melodic cells or fragments geometrically, using relations I know from my study of music theory back in the day (e.g. a chord is every odd member of a scale is an approximation to 7 stacked fifths in a chromatic space generated using the twelfth root of two)
  • making an interface to expose these options interactively

And so on. The obvious common factor among all of these, I would say, is that there is no common factor. They are an extremely heterogenous bunch suggesting a whole lot of varied perspectives. Some are from a coder’s perspective (fixed-point calculation), some are about synthesising technique, some reflect my own limitations of ability, some are more like observations about the nature of music. Many have a distinctly contrarian flavour.

You can probably see the problem. There is no way to ever succeed at a project “defined by” such a list. And I’m not talking about success defined externally, but even just personal satisfaction. There are far too many requirements, many of which are contradictory.

The ideas there which are more rooted and achievable have another issue: they are technical challenges only, which makes them arbitrary. There’s no way to know if the solution arrived at is good, because there’s no agreed-upon way to measure performance. Should my bezier curve envelope allow two values for a given x input (which an unconstrained quadratic bezier can easily have), or forbid such curves? There’s no right answer because I haven’t defined what I’m trying to do.

This is in stark contrast to the school projects I’ve done for my higher diploma course: making a banking app, or an e-commerce website. Even if the work could get boring with those, the remit is clear and it’s satisfying when such a system comes together.

How did I manage to put a few weeks into my synth without realising that I hadn’t fixed upon a goal?

Or do I even need a goal? There’s nothing wrong with just fiddling about with stuff for fun, there’s even a fancy phrase to make it sound more official: “stochastic tinkering“. However, I know that I get my fun as a programmer in quite a specific way – by turning stuff that doesn’t work into stuff that does. When definitions are too loose, there’s no way to decide whether something works.

I’ve come up with a few pointers on how I might avoid this looseness in future.

The first is to design something conventional with a fictional, “normal” user in mind. This was how my school projects got done. This is good because convention (shopping carts in an e-commerce site, account selection in a banking site) guides you. This leverages the part of my mind that can’t stand to be wrong, and that likes tidying up: as long as the project fails to meet conventional expectations, I’ll be nettled into improving it.

However, finding the motivation to develop without the ego boost of originality would be hard for me. I know from the experience of finishing up my synth that work done just for the sake of appearing competent to strangers who come across my Github profile, isn’t very sustaining. The school projects had the virtue of being compulsory.

The second solution is to design something I’d like to use. This is… hard actually. It requires some self-awareness and honesty. I made a synth because I thought it’d be cool… or so I thought. Yet if I truly believed synths are cool, I’d probably have used one in the last few months; I haven’t done any synth-based music-making though in that time (despite having dozens of software synths installed on my computer). My conclusion is that I find synths to be a pleasantly intricate subject for mental distraction, but that I don’t actually have much desire to use them.

And similarly with the pixel art app I made before the synth. I like thinking about pixel patterns and generating them, yet if I liked making pixel art I’d be making some.

So, thinking honestly about one’s interests and requirements isn’t all that easy.

A third approach is to make something new, but very small. This worked well with something I made last year, an interactive sine wave visualiser. I actually made use of it, just for a second, while working on my synth to help me think about differentiating sines.

I’ve read advice to programmers about making tools that do one thing very well, and I can see the sense of it.

A fourth thing that has worked well for me is collaborating. When I’m working closely with others, my desire to appear right is a strong motivator. The hard part though for a side project is putting the energy into finding collaborators and the contradictory twin fears of not being good enough versus working with someone I feel is holding me back.

Those are actually familiar negative thoughts from my musician days.

Well, that’s what I wanted to write. Conclusion: even though my brain likes nothing better than lashing out idea after idea, finding the right one takes courage and deliberation. And it seems likely that good project ideas will combine a couple of the following: doing one thing only; being conventional; solving a real problem I have; being collaborations.

Postmortem for my First App

Three and a half months ago I started a software development course, to build a new career as a programmer. I’ve also quit playing music. As we come to the start of a new year I’m very happy with these choices and with the prospects ahead.

One of the things I’ve loved for a long time about programming was the culture of sharing knowledge. It directly influenced the tone of this blog. So, while I don’t know whether I’ll continue writing about programming on Drum Chant or make a new blog for it, today I want to try my hand here anyway at a software postmortem.

Following the time-honoured format (which I first saw on gamasutra.org I think, and also used on this blog before), I’ll introduce my project, discuss what went right in the production, what went wrong, and what lessons can be learned about software development.

Introduction

I developed my first app in Java this month. It’s a number-guessing game simulating a lottery draw. I call it Gambler’s Delight. I had previously worked on a similar brief for a group project in school which I enjoyed enormously. So I decided to remake that idea on my own, using a more elegant, manageable design.

The game affords multiple rounds of attempts to win the lottery. In each round the user fills out a lottery ticket of 1-3 lines, where each line is a guess at the 6 random lottery numbers which will be drawn. When the user decides to play their ticket, the application draws the 6 random numbers and tells the player how good their guesses were, awarding fictional cash prizes depending on performance. Finally, the app keeps a record of all the rounds played and displays this when the user requests to end the play session.

So, it’s rather like the Irish National Lottery‘s digital play platform. I didn’t refer to this when designing my game, but my effort converges on the same concepts.

How the pros did it

You may be already seeing some of the design issues I faced:

  • how to make sure the player chooses six unique numbers
  • how to have automatically chosen (“quick pick”) numbers
  • given that some lines are active and ready to play, some not, how to deal with the transitions from inactive to active and back

My own learning goals for the project were to get my head around Java’s Swing library, which provides platform-independent graphical user interface (GUI) widgets, as well as some intermediate programming concepts like interfaces, events, exceptions, and of course the object-oriented programming principles of encapsulation, inheritance and abstraction. (The fourth technique always mentioned with that group, polymorphism, didn’t really come into my app.)

Or, to put it crudely, I wanted to make a simple game but:

  • split it out intelligibly into classes representing real-world objects,
  • use the premade Swing classes more or less as they’re meant to be used, and
  • keep a handle on the structure and proliferation of my code.

What Went Right

A clean GUI: I designed the look of my program in one blast of inspiration, which luckily was then realisable in one of Swing’s layout manager classes, the euphoniously named GridBagLayout.

I decided that each line of 6 numbers would be built as 6 text fields, similar to a licence key or credit card number entry form. And I was able to have all necessary user actions available on this one, small window, in a layout based on the number 3.

The concept ((also demonstrating the importance of having appropriate stationery on hand 🙂 )
My GUI in action

Encapsulation: as you can see in the source code, I split the functionality of my game into 9 classes and and an interface. Many of these are satisfactorily neat and conceptually self-contained. Even the bigger ones use some basic principles to hide their data, i.e. private variables. Also, I followed the correct convention for passing arrays – copying their contents into a new array before passing that – to avoid giving the requesting class access to a private array in cases where this matters, e.g. the LottoTicket’s getResults() method.

Keeping control of my code: I was pleased that by the end of this project, I still knew clearly what everything did and where to look for any particular functionality. I could skim my code and know what I was looking at thanks to nice variable and method names, whitespace and my reasonably clear class structure. I’m an ultra verbal thinker so writing lines like “history.updateWith(ticket.getResults());” that do what they sound like they do in approximately readable English, pleases me no end.

Use of Java classes: the built-in classes I used – probably around 12 or so – were rewarding to study and build around. Of course they are, they’re made by top people! The experience of using things like Swing’s InputVerifier or Container classes, say, is of initial simplicity giving way to great depth and flexibility. The big but logical inheritance hierarchies of these components are inspiring. I’d love to make something someday as worked-out and usable as these. The only downside of my plug-and-play approach is that I ended up using a good few classes quite superficially. I can see how later it’d be all about digging down deeper and overriding parts of fundamental classes to put my desired custom behaviours in more unified containers that match their purpose more elegantly.

Use of events: I used a good few of Java’s event types and wrote one of my own, so now I have at least some insight into event-driven programming.

Bullet-proof UX and validation: I’m proud of this achievement even though conceptually, behind the scenes, it could’ve been cleaner. My interface instantaneously reacts to invalid input with an appropriate error message and an updated count of how many lines are ready to be played. The only exception is, I allow the user to leave fields blank for smoother navigation. It’s impossible to leave an unplayable input in a field, and impossible to play a line that doesn’t validate correctly (e.g. has blanks). There are two subtleties to this. First, I had to use multiple kinds of events to make sure and cover all interactions: DocumentEvents, FocusEvents, ActionEvents… and then pass the appropriate EventListener classes back and forth when building all the objects (lines, fields, verifiers) in my game. That’s overly complex for sure. On the plus side, I had to take care of weird player behaviour like going back and deleting numbers from previously finished lines. That involves some slightly tedious code but the result is robust.

Minimising hardcoding: I kept crucial gameplay variables in a separate LottoRules class and made sure that they only need to be changed once to correctly change the game’s behaviour. However, I chickened out of following this to its proper conclusion of allowing customisable line lengths and number of lines per ticket – the game logic can handle this, but the user interface wouldn’t dynamically adjust.

What Went Wrong

Having no target audience: the game was inspired by a brief for a college project. It doesn’t fulfill any real user’s needs and therefore there’s no point continuing to develop it. Honing in on excellent design becomes arbitrary in this situation. For my next project I’ll come up with something that I, and hopefully many people, definitely want.

Insufficient use of exceptions and defensive programming tactics: changes in one part of the code could easily cause crashes or undetermined behaviour because I don’t check the validity of values or handle wrong values (apart from the user input discussed above). That said, I think I’m right to move on without polishing this more.

Hardcoding: this point also contravenes good programming practice. The UI I made isn’t adaptable to different line lengths/amounts, and has some per-pixel hardcoding. But here I was coming up against the fact that system-independent GUI design is hard. Java’s Swing layout managers are pretty flexible and wide-ranging, but with that comes a lack of predictability and precision. I think next time I might try a different GUI technology – AWT, or something web-based – just to keep learning new stuff.

Lack of reusability: although I made one class explicitly for future use, (TextNiceties which deals with plurals and counting words) I mostly failed at designing for reuse which is, so my software engineering teacher says, the holy grail. Partly this is due to the trivial but finicky task: much of what I wrote is for the necessities of this particular game. Then there was the spreading of GUI code throughout my classes, which I’ll discuss below, but which obviously makes my code specific to one task.

Refactoring is hard: “restructuring existing computer code—changing the factoring—without changing its external behavior” as Wikipedia defines it, was challenging. I already had a working prototype at the start of this project, from my school group work. Unfortunately I tended to unthinkingly take elements of the previous design into my new one. And I also assumed that problems were already solved without noticing that with changed design assumptions, my previous techniques were now invalid. So, I made two failed attempts at restructuring my GUI code before settling on a pattern of passing a window (JFrame) object into my various classes so they could paint themselves onto it.

Deciding what object should have what responsibilities: I had the concept of an overarching “app” creating a lotto “ticket” containing “lines” which each contained number “fields”. Reasonable enough, but the question of which object should know about which other objects was hard! This applied to the UI creation, the validation, and the game state changes… I have tons to learn here.

Over-engineered interface: I allowed the user to invalidate a completed line of numbers by clearing text fields. This adds nothing, it’s merely a standard text input convention. I think an excellent design would intentionally limit such possibilities in the interests of clarity and simplicity. I note that the Irish National Lottery interface:

  • uses a graphical grid of numbers that are either selected by a single left click, eliminating duplicates automatically
  • doesn’t allow deselection (deletion) of numbers, so lines are simply always valid once completed – and bright colours are used to indicate the change to valid
  • therefore never has e.g. the first line invalid while subsequent ones are playable, an edge case I had to write a fair bit of overly-involved code to deal with

Conclusion

I’ll stop there. That’s a lot of chat about a learning project, but hey I’m proud of it and I sank a few days into it. Nice one if you read through it all and perhaps I’ll be back soon with a cooler project. A friend of mine gave me the idea of doing something that queries a web API, and I also think I want to do something with non-trivial calculations (maybe some geometry/graphics) and file handling.

*EDIT* Oh, one last addition – I think I know now what the solution is for organising the line, field and ticket objects and their drawing code. Sticking closer to the paradigm from John P. Russell’s excellent beginners Java book, I would go back to having each major element subclass a Swing component such as a JPanel, and draw itself in its constructor. And, I would structure the lines like a group of radio buttons: first a LottoLine would be created, and it would be passed into the constructor of all its LottoFields.

Ah well, there’s always more improvements to make. Still happy to draw a line under this project, cause it works and the code is readable.