I was working on video rendering on Android devices – reading through and mashing together libraries and example code I found. The task as it’s currently structured has 6 main classes interacting:
An extractor to present encoded frames of video from a file
A decoder to decode each frame
An animator to keep this happening regularly and check if it’s time to show a new frame
A view which is the interface element the video will be shown in, and which I’m using as a convenient repository for all the other action (this seems to be pretty standard approach)
A renderer which talks to OpenGL, the rendering API
I’ve been bodging everything together in one prototyping/sketchpad project. My approach was to keep it just-about-functional for my one test case (displaying three overlaid alpha-blended video elements) as I swapped in and out components and ported bits from Java to Kotlin.
I definitely learnt a lot from that, but today I tried a different tack.
Instead of accomplishing a fairly complete task messily and crudely, I took the single component I wanted to understand and placed it in a blank, fresh project. Then I tested it!
I got the class, TimeAnimator, to report all salient variables to GUI elements. Like debug text but organised so I can see the situation at a glance.
I realised I couldn’t make it do what I thought it had been doing, which was fire events at a steady framerate equal to the video framerate (of 30fps).
I shuffled through variants of the class and learnt what they do.
After a bit, I realised none of them did what I wanted. I went back to the library I was studying and finally twigged that the controlling framerate/timing information came from the decoded video data, which was merely being checked (at a frequency much faster than the framerate) by the TimeAnimator, which is not meant to run anywhere near as slow as 30fps.
So far, so trivial. But I might have kept playing with the original code for ages without actually challenging my assumptions, distracted by all its moving parts and the fact that it looked somewhat like the final product I wanted.
I will definitely be reusing this technique of isolating the smallest component that I don’t understand, and “prickin’ it and frickin’ it” as my favourite rapper Guru might say.
Hope to be back here soon to discuss what this video functionality is actually for!
I coded up a prototype recently in a new realm: video compositing in real time. It was enjoyable for a few reasons, one of which was the language I used.
Processing has been around for a while. It’s designed to give instant access to graphics programming. It does that by exposing one-line functions that draw to a window without pretty much no configuration needed, by locking in a folder structure (e.g. images you want to work with go in a “data” subfolder), and tying everything together in a simple dev environment.
It’s ostensibly aimed at visual artists and students, and it fits in a lineage of what we could call “beginner languages” that intentionally hide complexity. Because I’ve now had a variety of encounters with such languages, including QBasic in my teens, a previous failed attempt to learn Processing, and a bit of a tool around with Racket earlier this year, I thought I’d try pin down some thoughts on them.
Let’s maybe take a wide perspective on my whole encounter, from the first thought of “Maybe I could use Processing?”
I DuckDuckGo-ed it, and a nice .org site popped up. Wow! First impressions were really good. The whole vibe of the community and ecosystem is cheerful and welcoming.
There are pleasing non-jargon phrases all over the front page: “Creative Applications”, “Sketchpad”, “Visual arts”, “books available” – and yet it also feels like extensive technical work has been done, with links to implementations in multiple languages, and the obligatory Github profile, and many sponsors and partners.
I’m gonna go through this experience both from my current point of view, and an imagined one of a total beginner programmer. Both of these would give the Processing site, and related ones like Happy Coding, a hearty
The default installation format is a portable one. That might be a hassle to some. Or maybe it’s actually good. Unzip it and go, no permissions or shortcuts or whatevs.
The IDE loads showing a crisp splash screen with generative vector patterns like the main site. It’s perfectly decent, except for one thing which the docs make a lot of fuss about. Each code file is called a “sketch” – and the folder where these are kept is insistently referred to as “the sketchbook”. This caused me about ten minutes of trying to figure out if the sketchbook was a file format, and figuring out that it was automatically placed in a different location than where I’d unzipped the IDE. Calling it “sketches” would have made that clearer.
But for our absolute beginner, it’s not so bad: the IDE opens with a blank file, and you can copy in some example code and click the run button, and it works straight away. That’s REALLY nice.
This is what I really want to explore today. The trade-offs of simplification.
I think there’s an interesting conflict, in that simplification can make the first steps easier – which again is super valuable – but the next steps harder.
Both times I tried out Processing, I had this sinking feeling of near-betrayal as the light fun of the early steps sank into a morass of what feels like second-guessing the developers.
And I had it years back with the natural-language interactive fiction system, Inform 7, which I gave up on.
Making first steps easier is an illusion – one gets the feeling of accomplishment without really understanding what’s going on.
That can be all right. Computers hide inner workings from us all the time. The issue is finding the right balance of illusion so that the first peek behind the curtain doesn’t shatter fragile confidence.
This tendency of mine clashes with the carefully-laid illusions of beginner languages. It took me say twenty minutes and half a screen of code to display one layer of a candle flame and show it, chuffed, to my dad. Implementing my actual goal of compositing a few layers with filtering and interactive controls, took a whole day, much of which was spent in frustration as the seemingly generous affordances of the language turned to cobwebs.
Here were some things that tripped me up after my first joyful success:
drawing off-screen required trawling the none-too-voluminous developer docs to figure out the difference between a PImage and a PGraphics object -I’m guessing working on an off-screen a buffer wasn’t considered a fundamental use case for people coming from a visual arts background
a Movie object acts like an Image, but its width and height fields are set at 0 until the first call to its read() method (making some initialisation stuff harder)
some graphics functions are overloaded so that they can take either the red green blue components of the desired colour or a single component as a greyscale value – but not consistently
drawing to an off-screen buffer has to take place inside the single set draw() method, otherwise it randomly will draw to the main window in flickers
And there was more, but this is boring. Once I figured them out, it was okay. Even the rather arcane pipeline I ended up with – a Movie’s read() setting a flag that an update is required, which is checked in draw() which if necessary calls loadPixels(), updatePixels(), beginDraw() and endDraw() on the various buffers, and the actual draw calls and filter calls, in the correct order – is more or less understandable. (I didn’t say optimal!)
It’s just that… there’s a let-down from the initial hey presto! to what feels like debugging a somewhat complex system you don’t understand. Now there’s nothing wrong with that process. That’s how to learn. But could the journey from delight to disappointment be mitigated?
It was that feeling of having waded in a bit beyond my depth and being swept away by uncontrollable factors, that turned me off Processing and Inform 7 before.
However… I don’t think Processing could be much better. The only ways to smooth my experience would be tiny tweaks: if someone rewrote a tutorial, tweaked a line in an example, and so on. I see why people talk of community as a central aspect of high-quality software.
My conclusion is that despite the inevitability of harsh contact with system complexity, beginner languages are great – but don’t start one, join an existing one.
Community building is also behind those good vibes that are maybe even more important than the technicalities, when welcoming interested newbies.
Oh, and yeah, what did I actually code? Glad you asked. Taking some tips from an awesome visual effects tutorial by an enthusiastic fellow called Lendon Bracewell, I made some candle flame effects for a votive shrine app I’m planning. The idea is for this to become a fast prototyping/design tool when I’m assembling my app’s candle animations and effects. Check it.