01 Apr 2022
Again, a long time has passed since the last update. My apologies for keeping you waiting so long. Things got way to busy over here at some point while I was processing last year’s harvest, finishing my language classes, making a new album with my band Doppelplusungut, and working on various other side projects. To be honest I also got a bit stuck on that rewrite of the MDAL compiler I mentioned last time.
That said, here are the good news - for the past two months, I’ve been making steady progress again. I’ve had plenty of time to think about how to implement the new compiler, and in the end I’ve come up with a design that I believe to be sound and solid. And what’s even better - I’ve actually finished implementing it!
Before I delve a bit more into the details of the new compiler though, I’d like to take the opportunity to thank all the kind folks who’ve donated in the past months. Thank you all for continuing to believe in the project despite the lack of (public) activity.
Now, what’s the deal with this new compiler? Was it really worth sinking two months of coding work into this when we already had a working compiler before? The truth is that the old compiler had a major design flaw that would have made it impossible to achieve Bintracker’s goal of supporting any arbitrary sound engine, while also making it difficult to implement an efficient optimizer later on. Aside from that, as I mentioned last time, the old design had become quite a mess as I was essentially making things up on the fly.
The aforementioned design flaw has to do with the way references are resolved. The old compiler was tracking the output origin (ie. the address in memory where things get compiled to) during the whole compilation process, resolving references as early as possible. While this approach allows for a fast compilation, it makes it impossible to modify the generated data later on. Say we have an engine that uses samples, and the pattern data references those samples by their address. Now we want to eliminate unused samples from the output. In order to determine which samples are unused, we need to first evaluate the pattern data. Once the pattern data is evaluated, the sample references are now resolved to addresses. However, eliminating samples will change the addresses of the samples that come after the ones that have been eliminated. So now the references in the pattern data must be changed - but this is impossible, since they are already resolved, so they are now just numbers, indistinguishable from the rest of the data.
I guess this would have been obvious to any software developer who’s ever studied compiler design. Unfortunately, not being a “real” software developer myself, it wasn’t obvious to me. So, that was a rather costly mistake that was nevertheless bound to happen, as I was predominantly concerned with making the compiler as fast as possible.
The new MDAL compiler does not track the output origin at all, and does not resolve any references. Instead, the compiler generator first builds an assembler-level abstract syntax tree when reading a given MDAL engine definition. This AST is passed to the assembler, which will already translate as much as at it can to machine code. The result of this operation is cached. The compiler generator then eliminates all output nodes that require no further processing at the MDAL level, and reorders the remaining ones to minimize the number of compilation passes required. On compiling the actual module, the compiler stores the result of evaluating each node in a global symbol table. Bindings in this table can be updated throughout the compilation process, so in the process of evaluating one node, the result of another node can be modified. The final step of the compilation process, which includes reference resolution, is then offloaded to the one component of Bintracker that is good at resolving references - the assembler.
Regarding speed, the new compiler looks to be generally more efficient than the old one - unless there are a lot of references to be resolved. Currently, wiz4v is the one engine that suffers from this, now having a noticible lag when entering notes. I believe there is plenty of room for optimizing reference resolution though.
Overall, I’m very happy with the new design, and also much relieved that I finally managed to eliminate this massive roadblock. Not only did the rewrite solve some major problems, but the code has also become more readable in the process (though it still needs work in that respect). In total, the rewrite added just about 200 lines of code, while packing significantly more functionality than before.
While there are still a number of glaring bugs and missing features (still no Label commands, for instance), I think it’s time to tackle Windows builds next. There’s actually been some progress in this department - at least it’s possible to build Bintracker with MinGW now. Unfortunately that doesn’t mean that the builds are actually working, yet. On Wine, Bintracker starts up, but then hangs on opening a module. On actual Windows, it either crashes on startup or fails to start at all so far. Not sure how long this is going to take, so I’m not going to make any promises in terms of a release date just yet, especially considering this project’s history of missed deadlines.
As I’ve mentioned before, I don’t have much experience building stuff for Windows. Someone more knowledgeable could probably figure out things much quicker than me. In short, I could really use some help with this. If you’re up for some good old head-against-wall-banging and pulling-hair-in-frustration then please get in touch via the contact form.