BINTRACKER
19 Jun 2020
Hello Friends of Bintracker,
Here’s another update before taking a break from Bintracker development. Things have progressed nicely, managed to get quite a few things done in the past couple of weeks. Among the highlights this time are support for two new platforms, a vastly improved assembler, and several new sound engines.
The most important news is that the major rewrite of Bintracker’s assembler is done. There’s still much room left for optimizations, but considering the lag on note input is overall quite tolerable, I will tackle this at a later point. The code is now much cleaner, no more ugly global state. The main change is that assemblies are now closures. This fulfills an important requirement of libmdal: Support conditional player code that depends on parameters in module data, and do so at least somewhat efficiently. The way it works is roughly this: On loading or creating a module, the MDAL compiler attempts to assemble the player with a limited number of assembler passes. If this succeeds, the compiler caches the result and uses that during module compilation. If it fails, then the compiler caches the closure holding the assembly, and then finalizes the assembly once the missing information becomes available during module compilation. One small tweak I’ve also done is change the output format of the assembler, so you can now run such output directly on an emulator without having to awkwardly convert it first.
I’ve added support for one new instruction set (Fairchild F8), and two new target platforms: The Fairchild Channel F console, and the Sharp MZ-700 computer. The former was surprisingly hard to do. I hadn’t anticipated that hot-reloading on cartridge based systems is very different from doing it on a regular home computer core in MAME. Either way, I managed to pull it off in the end, and with that we now (hopefully) have a workable blue print for other consoles. Adding MZ-700 on the other hand was a bit disappointing. Apparently sound support for this machine in MAME is not very accurate, so its usefulness remains limited for now.
A number new engines have been added: SleizsaTrio for the Channel F, Squeeker and PhaserX for the ZX Spectrum 48K, as well as MooZ’ PhaserX port for Sharp MZ-700. As mentioned above, the latter doesn’t sound all that great in MAME though. You can expect the number of engines to grow much faster after the scheduled development break: As libmdal and the assembler get more complete, it becomes easier and easier to add more of them.
Much to my frustration, I hadn’t been able to include a working plugin in the initial source release. I found the obnoxious bug that was preventing my test plugin from working a couple of days afterwards, so we now have our first plugin! It’s nothing too impressive, just a simple Music Macro Language parser, so you can now write melodies in MML.
Speaking of bugs, I’ve been fixing those left and right. At this point I’m nearly done working through the big stack of notes on known bugs that I was holding up in the last video. What’s left are mostly UI-related bugs, which are mostly nuisances rather than critical crash-causing errors. There are two exceptions here, which I will touch on in the next section.
Furthermore I’ve completed the big MDAL rename I had planned for a while. So, no more “MDAL Configurations”, they’re now called “MDAL Engine Definitions” or short “MDEF”, which I believe makes their purpose much clearer. The libmdal API is now also significantly more consistent, though there’s still a fair bit of work left to do in that respect.
Last but not least, I’ve done a number of small improvements to the UI, as well as improving discoverability. For example, moving around in the module now displays more detailed information of what’s under cursor, and the “New Module” dialog now displays information on the selected engine.
Normally this where I explain what I’ll do next. However, this time I’ll give a more general overview of what still needs to be done, a sort of general road map, if you will.
As mentioned, there are a couple of known bugs left, and taking care of those is the top item on my to-do list (though I’m not of course not always in the mood of fixing bugs, implementing new features is much more fun!). One major bug that’s still alive is related to Undo/Redo: Registering block edit actions (copy/paste etc.) in the edit journal is not fully implemented yet, so calling Undo after such an action may break things in spectacular ways. The other bug is related to emulation: Row play will sometimes fail, producing horrible noise instead. This is caused by a certain class of bugs in MAME’s CPU cores. The MAME team is aware of the problem, but since the issue doesn’t really affect “normal” users, I think fixing this has pretty low priority for them (and I don’t blame them, in fact they’ve been pretty helpful as of late). So I’ll need to find a work-around for this. There is also some dodgy behavior going on in popup dialogues. This will probably require some digging into pstk, the Chicken Scheme extension that provides the interface to the Tk GUI toolkit.
Another item I want to get started on soon is building Bintracker on Windows. My aim is to provide a self-contained, statically compiled package that’s as easy as possible to set up. I haven’t used Windows since the XP days, so surely this is going to be… fun. Any help with this is highly welcome!
Furthermore, I’ll be adding support for more CPU architectures and target platforms. Next on my list are the Atari VCS/2600, and Tandy CoCo/Dragon 32/64, which means adding the 65xx and 6809 architectures to the assembler. Once 65xx support is done, adding the Commodore line of home computers will probably be the next step.
Documentation is still sorely lacking. After going on a writing spree before the source code release, I’m pretty burned out in this regard, but I’ll get back to it eventually. Also, I still need to make adjustments to scm2wiki, the tool that generates the API documentation. I actually want to integrate scm2wiki into Bintracker itself, to make it self-documenting. During the last round of work on scm2wiki, I already implemented the necessary changes to make this possible, but nothing is done yet on the Bintracker side.
There are a number of crucial features still missing in both the Bintracker GUI and the MDAL compiler, which amount to a sizable chunk of work. These form the bulk of what needs to be done before the beta release. I estimate that I will need around 8-10 weeks to complete those. MDAL misses three major things at this point: Support for unordered blocks (which will be used to implement things like samples and effects tables), modifier commands (which will be used to implement detune), and the output optimizer. I already have a pretty solid concept for the optimizer. However, I need to do unordered blocks first, and I’m still very much in the “throwing around ideas” phase regarding that. Two smaller things that are also still missing is loop points (still waiting for a eureka moment here as well), and overrides, which will offer a way to tweak the engine definition used by a module. The latter will be the basis for implementing microtonal scales and alternative tunings.
Regarding the GUI, there are a number of things that still need to be done. One of the most heavy duty jobs here is implementing simplified sequences and automatic sequences, though the latter will very likely have to wait until after the beta release. Another big item is folding. See those double-arrow buttons on the left side? The idea is that you can click those to collapse the buffer that they’re attached to. It’s not just a convenience feature: Sound engines with a more complex data structure (think module -> subtune -> sequence -> chain -> patterns -> instruments -> tables) will definitely require this. Aside from these big baddies, there are plenty of smaller things left to do as well, especially in regards to cleaning up the user interface and improving/extending the plugin API. There are also a couple of MDAL command types that aren’t fully supported yet.
Once all these things are done, the core of Bintracker will be mostly complete. Which means I can finally focus on adding more sound engines, and even more exiting: Plugins! Already got a huge list of ideas for plugins, and probably I won’t be able to resist the urge of implementing some of those before finishing the core features mentioned above.
I also want to make a more active effort to build a community around Bintracker. At this point I feel that I’ve implemented the core parts of my vision of what I want Bintracker to be, so it’s time to move from “me” to “we”, so to speak. Needless to say, I’m counting on the support of all you good folks out there, which brings me to…
Since I squashed most of the bugs I knew about, I’m now more than happy to receive user testing and bug reports! There’s definitely plenty of bugs left, I’m just not aware of them.
The main topic on which I would very much appreciate some input and help concerns infrastructure. There are a number of things that come to mind. One thing I would like to have are synchronized mirrors of the main Github repository. I’ll probably set up a mirror on Gitlab myself, but if some of you were to self-host additional mirrors, that would be great.
Also, I’m wondering what you folks would recommend as a bug tracker or project management platform in general. I guess for now we’ll stick to Github Issues, but in the long term I would prefer a solution outside of Github, to avoid becoming vendor-locked. However, I don’t want to self-host this on the Bintracker website. The same goes for a project wiki. Let me know your thoughts on that.
Another issue concerns setting up continuous integration, ideally with automated nightly builds. Obviously this will require figuring out how to build on MacOS and Windows, first. I will probably try to tackle Windows myself soon-ish, but as mentioned previously, I would very much appreciate help with MacOS.
Furthermore, I would like to hear your opinions on communication infrastructure, ie. setting up a chat and/or forum. Do people even use forums in 2020? Do we need one? I mean I like them, but as things stand now forums seem to be in decline everywhere. Regarding chat, I personally favor IRC (yeah I’m old), but I’m open to suggestions. Not a big fan of Discord, but I understand that’s where a lot of people hang out nowadays. So perhaps an IRC channel bridged to Discord would be a good option. Would be great if someone trustworthy would step forward to take care of this, as I don’t have any experience with it myself, and my internet access is often spotty so I can’t run a bot. As an alternative, I’ve been looking into Gitter. I like how it integrates with Github/lab repositories, but again I’m kind of weary of vendor lock-in.
Lastly, please keep spreading the word about Bintracker! The more people get interested in this, the more motivated I will be. Perhaps it’s also time for establishing a presence on Twitter/Mastodon and Facebook. I would very much prefer not having to do that myself, so if anybody out there wants to take on the job, by all means go for it (talk to me first, though!).
Let me just say that overall I’m pretty happy with the recent progress. Compared with the initial source release, stability has improved a lot, and Bintracker is slowly shaping to become a tool that’s actually useful, rather than just an experimental mess. That said, there is obviously still a ton of work left, which I’ll start to tackle in about 2 months time, after taking a break to take care of some big personal life changes. I might be able to get some smaller things done during this, but I can’t promise anything because it’s hard to say how packed my schedule is will be in the coming weeks. So for now, stay tuned, and as always, let me know about your thoughts, questions, and suggestions. Note that I may lose internet access for a couple of weeks in late July/early August, so don’t be mad at me if I don’t reply to your messages immediately.