06 Mar 2020
After a very busy December and first half of January, Bintracker development is now in full swing again for the time being. Unfortunately there are some bad news. Due to some unforeseen circumstances, I will have much less time for Bintracker in the coming months than I had originally hoped. In other words: I’m afraid I won’t be able to keep my promise of releasing the first beta by the end of Spring. I will try my best though to at least release the source code by then, so you all you hackers out there have something to play with over the summer.
The pattern/sequence display rework mentioned in the previous newsletter is complete. The new implementation is based on a single Tk Text widget, rather than a hack to combine several Treeview widgets. This reduces the complexity a fair bit. More importantly, it also results in a significant speed boost, so the display is much more responsive now. It’s also more compact, making better use of vertical and especially horizontal space. Check it out:
I also made good progress on rewriting scm2wiki, my in-source documentation tool. Before the next public release I still need to add back the svnwiki backend (not used for Bintracker itself, but a couple of other projects of mine). As far as Bintracker and libmdal are concerned, scm2wiki is now capable of creating proper documentation for the entire API. Also, the code is now nicely modular, which means I can potentially use it to generate documentation that can be used in Bintracker itself, so Bintracker would become self-documenting.
I’ve moved on to the MDAL compiler again. The most important news here is that the old, custom MDMOD format is no more. MDAL modules are now symbolic expressions, which brings them in line with everything else in Bintracker. For those of you who want to know all the nitty gritty details, check out the latest MDMOD Specification Draft. Gotta love the new MDMOD parser: It’s a whopping 125 lines of code, including a major upgrade to the error handling.
While I’m at it, I thought it might be a good idea to change the internal representation of MDAL modules in libmdal, too. Yak shaving commences… I think it was very much worth spending a few days on this, though. In fact I just got the compiler generator to work again with the new design, just need to fix up a few loose ends here and there now. The original implementation had some heavy overhead in places, due to using custom datatypes to represent the input nodes of a module. The new design uses only alists (aka associative lists, a common and simple way to implement lookup tables in Scheme). Block fields nodes (which are used to represent channels in a pattern, for example) are abstracted away completely, so instances of Block nodes now contain just the raw data, organized in rows. I’m not entirely sure yet if the latter was a smart decision, but I’m rolling with it for now. In any case, all those changes brought libmdal’s code size down by a few hundred lines, so I think we’re on the right track.
Furthermore I had a great idea that will be helpful in tackling various problems regarding sequences. The idea is to simply make Block nodes endless by default, and control the actual length through an explicit length parameter in the sequence. So what exactly does this solve? Say we have a set of patterns (per-channel, since MDAL uses matrix sequences as the lowest common denominator to represent all sequence types found in existing sound engines), and we want to use them at a given step in the sequence. What happens if the patterns aren’t the same length? Do we pad shorter patterns up to the length of the longest one, or do we shorten everything to the shortest one? If we opt for the former, how do we even represent this in the GUI? What does it mean to “insert a row” in such a setup? Endless Block nodes basically allow us to bypass all these questions. No block length mismatches can exist, and adding a row to a block does not change the length of any step in the sequence. And the best part is: The new internal module representation makes this very easy to implement, and in fact I have already implemented the basics of it.
Once I’ve finished the next iteration of the MDAL compiler, I plan to finally tackle the emulation part, ie. connecting Bintracker to MAME. I’ve got a reasonably worked out plan on how to do things on the MAME side, but I still don’t have a clear picture how to implement things on the Bintracker side. Collecting some ideas at the moment. In terms of concurrency, CHICKEN offers a standard SRFI-18 implementation. The alternative would be gochan, which, as the name suggests, is inspired by Go Channels.
Implementing emulation is the major obstacle on the way to the first release of Bintracker. However, there are a number of smaller items on the to-do list as well. These include:
Several people got in touch with questions and suggestions after the last newsletter. Very happy about that, please keep them coming. Special thanks to adkd and nyanpasu64 (who’s working on a next-gen NES tracker) for the great discussions.
I prefer a compact user interace, like Protracker 2.3d, with a full set of keyboard shortcuts. Will this be possible with Bintracker?
Yes, this is in fact a major design goal. There will be keyboard shortcuts for everything, and they will be customizable, too. Regarding compact UI, you can disable toolbars and the menu (and of course the console) to increase vertical space. Font size can also be changed. Also, the re-worked main module view is more compact horizontally, and will have configurable line spacing as well. Furthermore, I want to make channels “collapsible”. So in collapsed mode, each channel will only show the primary column (ie, note column or trigger column for drum channels).
I would like a “search box”, for things like finding uses of a certain instrument in the song, or searching for a certain element by name.
That’s a great idea. It would be a great showcase for a Bintracker plugin. I’m actually liking the idea to the point where I’m considering implementing it for the first beta release.
Are you intending to write a custom audio renderer for Bintracker?
No, at least not for now. For the moment, the plan is to just let MAME do the audio. In the future Bintracker will support multiple emulation backends. For each backend, there must exist an abstraction layer which exposes at least a minimum set of features (eg. run/stop/load), and may optionally support more advanced features like low latency playback or MIDI IO.
I do have an idea for adding a sort of native emulator at some point. It would be in the form of an abstract, customizable virtual machine running low-level bytecode. MDAL configurations could then include a bytecode translation of the sound engine which can run in the VM. The main use of this would be to provide support for machines for which no viable emulator exists, such as mini-computers and mainframes. Either way, this certainly won’t happen any time soon.
What’s the plan regarding Undo?
MDAL uses a tree to represent module data. (I considered using a dictionary instead, but it would be too impractical considering MDMOD allows for infinite recursion.) All changes to the tree pass through a single dispatch procedure. The dispatch procedure invokes one of 4 edit procedures, depending on the argument list (set, insert, remove, or compound-edit). It returns an argument list. Calling the dispatch function with that argument list reverses the initial edit. The returned argument lists are managed by two stacks for undo/redo. In other words, the dispatch function is handed a description of the change, mutates the document, and returns an undo description.
Will Bintracker support macros, and if so, what’s your plan regarding security?
Yes, MDMOD will support macros, in the sense that any data can be substituted with Scheme code. Regarding security, Bintracker is basically a Scheme interpreter, which is exposed to the user at runtime. In this sense it is insecure by design. However, this is a conscious decision: My idea is for Bintracker to be this hacky tool that you can use for pretty much anything. So you should be able to use Bintracker like a tracker, but also like an IDE or a live programming environment, for example. That said, I’m of course considering mitigations especially in regards to plugins. The general idea right now is to have an “official” central repository which only admits plugins that have been vetted, and display a warning when users attempt to install a plugin from an untrusted source. For mdmod files, I may actually sandbox the interpreter, though this poses some challenges with regards to file access (externally stored sample data etc).
I’ve had multiple discussions on the subject of sequences that I touched upon in the previous newsletter. I’ll do a separate write-up on that in one of the upcoming newsletters. In the meantime, if you have any more ideas on that, I’d be happy to hear them, of course.
Another subject I’d like to get your input is “folding”, as in “code folding”. For those of you who aren’t coders, code folding is a feature in some programming editors that allows you to collapse parts of the code that aren’t currently relevant to you. For example, say you have a loop, then folding the loop would hide anything but the first line of the loop code from view. So programmers know that there is a loop there, but don’t need to scroll through the details of it unless they actually want to work on that. I believe that this concept could be useful in a tracker as well. However, I haven’t gotten any further than this vague idea in my head. So if anyone has some thoughts on that, let me know!
Pretty much all the points from the last newsletter still apply. Some of you good folks out there are already spreading the word on social media, so thanks for that!
Still looking for someone to help with custom icons, and with MacOS builds and/or CI things in general. I also could do with some advice on how to securely implement an update function that pulls the latest plugin version from the planned central repository.
Well, I guess that’s all for this time. I’m planning on sending out newsletters a bit more frequently from now on, but with a twist. Instead of giving a full update each time (which will still happen of course), I’ll mix things up with write-ups on one specific topic at a time. The plan is to eventually make these write-ups part of the Bintracker documentation as well.
As always, if you have any questions, suggestions, ideas, or you want to help with stuff, please get in touch.