Proposal: Enable unit testing Kaleidoscopes core and client plugins by Dependency Inversion

[noseglasses]

@jesse

I can’t recall if we previously talked through the various possible ‘compromises’ for build systems.

No, definitely not talked through. We only discussed here and there about regression testing and CMake.

I only mention that you are concentrating on Arduino to make it clear to potential users of my CMake build system that they should not bother you with any questions/issues/etc. about CMake.

@jesse

The CMake implementation you’ve built out is, to the best of my understanding, based on a CMake-native reimplementation from scratch of the Arduino build process.

Correct.

@jesse

It is, however, not ideal as a ‘second’ build system, as maintaining bug-for-bug compatibility between platforms is difficult burden.

Agreed. It took me a while and many contributions to Arduino-CMake to get everything to work nicely. And, as you say, this solution is far from ideal.

The reason why I did not wrap Arduino-Builder is that I wanted several features that the Arduino build system is lacking. The most prominent are

1. Partial rebuilds.
2. Generate preprocessed input files (important when dealing with complex nested prepro-macros).

Apart from that, I need a build system that allows me to have Kaleidoscope-components (plugins, modules, whatever we call them - Arduino libraries) with additional sophisticated build steps. An application example is a plugin that I am working on:

The plugin does complex key event pattern recognition (chords, clusters, multi-key tap-dances, sequences, …). For this to work it uses an external library, which is another project I am working on and already use with QMK. The C-interface of this library is far too complex to bother the average Kaleidoscope user with and the search tree that is used for the pattern recognition is better allocated statically.

That’s why I implemented an optimizing compiler, that maps a very simple descriptive language to a complex amount of C code that is build automatically and linked with the Kaleidoscope-Plugin. The user only writes some lines of descriptive language in a commented region of the sketch’s C++ code, e voila she/he gains very powerful functionality.

For this to work, several steps are automatized as part of the firmware build

1. The external C-library must be set up as an Arduino library.
2. The pre-compiler is build.
3. The sketch is parsed and compiled by the pre-compiler and converted to a C file.
4. The firmware is compiled and linked, including the sketch, other plugins, the Kaleidoscope-Papageno plugin, the Papageno C library and the search tree C-code generated as part of the build.

As you can see, this is quite a complex task. I am well aware that 99% of the Kaleidoscope-plugins are less complicated but there might be other devs that have crazy ideas that also require features that the Arduino build system cannot provide.

A CMake based build system makes all this possible and allows to provide a very simple build recipe in the plugin’s github README. Users only have to make sure that CMake and a native C++ compiler are installed, apart from Arduino. The rest is just the ordinary two stage build process of any other C++ FOSS project. I expect this to be feasible, at least for some advanced Kaleidoscope users.

@jesse

Would you be amenable to a CMake implementation that, instead of reimplementing the Arduino build system and our build tools, wraps them? That’s something that I’d be thrilled to have in core alongside the traditional build infrastructure.

Yes, definitely, because I understand your concerns towards a all-CMake build system very well.

What about a build system that is CMake based but modular. The overall build handling and module/plugin pre-build stage is CMake-driven. The actual firmware build part could be made modular.
The user could choose between a Arduino based build core and a purely CMake based one, one that provides all those features that Arduino is (currently) lacking.

The core build modules could be two independent git repos. One that is a very simple CMake wrapper of Arduino-CMake or Kaleidoscope-CMake and one that is a restructured version of what I am using so far.

I could create something like that but currently I am busy with some other projects that I need to finish first. At least one needs to be finished before I can use my M01.

@jesse

As to automated testing, it’s something we very much want to do. I -am- very picky about making sure the test infrastructure is simple and straightforward to use. The current ‘does it build’ smoke tests, coupled with the C++ linter are a start, but are a poor substitute for any sort of real automated regression testing.

@jesse

Getting the usability of a test system right is pretty important. Otherwise, it can hurt development more than it can help.

I agree. The problem is that there are different types of tests with completely different levels of complexity.

While it is very simple to test the keyboard output that results from a series of keystrokes, it is far more complicated once the exact order of HID reports or the internal state of the firmware matter and are to be tested. For the former, craigdissels work seems a perfect match. For the latter, something more complex is necessary…and available.

To ensure that my own complex plugins (e.g. the one that I mentioned above) does and will do what it is supposed to, I came up with Leidokos-Python. It wraps most parts of the Kaleidoscope core module as Python code and allows any other plugins to export their internals as well. This way it is possible to monitor and test the IO behavior but also to keep track of what happens inside the firmware - if necessary. There are already quite sophisticated plugins around where it might be more efficient to test parts of the internal state instead of defining an infinite number of simple IO tests.

Leidokos-Python’s tests are mostly based on formulating expectations. E.g. somthing in the line of “the next HID report must contain certain keys or modifiers or must not… or whatever combinations” or “there must be n HID reports” or “no HID report after the next key event”. First you define a series of expectations for an expected series of HID reports, then you feed the virtual keyboard with key events. Tests fails if expectations are not met.

All this works with simulated real time. Even complex timing issues can be tracked down. As a bonus (not necessary for regression testing), the LED state can be visualized at any time and the whole beast can run in any ordinary debugger on the host system. All this makes Leidokos-Python a valuable tool when developing sophisticated plugins.

To make the handling of tests as convenient as possible, I started another project, Leidokos-Testing that reads a testing directory tree structure and automatically defines a number of regressions tests. A test is defined by its Kaleidoscope plugins (git sha/branch/…), the firmware sketch and the python driver for Leikodos-Python. This information consists of files (the sketch and the python driver) and a yaml-file that contains the information about which Kaleidoscope-plugins to build a firmware of. I chose yaml to make the testing tree structure definition independent from CMake and to be able to replace the CMake parts of Leidokos-Testing if necessary.

Information stored in subdirectories overrides the information of parent directories. This allows to specify a large set of tests with a minimum of spec redundancies. Every leaf of the testing directory tree represents an individual regression-test.

For my purposes this proved to be useful. I already use it for simple tests of Leidokos-Python and am planning to use it for all my plugins’ regression testing.

@jesse

(I’ve also been working 12+ hour days getting hardware shipped and supporting customers for months, which hasn’t given me a lot of time to dig into this.

I read your backer updates and followed your announcements on twitter. I continuously wondered when you actually sleep? You must be either supernatural or a zombie. Probably a supernatural zombie that builds nice keyboards