When working on changes to the firmware core it would be desirable to have a generalized testing procedure that defines what must have been tested before submitting a PR. This is mainly important in those rare cases where a PR comes with changes to core header files.
It is certainly not possible to test all available features manually and only few of us might have all keyboards available that are supported by Kaleidoscope.
But in my opinion it would already be a good start to maintained a firmware sketch that includes all parts of the firmware, i.e. all core plugins and features, to make at least sure that everything compiles. This sketch would definitely not create a reasonable firmware for most devices as its binary would be way too large. But for a compile test that’s of no importance.
It would also be desirable to build this testing firmware for all supported hardware before submitting PRs that comprise changes to header files.
Current practice is that we try to maintain examples for most plugins, so large parts of Kaleidoscope are compile tested. I don’t feel good about ignoring firmware limits, even if only as a special case…
Also, the current build system has a serious limitation that prevents this: you can’t easily compile the same sketch for multiple boards. The new build system (see Kaleidoscope#497 and the WIP PRs linked from there) will make this easier, but I need to have enough cycles to finish it up.
This is about compiling and linking, about syntax and symbols. The firmware won’t actually be used.
What’s this limitation about?
Good to hear that.
I have a PR that is waiting to be submitted. But first I want to run as many compile and link tests as possible to ensure the best quality possible. That’s why I need to work with what is possible right now.
make smoke-examples check-astyle (or make test, but that’s a bit more noisy)
That, and kaleidoscope-builder being a tad dumb, and only supporting one board per sketch (it’s possible to work that around, but its better if we don’t have to).
Jesse and I talked about this topic not so long ago, and one idea was to require each hardware plugin to supply a KEYMAP_TEST macro, with 9 keys or so, so it’s the same across all keyboards, thus allowing us to compile the same sketch for all boards, without having to ifdef anything. I think this is pretty close to what you had in mind too?