Indeed. When I get a chance, I’ll look into it.
Your directory structure and the idea to add a description, comes close to what I am working on. Apart from that, my approach is a bit broader. Here are my basic ideas. The implementation is in progress.
-
The Python wrapper plugin will not only wrap the Hardware-Virtual plugin but also any other module that is to be tested and that specifies basic information about what to wrap. This of course also includes the firmware core. Essencially all classes, functions and globals will be accessible from Python. This will enable us to check even very complex post-conditions of a keypress such as modifiers being set or whatever. You could e.g. check the highest layer after a keypress or the content of the keymap arrays. I think there shouldn’t be any limits. It is important to mention, that the Kaleidoscope modules are going to responsible to define what is meant to be exported. At the bottom of the Kaleidoscope-Python-Wrapper’s README, you can see such a specification. The files that define the Python export have the file extension
.python-wrapperto prevent them to be compiled in ordinary avr-builds. They are only used when the Kaleidoscope-Python plugin is active. -
I am planning to add a Python callback that intercept the hid_report. Thus, it is is simple to write an assertion that expects a specific keyreport or that does any type of complex checks on what is reported. For this to work I would have to modify the hid_report implementation of your plugin a bit, allow for callbacks to be registered. What do you think?
-
The testing directory structure will be recursive. On every directory level a sketch, a Python driver and a yaml-file with additional information can but do not have to be present. By this means we can have an inheritance-like overriding of information that is defined in parent directories by files and thus information define in child directories. This enables a great reuse of information and minimizes the amount of redundancy in test specifications. I might even support to references to such information defined in external git repos to maximize the amount of reuse of testing specifications between Kaleidoscope modules.
I am open for any type of cooperation but I also see no reason why we shouldn’t have different testing approaches. What I am sure of is, that we should agree on the testing system file structure. As I wrote, I plan to store most information in yaml files that I am going to parse with python. But I could also imagine to keep it simple and go without yaml. I just don’t want to have too many different files that are necessary to define one test.
EDIT 1: Updated the name of the module from Kaleidoscope-Python-Wrapper to Kaleidoscope-Python.
2 Likes
Awesome! This seems like a great approach. I’ll take a look at that plugin when I have the chance.
2 Likes
I’ve updated my POC regression-testing framework for recent changes in the Arduino-Boards repo. So, anyone interested in trying my regression testing code should now check out the regression-testing branch in my forks of the Kaleidoscope and Kaleidoscope-Build-Tools repos.