In a recent post@jesse made it clear that he is not fond of CMake. What you read below is therefore likely not to be supported by keyboardio.
Direct any future questions about the matter @noseglasses. Leave issue reports and pull requests here.
support of faster, modern build tools such as e.g. Ninja,
only builds what is out of date,
platform independentness,
ā¦
I added Travis based regression testing. Currently, the stock firmware is build with Kaleidoscope-CMake and the stock GNU make build system. Both generated .elf files are compared to assert identity of the contained symbols.
Kaleidoscope-CMake was developed under Linux. Testers working on other platforms are warmly welcome.
I couldnāt resist the temptation to port all of kaleidoscope_builderās features. So most of the make targets that you are used to, such as e.g. make astyle, are available with Kaleidoscope-CMake.
Tomorrow Iām going to try this in CLion which needs a CMake file. I was trying to hack one together. But with no āreal worldā C experience, and absolute no CMake experience, it was very slow going. So thanks @noseglasses for putting this together! Iāll let you know how it works.
In Kdevelop, the IDE I mostly use, you just select Open/Import Project... then the CMakeList.txt file in your Kaleidoscope-CMake folder and then a binary folder where the generated files are placed. The latter might be preexisting if you already build at the command line. The rest is handled by the IDE.
I suppose that I will be quite similar for other IDEs that directly support CMake. Others, like e.g. Virtual Studio require a one-time preparation of the build system at the command line, as described here, before it can be loaded in the IDE.
Let me know it you need support for your particular IDE.
FWIW, embedded or not embedded. From the point of the build system and IDE integration that does not matter at all. The build system sees just a compiler, a bunch of files and some relations between them, just as if it was an ordinary host build.
CLion supports CMake naturally. Actually it uses it for its own project management. The following link points to some information about how to open/import CMake projects (Kaleidoscope-CMake).
@noseglasses - Iāve been trying to tease apart the reason behind Arduino-CMake and its duplication of the actual compilation logic built into arduino-builder. As far as I can tell, the original dates from before arduino-builder existed as a standalone tool. At that time, the only option was to use the GUI, which didnāt expose any way to get at the compilation process.
In that context, it makes a ton of sense. (Weād initially adopted an alternate implementation that was a crazy, baroque set of Makefiles.)
Easy integration into other IDEs is a valuable feature and one Iād like to do what we can to support.
Is there a reason, other than āhack valueā, that you decided to reimplement, rather than wrap the various non-compilation targets and kaleidoscope-builder features? It seems to me like it would be less work for you and weād have less chance of feature drift if those targets were simply wrappers around the existing implementations.
I am glad you asked, and thank you for taking the time. There are several reasons behind Kaleidoscope-CMake (and Arduino-CMake):
When developing C++ code, the compile overhead that comes with compile-modify-compile cycles sums up heavily. This is usually acceptable, as tools like GNU make try to determine what actually needs to be compiled. Kaleidoscope builder currently compiles the whole firmware every time which is a tremendous waste of time and energy that sums up. Even very skilled programmers need to run the compile process several times before they get rid of all syntax errors. And then, they are still not done as they need to fix the run-time errors and then compile againā¦and againā¦and againā¦
Kaleidoscope-CMake ensures that using any underlying build system on any platform will build only the necessary bits. Moreover, those that have tried Ninja in comparison to GNU make will confirm that it is pleasantly faster in determining what needs to be compiled, when e.g. compared to GNU make. That really pays off, even for small projects with a small amount of files to compile.
Everybody nowadays has multi-core processors. Why not use them for compiling as they are there. Actually, compiling is one of the processes that can most easily be parallelized. I never do anything else then at least make -j 8 when I build the firmware (with Kaleidoscope-CMake). Parallel builds are not supported by arduino-builder, as far as I can see, at least it is not by kaleidoscope-builder.
I thought the last point, platform independentness, to be quite important, too. Isnāt it one of the major advantages of the Arduino IDE, that anybody can run it on every platform? So, why would we want to force advanced users to use Unix-compatible platforms or to run Cygwin on Windows, just to be able to use the GNU make based build system? Donāt Windows users deserve to use the advanced features like make astyle, too. And why shouldnāt Windows users be able to use their favorite IDE for code development instead of the pretty limited Arduino IDE?
Just to make this clear, I am a hardcore Linux user. But it sometimes makes me angry, when I find non-portable open-source tools that are Windows-only. I try to keep my projects portable.
One more word about portability. I could not make everything supported by kaleidoscope-builder truly portable. Build targets like make astyle require the astyle binary which is currently only bundled together with the Kaleidoscope project as a Linux version. If Windows and OSX binaries would be added to the Kaleidoscope repository, I could easily make Kaleidoscope-CMake support the astyle-target in a platform independent manner.
In other places I mentioned, that I certainly hate reinventing the wheel. But as you pointed out, Arduino-CMake was there even before arduino-builder became a standalone tool. Which raises the question, which wheel was invented first? The mere thing I did, is wrapping the, in my eyes, more suitable wheel in a way to make it usable to build Kaleidoscope and to provide advanced functionality that was actually missing
Nope, the term module is very old and modular programming has been around for ages. At least slightly longer than the term Arduino Library. It is right, though, that future C++ will have a new concept that might be called modules. Mr. Stroustrup and friends have to deal with the same problems as we do, i.e. finding appropriate terms for wug.
Using the term Arduino Library too excessively does not seem to be a good idea. Even though the Model 01 currently uses Arduino as a platform, nobody can tell what is going to last longer, Kaleidoscope or Arduino. Quite often software outlives its initial platform and it is wise to anticipate that.
To me it seems that Kaleidoscope is defined more by its core structure and plugins than by the fact that it currently uses Arduino under the hood. Soon there might be ports to teensy or other hardware. The bottom line is, it is just a C++ program. It can be ported to and build everywhere where a C++ compiler is available. It already runs on x86 without Arduino.
Letās return to the proposed categories.
How would you categorize Kaleidoscope-Hardware-Virtual? It is not a pure Arduino library as it cannot be build without creating symbolic links manually.
Thatās not correct. The project has an explicit goal of being built on top of the Arduino core and API. I do not expect that to change the foreseeable future. Even the virtual x86 work that you mention works by implementing an Arduino core.
My point is that using the Arduino core and API is an explicit design goal that, for many reasons, Iām not interested in backing away from. Using Arduino makes many, many things easier. From community knowledge to well-maintained cross-platform toolchains to ports to almost every microcontroller available, Arduino ticks the boxes for me.
Heck, even your CMake implementation was easier because someone had already built the infrastructure for Arduino-based projects.
Autotools and, since more than a decade, CMake provide quasi standardized ways to develop free software.
They were used by billions of devs for millions of FOSS projects before Arduino came around. The Arduino-CMake project that I based my work upon just tries to restore some of the well established techniques of free software development, that Arduino threw unnessesarily over board. Believe me, having to restore this is a veritable PITA, for the Arduino-CMake folks and for me.
So the implementation was not easier but harder because Arduino exists. Arduinoās hardware definition system is nice. But why didnāt the Arduino folk just use Autotools or CMake under the hood?
But all this is already off topic. When a software is good (and I hope you believe in yours), it is ported to other platforms sooner or later. With or without the Arduino stuffā¦who can tell.
I understand and appreciate your motivation to make Kaleidoscope and your keyboard(s) as user friendly as possible, thus choosing Arduino. But we are talking about a FOSS project. This means that someone needs to fix bugs and develop new fancy features. And that is in most cases not the unexperienced end user. Because of that, it might be worth also considering developersā needs. Those are the ones that drive the project and keep it usable.
Please be honest, do you ever use the very limited Arduino IDE? If not, wouldnāt it be nice to build from your preferred IDE without complicated tweaking?
About my motivation for Kaleidoscope-CMake
I am currently working on a Python based firmware simulator. It provides a Python API for the whole Kaleidoscope core and for all those plugins that will support python export (which will be very easy and not break the ability to build the firmware with Arduino-Builder). This python API enables prototyping new plugins without touching C++. I personally love C++, but interpreter languages allow for much faster development cycles and less frustration. Once a python implemented plugin works, it is fairly simple to port it to C++ and then test it again with the Python simulator before building and flashing it. My goal is to take the current burden to build, flash, test and be frustrated because it does not work from the developers and allow them to develop their extensions in a more developer friendly environment. This overall process will be way faster than implementing directly in C++. The Python API furthermore provides very convenient means of testing every least part of the firmware and its plugins.
As a proof of concept I have backported one of the stock plugins (DualUse) to Python. I selected DualUse because it is not too much code and because I recently read that it might need some debugging.
As you can imagine, the Python simulator toolās setup and build process is far too complex to squeeze it into the Arduino-Builder framework, although it uses great parts of craigdissels Kaleidoscope-Hardward-Virtual wug. The complexity of this build and the limits of Arduino-Builder are the main reasons why I came up with Kaleidoscope-CMake. I hopefully soon can offer a development tool that developers can setup with a single command on any platform. And, build on that, a powerful and easy to maintain regerssion testing system.
I hope that explains my motivation a little better.There are certainly nicer things to do than fiddeling with build systems. I am looking forward to play with my own M01 soon!
P.S.: Could you please somehow move the last part of this discussion here? It is somewhat separate from the topic about terminology.
Not willing to move away from the Arduino ecosystem does not mean oneās supposed to use the Arduino IDE, or be subjected to complicated tweaking. I never once used the Arduino IDE for my work on Kaleidoscope, did everything from the comfort of my Emacs. Nor did I have to go through complicated steps to work on neither Kaleidoscope, nor the plugins. So right now, as a developer, I can comfortably work with Kaleidoscope & plugins, from the comfort of my IDE, without anything more complicated than a git clone & make setup.
Developers adapt much easier than users. For an experienced developer, adapting to a projectās coding style, build environment, and so on, should be considerably easier than for an end-user. Mind you, the two things are not exclusive: we can make things easier for the end-user, and still keep it workable for the developer. There is a middle ground, and Kaleidoscope is - in my opinion - close to that goal.
It is also worth keeping in mind that there are a lot more users than developers, and without users, the work of developers is useless too. Therefore making it easy for end-users is a goal I fully support, even when that sometimes goes against my desires as a developer.
Not to mention, that from all of the firmware I worked with (QMK, Kaleidoscope, UHK) or tried (Animus, KLL), Kaleidoscope was by far the easiest to get started with. And when I got started with it, a lot of the convenience stuff we have now (kaleidoscope-builder, plugins, etc) werenāt even there.
As a developer, I can adapt with reasonable ease. I can develop tools that help me work with the firmware (like you did with Kaleidoscope-CMake). Users canāt do that, or if they can, it is much, much harder for them. Iād rather see developers have to adapt, and be slightly inconvenienced, than see users try and fail to take full advantage of their keyboards. Besides, who is a user first, may very well become a developer or contributor later. If we make that path easy, weāre on the right track.
Using Arduino, and sticking to Arduino conventions by default is the middle ground. The path that allows end-users to experiment, and allows developers to get their work done too. For those who have worked with Arduino before, things are even easier.
This is a commendable goal, but only as long as it does not go against the default of using Arduino tools.
(Myself, Iād rather prototype in C++ directly, if Iāll have to port it anyway, I may as well skip that step. Not a big fan of the language, but Iād rather write in one language than two.)
Unlikely. Most developers are also users. So whereās developers you will always find users. But whereās users there are not necessarily developers. My motivation is - apart from boundless altruism - to use the stuff I develop.
Agreed.
Lots of the discussion in the area of programming in this forum is about questions concerned with how is this done or can this be done given the Arduino project structure. There have been many questions e.g. about where to put plugins for them to be picked up by the build process. I wouldnāt call that even easier and it is due to the fact that we have to live with how Arduino forces us to do things.
Even for you it appears to be difficult. Jesse recently wrote that Arduino changed their build system from one
version to another without properly documenting it. Having a opaque build system seems far from optimal. That is my main point here.
What I mean with tweaking is ā¦ the opposite of choosing Open Projectā¦ from a menu, selecting a file and ā¦ there it is, with all build targets and code files and include paths. Everything necesssary for the project language and symbol parser as well as auto-completion to work. My IDE even has a real time clang based syntax check that tells me while I type if the code will compile.
I know that Emacs is powerful but I doubt it is that smart with handling Arduino projects as all those IDEs that allow CMake import.
In the beginning of my programmer life I was a passionate vi user - for a long time. It took me at least as long to overcome my religious believe (typical for vi and emacs users) that this is the most productive way to work. Now, with a proper modern IDE, I am much faster than I ever was with vi. But that, of course, requires a proper build systems. And there we are back again.
I am a very experienced C++ and a much less experienced Python programmer. Still I am very sure that, when I have to develop a complex algorithm for an embedded platform such as Arduino, I am faster with two languages and porting. Mind you, Arduino lacks a debugger (at least most developers do not have one), doesnāt even allow for simple printf output and, moreover, requires an upload step in each change-test cycle (I am writing this for other users, I am sure you know how annoying it can be to work under such circumstances ).
BTW, there are roughly three times as many Python programmes as there are C++ programmers. So, if people could, they probably would prefer to develop their Kaleidoscope plugins in Python.
Thompson's Rule for First-Time Telescope Makers
It is faster to make a four-inch mirror and then a six-inch mirror than to make a six-inch mirror.
Bill McKeenan
Wang Institute
- to use the stuff I develop.
).