Kaleidoscope-CMake

Disclaimer

In a recent post @jesse made it clear that he is not fond of CMake. What you read below is therefore likely not to be supported by keyboardio.
Direct any future questions about the matter @noseglasses. Leave issue reports and pull requests here.

Description

Kaleidoscope-CMake is an alternative configuration and build system that is based on Arduino-CMake.

It provides the following features in comparison to the stock GNU make build system.

  • support for parallel builds,
  • integration with IDEs,
  • support of faster, modern build tools such as e.g. Ninja,
  • only builds what is out of date,
  • platform independentness,

I added Travis based regression testing. Currently, the stock firmware is build with Kaleidoscope-CMake and the stock GNU make build system. Both generated .elf files are compared to assert identity of the contained symbols.

Kaleidoscope-CMake was developed under Linux. Testers working on other platforms are warmly welcome.

I couldn’t resist the temptation to port all of kaleidoscope_builder’s features. So most of the make targets that you are used to, such as e.g. make astyle, are available with Kaleidoscope-CMake.

Detailed usage information is provided here.

Have fun!

6 Likes

I forgot to mention that my M01 is still in China. Thus, I couldn’t test Kaleidoscope-CMake’s upload feature. But it’s there.

1 Like

I’d love to see how the IDE integration works for an embedded project like this. Does everything magically ‘just work’?

Tomorrow I’m going to try this in CLion which needs a CMake file. I was trying to hack one together. But with no “real world” C experience, and absolute no CMake experience, it was very slow going. So thanks @noseglasses for putting this together! I’ll let you know how it works.

1 Like

Yes, this magic is the whole point of it.

In Kdevelop, the IDE I mostly use, you just select Open/Import Project... then the CMakeList.txt file in your Kaleidoscope-CMake folder and then a binary folder where the generated files are placed. The latter might be preexisting if you already build at the command line. The rest is handled by the IDE.

I suppose that I will be quite similar for other IDEs that directly support CMake. Others, like e.g. Virtual Studio require a one-time preparation of the build system at the command line, as described here, before it can be loaded in the IDE.

Let me know it you need support for your particular IDE.

FWIW, embedded or not embedded. From the point of the build system and IDE integration that does not matter at all. The build system sees just a compiler, a bunch of files and some relations between them, just as if it was an ordinary host build.

1 Like

CLion supports CMake naturally. Actually it uses it for its own project management. The following link points to some information about how to open/import CMake projects (Kaleidoscope-CMake).

https://blog.jetbrains.com/clion/2015/04/cmake-support-in-clion/

@noseglasses - I’ve been trying to tease apart the reason behind Arduino-CMake and its duplication of the actual compilation logic built into arduino-builder. As far as I can tell, the original dates from before arduino-builder existed as a standalone tool. At that time, the only option was to use the GUI, which didn’t expose any way to get at the compilation process.

In that context, it makes a ton of sense. (We’d initially adopted an alternate implementation that was a crazy, baroque set of Makefiles.)

Easy integration into other IDEs is a valuable feature and one I’d like to do what we can to support.

Is there a reason, other than ‘hack value’, that you decided to reimplement, rather than wrap the various non-compilation targets and kaleidoscope-builder features? It seems to me like it would be less work for you and we’d have less chance of feature drift if those targets were simply wrappers around the existing implementations.

I am glad you asked, and thank you for taking the time. There are several reasons behind Kaleidoscope-CMake (and Arduino-CMake):

When developing C++ code, the compile overhead that comes with compile-modify-compile cycles sums up heavily. This is usually acceptable, as tools like GNU make try to determine what actually needs to be compiled. Kaleidoscope builder currently compiles the whole firmware every time which is a tremendous waste of time and energy that sums up. Even very skilled programmers need to run the compile process several times before they get rid of all syntax errors. And then, they are still not done as they need to fix the run-time errors and then compile again…and again…and again…

Kaleidoscope-CMake ensures that using any underlying build system on any platform will build only the necessary bits. Moreover, those that have tried Ninja in comparison to GNU make will confirm that it is pleasantly faster in determining what needs to be compiled, when e.g. compared to GNU make. That really pays off, even for small projects with a small amount of files to compile.

Everybody nowadays has multi-core processors. Why not use them for compiling as they are there. Actually, compiling is one of the processes that can most easily be parallelized. I never do anything else then at least make -j 8 when I build the firmware (with Kaleidoscope-CMake). Parallel builds are not supported by arduino-builder, as far as I can see, at least it is not by kaleidoscope-builder.

I thought the last point, platform independentness, to be quite important, too. Isn’t it one of the major advantages of the Arduino IDE, that anybody can run it on every platform? So, why would we want to force advanced users to use Unix-compatible platforms or to run Cygwin on Windows, just to be able to use the GNU make based build system? Don’t Windows users deserve to use the advanced features like make astyle, too. And why shouldn’t Windows users be able to use their favorite IDE for code development instead of the pretty limited Arduino IDE?
Just to make this clear, I am a hardcore Linux user. But it sometimes makes me angry, when I find non-portable open-source tools that are Windows-only. I try to keep my projects portable.

One more word about portability. I could not make everything supported by kaleidoscope-builder truly portable. Build targets like make astyle require the astyle binary which is currently only bundled together with the Kaleidoscope project as a Linux version. If Windows and OSX binaries would be added to the Kaleidoscope repository, I could easily make Kaleidoscope-CMake support the astyle-target in a platform independent manner.

In other places I mentioned, that I certainly hate reinventing the wheel. But as you pointed out, Arduino-CMake was there even before arduino-builder became a standalone tool. Which raises the question, which wheel was invented first? The mere thing I did, is wrapping the, in my eyes, more suitable wheel in a way to make it usable to build Kaleidoscope and to provide advanced functionality that was actually missing :hushed:

2 Likes

Just out of curiosity, what’s your experience? Did you get along with it or do you need any support?

1 Like

Nope, the term module is very old and modular programming has been around for ages. At least slightly longer than the term Arduino Library. It is right, though, that future C++ will have a new concept that might be called modules. Mr. Stroustrup and friends have to deal with the same problems as we do, i.e. finding appropriate terms for wug.

Using the term Arduino Library too excessively does not seem to be a good idea. Even though the Model 01 currently uses Arduino as a platform, nobody can tell what is going to last longer, Kaleidoscope or Arduino. Quite often software outlives its initial platform and it is wise to anticipate that.

To me it seems that Kaleidoscope is defined more by its core structure and plugins than by the fact that it currently uses Arduino under the hood. Soon there might be ports to teensy or other hardware. The bottom line is, it is just a C++ program. It can be ported to and build everywhere where a C++ compiler is available. It already runs on x86 without Arduino.

Let’s return to the proposed categories.

How would you categorize Kaleidoscope-Hardware-Virtual? It is not a pure Arduino library as it cannot be build without creating symbolic links manually.

And to which category does Kaleidoscope-Ranges belong? A library commonly consists of functions or at least binary code. But there are none, see https://github.com/keyboardio/Kaleidoscope-Ranges/blob/master/src/Kaleidoscope-Ranges.h.

1 Like

That’s not correct. The project has an explicit goal of being built on top of the Arduino core and API. I do not expect that to change the foreseeable future. Even the virtual x86 work that you mention works by implementing an Arduino core.

Do you have some sort of crystal ball? If you have, I bet it runs Arduino.

I don’t, though one exists:

My point is that using the Arduino core and API is an explicit design goal that, for many reasons, I’m not interested in backing away from. Using Arduino makes many, many things easier. From community knowledge to well-maintained cross-platform toolchains to ports to almost every microcontroller available, Arduino ticks the boxes for me.

Heck, even your CMake implementation was easier because someone had already built the infrastructure for Arduino-based projects.

That is a very strange way to put it.

Autotools and, since more than a decade, CMake provide quasi standardized ways to develop free software.

They were used by billions of devs for millions of FOSS projects before Arduino came around. The Arduino-CMake project that I based my work upon just tries to restore some of the well established techniques of free software development, that Arduino threw unnessesarily over board. Believe me, having to restore this is a veritable PITA, for the Arduino-CMake folks and for me.

So the implementation was not easier but harder because Arduino exists. Arduino’s hardware definition system is nice. But why didn’t the Arduino folk just use Autotools or CMake under the hood?

But all this is already off topic. When a software is good (and I hope you believe in yours), it is ported to other platforms sooner or later. With or without the Arduino stuff…who can tell.

Nobody can foresee the future.

Sorry, what I meant was that your Kaleidoscope-CMake was easier to build because someone had already created Arduino-CMake.

I understand and appreciate your motivation to make Kaleidoscope and your keyboard(s) as user friendly as possible, thus choosing Arduino. But we are talking about a FOSS project. This means that someone needs to fix bugs and develop new fancy features. And that is in most cases not the unexperienced end user. Because of that, it might be worth also considering developers’ needs. Those are the ones that drive the project and keep it usable.

Please be honest, do you ever use the very limited Arduino IDE? If not, wouldn’t it be nice to build from your preferred IDE without complicated tweaking?

About my motivation for Kaleidoscope-CMake
I am currently working on a Python based firmware simulator. It provides a Python API for the whole Kaleidoscope core and for all those plugins that will support python export (which will be very easy and not break the ability to build the firmware with Arduino-Builder). This python API enables prototyping new plugins without touching C++. I personally love C++, but interpreter languages allow for much faster development cycles and less frustration. Once a python implemented plugin works, it is fairly simple to port it to C++ and then test it again with the Python simulator before building and flashing it. My goal is to take the current burden to build, flash, test and be frustrated because it does not work from the developers and allow them to develop their extensions in a more developer friendly environment. This overall process will be way faster than implementing directly in C++. The Python API furthermore provides very convenient means of testing every least part of the firmware and its plugins.

As a proof of concept I have backported one of the stock plugins (DualUse) to Python. I selected DualUse because it is not too much code and because I recently read that it might need some debugging. :wink:

As you can imagine, the Python simulator tool’s setup and build process is far too complex to squeeze it into the Arduino-Builder framework, although it uses great parts of craigdissels Kaleidoscope-Hardward-Virtual wug. The complexity of this build and the limits of Arduino-Builder are the main reasons why I came up with Kaleidoscope-CMake. I hopefully soon can offer a development tool that developers can setup with a single command on any platform. And, build on that, a powerful and easy to maintain regerssion testing system.
I hope that explains my motivation a little better.There are certainly nicer things to do than fiddeling with build systems. I am looking forward to play with my own M01 soon!

P.S.: Could you please somehow move the last part of this discussion here? It is somewhat separate from the topic about terminology.

P.P.S.: Nice Crystal ball!

Not willing to move away from the Arduino ecosystem does not mean one’s supposed to use the Arduino IDE, or be subjected to complicated tweaking. I never once used the Arduino IDE for my work on Kaleidoscope, did everything from the comfort of my Emacs. Nor did I have to go through complicated steps to work on neither Kaleidoscope, nor the plugins. So right now, as a developer, I can comfortably work with Kaleidoscope & plugins, from the comfort of my IDE, without anything more complicated than a git clone & make setup.

Developers adapt much easier than users. For an experienced developer, adapting to a project’s coding style, build environment, and so on, should be considerably easier than for an end-user. Mind you, the two things are not exclusive: we can make things easier for the end-user, and still keep it workable for the developer. There is a middle ground, and Kaleidoscope is - in my opinion - close to that goal.

It is also worth keeping in mind that there are a lot more users than developers, and without users, the work of developers is useless too. Therefore making it easy for end-users is a goal I fully support, even when that sometimes goes against my desires as a developer.

Not to mention, that from all of the firmware I worked with (QMK, Kaleidoscope, UHK) or tried (Animus, KLL), Kaleidoscope was by far the easiest to get started with. And when I got started with it, a lot of the convenience stuff we have now (kaleidoscope-builder, plugins, etc) weren’t even there.

As a developer, I can adapt with reasonable ease. I can develop tools that help me work with the firmware (like you did with Kaleidoscope-CMake). Users can’t do that, or if they can, it is much, much harder for them. I’d rather see developers have to adapt, and be slightly inconvenienced, than see users try and fail to take full advantage of their keyboards. Besides, who is a user first, may very well become a developer or contributor later. If we make that path easy, we’re on the right track.

Using Arduino, and sticking to Arduino conventions by default is the middle ground. The path that allows end-users to experiment, and allows developers to get their work done too. For those who have worked with Arduino before, things are even easier.

This is a commendable goal, but only as long as it does not go against the default of using Arduino tools.

(Myself, I’d rather prototype in C++ directly, if I’ll have to port it anyway, I may as well skip that step. Not a big fan of the language, but I’d rather write in one language than two.)

No, it doesn’t. Users will develop plugins that are going to be build with Arduino-Builder or the Arduino-IDE.

1 Like

Unlikely. Most developers are also users. So where’s developers you will always find users. But where’s users there are not necessarily developers. My motivation is - apart from boundless altruism :roll_eyes: - to use the stuff I develop.

Agreed.

Lots of the discussion in the area of programming in this forum is about questions concerned with how is this done or can this be done given the Arduino project structure. There have been many questions e.g. about where to put plugins for them to be picked up by the build process. I wouldn’t call that even easier and it is due to the fact that we have to live with how Arduino forces us to do things.

Even for you it appears to be difficult. Jesse recently wrote that Arduino changed their build system from one
version to another without properly documenting it. Having a opaque build system seems far from optimal. That is my main point here.

What I mean with tweaking is … the opposite of choosing Open Project… from a menu, selecting a file and … there it is, with all build targets and code files and include paths. Everything necesssary for the project language and symbol parser as well as auto-completion to work. My IDE even has a real time clang based syntax check that tells me while I type if the code will compile.
I know that Emacs is powerful but I doubt it is that smart with handling Arduino projects as all those IDEs that allow CMake import.

In the beginning of my programmer life I was a passionate vi user - for a long time. It took me at least as long to overcome my religious believe (typical for vi and emacs users) that this is the most productive way to work. Now, with a proper modern IDE, I am much faster than I ever was with vi. But that, of course, requires a proper build systems. And there we are back again.

I am a very experienced C++ and a much less experienced Python programmer. Still I am very sure that, when I have to develop a complex algorithm for an embedded platform such as Arduino, I am faster with two languages and porting. Mind you, Arduino lacks a debugger (at least most developers do not have one), doesn’t even allow for simple printf output and, moreover, requires an upload step in each change-test cycle (I am writing this for other users, I am sure you know how annoying it can be to work under such circumstances :wink:).
BTW, there are roughly three times as many Python programmes as there are C++ programmers. So, if people could, they probably would prefer to develop their Kaleidoscope plugins in Python.

Thompson's Rule for First-Time Telescope Makers
It is faster to make a four-inch mirror and then a six-inch mirror than to make a six-inch mirror.

Bill McKeenan
Wang Institute