Managing core and contributed Kaleidoscope plugins

I have installation instructions in the README for MacrosOnTheFly, hopefully they are intelligible to people with little experience with Arduino or Git. Improvements welcome. It probably makes sense to have instructions in the Kaleidoscope documentation for how to install 3rd-party (i.e. non-Arduino-Boards) plugins as well.

I suppose there are two possible models for the long-term. One is that Kaleidoscope includes an easy-to-use process for installing 3rd-party (i.e. non-keyboardio) plugins. Maybe eventually there’s a plugin manager (not unlike vim-plug, for Vim, or Antigen, for Zsh) that can automatically download and install plugins from various sources.

The other possible model is that all external plugins eventually get “approved” and added to Arduino-Boards. In this case, keyboardio acts as gatekeeper/curator for 3rd-party plugins, and a default install of Kaleidoscope includes “all” the plugins in the universe. (A comparable model might be that of oh-my-zsh, for Zsh.) Note that Arduino-Boards totally has the ability to include plugins from anyone, without necessarily putting them in the keyboardio org - org membership could be a separate issue from Arduino-Boards membership. Probably Arduino-Boards membership would be a low-ish bar, whereas org membership would indicate which plugins keyboardio takes either partial or complete responsibility for maintaining.

The first model is a little more scalable - if the number of plugins and 3rd-party contributors grows very large, it could be burdensome for keyboardio to approve each plugin, or especially if keyboardio has to approve each plugin update via pull request. However, the second model seems intuitively more keyboardio-like - approving plugins is a step towards making sure users’ experience meets some standard. Note that ambitious users could still always install non-Arduino-Boards plugins manually, but Arduino-Boards would be considered the canonical list of “official” plugins.

I agree that it seems like there should be a few levels of support for plugins. Maybe there are some “core” plugins, which live in the keyboardio org, are actively maintained by keyboardio, and are part of Arduino-Boards and therefore included with every default install of Kaleidoscope. Then “officially supported” plugins, which would be at least partially maintained by keyboardio (in cooperation with plugin author, probably); in the first model above, they would not live in Arduino-Boards so not be installed by default but could be installed with the package manager, whereas in the second model, they would live in Arduino-Boards. Probably these would live in their original authors’ orgs and not the keyboardio org? although I could see that go either way. If an author of one of these plugins wanted to leave, a decision would have to be made which other category to transition the plugin to. Then “community-supported” plugins, which would not be maintained by keyboardio at all; in the first model, they would probably just be in a different section of the package manager, or in the second model, maybe a different section of Arduino-Boards. (Or in either model they could live directly alongside the officially-supported plugins - end users may or may not care whether keyboardio is involved in maintaining the plugin.) In either model, these would not live in the keyboardio org.

Another thing I’d totally like to see is a master list of available plugins (ideally, all plugins in all three of these categories). This goes regardless of which plugin model we use. Even a list with plugin name, plugin author/maintainer, single-line description of the plugin, and link to install would be great; most plugins already have all this info in their library.properties file, so we’d just need to scrape that. The resulting list should be distributed with Kaleidoscope core, or perhaps (to allow more frequent updating) we could just distribute a script or something to pull the latest list. In any case it should be very accessible to end users with little knowledge of Arduino or command-line tools.

1 Like

My feeling is that there ought to be three ‘tiers’ of plugins in the Kaleidoscope universe. I believe that it’s pretty important that we ship a blessed ‘distribution’ containing both core functionality and ‘neat addons.’ It’s just as important that we not act as gatekeepers for every possible plugin written by an end user. That’s not how you end up with a community :wink:

  1. Core plugins. Plugins that are part of the ‘Kaleidoscope core’ - These are supported just like the Kaleidoscope firmware itself. They are shipped as part of the standard distribution and are managed by the same people who manage Kaleidoscope. https://github.com/keyboardio/Kaleidoscope/blob/master/CODE_OF_CONDUCT.md applies to these plugins

  2. Contributed plugins. Plugins distributed with Kaleidoscope that are not considered core functionality, but that we, at least to a reasonable approximation, maintain and support. They should be in the same github org as Kaleidoscope and the same folks who manage the core should have write access. https://github.com/keyboardio/Kaleidoscope/blob/master/CODE_OF_CONDUCT.md applies to these plugins.

  3. Third party plugins. We don’t distribute these ourselves. We don’t maintain or support them. They don’t live in our repositories. While we would hope that contributors to these plugins would embrace https://github.com/keyboardio/Kaleidoscope/blob/master/CODE_OF_CONDUCT.md, that would ultimately be up to the author.

The big difference from what Craig proposed in his second proposal is, I think, that I feel strongly that if a plugin is included in our default distribution, “we” have the ability to make changes to it. My goal isn’t to assert copyright on others work or to take anything away from them. I don’t expect any scenario where we’d be kicking the original author of a contributed plugin out, unless they were violating the CoC and we couldn’t resolve things through by talking them out.

It should be possible for plugins to go from being a third-party plugin to being a core plugin or a contributed plugin, so long as the parties involved agree.

Contributed plugins may end up adopted as core plugins.
If unloved, contributed plugins may be cast out as third-party plugins.
Core plugins may get punted down to being contributed plugins, but that will hopefully be very rare.

There’s an additional wrinkle to all of this. Hardware device vendors who ship keyboards with Kaleidoscope will likely have their own ‘distribution’ of Kaleidoscope with plugins they think are valuable to their customers. At least for the foreseeable future, I expect that Keyboardio will just ship the standard distribution of ‘Core + Contributed’

Does this seem sensible?

PS - I’m lifting this subconversation to a new topic.

4 Likes

Makes a lot of sense. I totally understand the need for keyboardio to have write access to all plugins in the default distribution - I wasn’t intending to say otherwise.

I was misunderstanding a bit the relative importance of membership in the keyboardio org - I was thinking that (in the second model) more plugins would be in Arduino-Boards than the keyboardio org, whereas clearly the correct understanding is that more plugins would be in the keyboardio org (that is, core + contributed plugins) than Arduino-Boards (that is, just core). I’m relatively new to this GitHub / open-source stuff and don’t really understand what an org means or entails.

Regarding the plugins I’ve developed, because I know that started this discussion - MacrosOnTheFly, or also Hardware-Virtual or LetterGuesser - if you want keyboardio to have push access or want any of them to be added to the keyboardio org or Arduino-Boards, I’m perfectly willing to help with that. I’d kinda like to retain push access myself too, if that’s ok. We can also take such a discussion somewhere else if necessary.

1 Like

Right, I was trying to say that I would not envision yanking push access from a plugin author, even if they agreed to have us ‘adopt’ their plugin :slight_smile:

I’m deep in some dumb logistics stuff, so it may be a little bit before I get there, but yes, both MacrosOnTheFly and Hardware-Virtual seem like things it makes sense to pull in. LetterGuesser is a good example of something cool that may not need to be distributed with Kaliedoscope.

I suspect that Arduino-Boards is a representation of core + contrib and that some libraries (the core) may end up lifted into the Kaleidoscope repo itself. But I’d rather move a little more slowly than do this wrong.

2 Likes

Sounds good. I was assuming LetterGuesser is not the kind of thing we’d want to move to contrib, but just included it for completeness. :wink:

2 Likes

Makes sense. At the same time, it’d be nice if there was an aggregated list of available plugins somewhere. With my IDE for example, there is a plugin repository (of compiled and packaged plugins) that I can search and browse. The third party writers still maintain their own code repos and support. Some parallel to that would be useful. Obviously something that would not create a lot of work for a two person company to maintain :wink: As a start, perhaps a topic category here for Plugin Authors to announce/list their plugins.

It is already possible to get libraries into the Arduino Plugin Manager (though the process isn’t the best), so third-party plugin authors can publish their libraries, and they’ll be available via the Arduino IDE.

A plugin announce topic (or perhaps category?) would be useful as an additional source of info, too.

Yes, a category is what I was thinking, but I used the inappropriate Discourse term. I’ll edit my previous post.

We already have a “Programming” category, which has seen plugin announcements before, I think that’s a reasonable place to post any future plugins, and each can have their own topic, for updates & whatnot. I’m not convinced a separate category is required (but then, it’s easy to create a category later, and move topics over, if that turns out to be a better approach).

Should make it more obvious how to announce plugins, to make the topics easier to discover.

My thought was a dedicated category would allow someone to browse that category for available plugins. Having a dedicated category, say “Plugin Listing”, removes the other noise of all the more low level programming questions and discussion. Ideally, IMHO, each plugin would have a single topic. Thus if you browse to the “Plugin Listing” category, you have a nice listing of available plugins.

It is my intent to have a plugins page on the wiki that has a listing like this. I’m not sure if this forum is the right place for a list like that, because it is mostly free-form conversations.

I think everybody is right. (Though @algernon, there’s enough plugin traffic that adding a category is pretty easy. I’ve been wanting to split ‘programming’ up a bit anyway. There’s “user sketches”, “kaleidoscope core” and “plugins” and they’re all pretty different.)

It’s not wrong to have a place for people to discuss plugins, with a thread per plugin. I think that the canonical plugin list being on the wiki makes sense. But I totally see the need for people to have threads to discuss individual plugins. (Especially the user-written ones that the Kaleidoscope authors don’t support or know about.)

1 Like

thread per plugin sounds perfect! I was imagining an announcement only category, with discussion squelched, and that sounded wrong. :slight_smile:

I would l like to see a plugin category like we have a programming category. I think these things are related but they are not the same thing. It is nice to be able to search a specific category when looking for something and I think that the programming category would have more noise to sift through.

I would love to see a plugin manager like Plug for vim or even a full blown package manager that people can add plugins and dependencies too. Something like hex, rubygems, or npm.

Jesse created it here:

https://community.keyboard.io/c/programming/Discuss-Kaleidoscope-Plugins-one-thread-per-plugin

1 Like

One of my plans is to do this within the context of Chrysalis (and also provide a CLI tool for those who do not want to use a GUI). Details are TBD, we’re not there yet.

2 Likes

Does such a plugin page in the Wiki exist by now? I just read this topic over. The wish for some sort of plugin manager was expressed several times.

Wouldn’t it be a good idea to collect the users informations about their plugins in a data format that could both be rendered as a table on a wiki page and parsed/read by a future plugin manager?

Are there any ideas for a suitable format, platform, etc? Of course it would be best if users could generate new entries by themselves. I find it important to define the information that describes a plugin (name, description, url, whatelse?).

Names might become a problem once there are many user plugins around. The likelyhood of naming-conflicts grows with the total number.
Any ideas about a way to introduce namespaces, domains, etc.?

Here is another topic where the naming of plugins is briefly discussed

A closely-related question about third-party (or, more specifically non-Arduino-Boards) plugins is where one should put them when building custom firmware. In the libraries directory, along with the official ones? Symlink’d there? As submodules of one’s own firmware sketch? In a separate library directory? Does the build system already have a place for them?

If there isn’t already a system for this, let’s make one.

“libraries” in the Arduino sketch directory is traditional (~/Arduino/libraries or ~/Documents/Arduino/libraries depending on the OS.)

I’d thought that you could also drop them inside lib/ in your sketch directory. (Though now I wonder if that’s libraries/ inside your sketch dir.) If the IDE supports one of those and the commandline build doesn’t, that’s a bug and something we should fix.

1 Like