User plugin database


(Noseglasses) #1

Lately there has been some discussion in the forum about a collection/database/installer of/for user plugins.

I am pondering for a while now on how that could be done and I am interested to discuss the matter with you.

Let’s start with the information I would regard important about a user plugin:

  • name
  • description
  • required/supported Kaleidoscope version
  • git URL
  • current maintainer (directions for issue reports, PRs, etc.)
  • dependencies (other plugins)
  • compatibilities (can replace other plugins)
  • known incompatibilities with other plugins

I heard about ideas towards a future installer for user plugins. Maybe there are already ideas about that in the Crysalis team?

Such an installer would certainly required the above plugin information to
be machine readable/parsable. Thus, we would best have some sort of database
or at least use a pre-defined data format to store the given information.

At first I thought that we might want to collect the information directly in the
Kaleidoscope wiki. But that is probably not a good idea as, to my knowledge, markdown does not
provide an appropriate way to define data in a machine readable fashion.

Another important point is that such a solution would probably be too brittle. Every user could
accidentally render the information unparsable by introducing formatting errors when adding or modifying
information about her/his plugin.

An appropriate solution could be to have an independent
dedicated GitHub repo that basicaly stores some sort of yaml/xml/json or whatever
file that represents the user plugin database. This information would
then be machine readable and we would have travis tests that ensures the
well definedness of the data.

The GitHub repository would have a maintainer that would act as gatekeeper
to ensure that the necessary information for a plugin would be provided and the formatting is ok.
Users could, thus, update the information regarding their plugins via PRs.

Instead of yaml/xml/json/etc. we could also use html tables which are
also quite simple to parse. This would have the great advantage that
we could include the current version of the html file in the Kaleidoscope wiki (GitHub wiki markdown can handle embedded html, no?).

What do you think?


Plugin: Adhoc Macros
Managing core and contributed Kaleidoscope plugins
(kajsa.anderson) #2

I agree that there’s a need, and think we can probably leverage one or more of the patterns used by (off the top of my head) cpan, pypy, yum, etc.


(Jennifer Leigh) #3

I agree that the wiki is probably not the right place to build this, though you could potentially create the wiki page as output from your database. GFM has limited html support, but there is a table building syntax. If you look at the wiki home page you will see an example. I first tried to build that table as a straight html block, but it wouldn’t render properly. I was able to create bulleted lists in the cells, though.

It would be nicer, though, to have a searchable directory with tags and pointers to the community thread for a given plugin, recommendations, notes, etc. Not sure where such a thing would be hosted, but I expect we could find someplace. This may be more work than you want to get into for this theoretical installer; I’m just spitballing here. :slight_smile:

At a minimum, I would like to end up with some clear way for the user to know if they are looking at core plugins or user contributed plugins, but ideally they would be able to search both groups in the same directory.

Thanks for working on this. Please let me know if I can help in any way.


(Jesse) #4

This is another thing that Arduino provides as a global service with direct Arduino IDE integration. It even publishes a public JSON index: http://downloads.arduino.cc/libraries/library_index.json

I have built and maintained a system like this in the past and helped others attempt to bootstrap similar things. If we’re going to reinvent this particular wheel, I’d like us to have a really strong case for why we shouldn’t use the existing infrastructure provided by the platform.


(Jennifer Leigh) #5

hooray for somebody else solved this so we don’t have to!

So what do folks need to do to register their plugins?


(Jesse) #6

(Noseglasses) #7

Just to clarify this. I did not intend to work on any kind of installer but I read other posts in this forum where people expressed thoughts about an installer, e.g. here

I am mostly interested to have a place where users can inform about their plugins, even in a way, that they list it with status under construction. There are so many good ideas, this is really a field where there is a lot to do and there is much fun in it. That’s why I thought it to be important, to have a place to look what is already there or what is in progress. Thus users can contribute to growing plugin projects instead of accidentally reinventing the wheel.


(Jesse) #8

for that particular use case, I really feel like threads on the forum are going to be a lot more useful and accessible than a database.


(Noseglasses) #9

Nice. So, I was not that wrong with my idea about about a database and a data format like json. Fortunately, I am not the first that came up with this idea, so no Heureka this time :smiley:

The only thing that I am not sure of is how to make the json information available to the wiki, in a way that would make it as simple as possible for users to only browse Kaleidoscope stuff and, moreover, to hide all the information that is irrelevant for a quick search, like architecture, types, etc.

There’s one thing, where I am not sure of. People might sometimes develop things that are not actually a Arduino library but something related, like any type of auxiliary scripting, etc. Information about such miscellaneous stuff would be at least as valuable as news and info about plugins.
How could we handle this? And could we reuse on of the entries of the json format for information that is relevant for us, say the distinction between user and stock, etc. plugins? Are we we free to add additional entries?

I’m afraid, I disagree. One reason why I started this topic is that I find it very inconvenient, having to use discourse`s search engine to determine if someone is already working on something or not.
Nobody writes forum posts with the clear intention that the provided information is supposed to be found easily, later on. At least, I don’t.

I just spend approx. 5 minutes to dig up algernon’s post about his ideas towards a plugin manager. Just as an example how hard it can be to find something that you even know it is there.

I would really appreciate some sort of database with nice search features. That does not mean that I wouldn’t want to discuss plugins, ideas, etc. here. I like open discussion and I am really fond of other peoples contributions about fascinating topics. I even read a lot of posts about hardware-hacking a keyboard that I do not physically have, at least no just yet. :wink:


(Jesse) #10

In my experience databases of ‘work in progress’ projects quickly become…graveyards of good intentions and placeholders for things that never got started. Can you show me a project that has done what you’re describing and had it work out well? That might help me understand better.

Alternatively, would turning on discourse’s ‘tags’ support help make it easier to find what you want?


(Noseglasses) #11

Not sure. As long as it is obviouse that a project is or was in progress, the information about it is still worth something. Even a dead project whose GitHub page I can visit, can be inspiring. Hope that didn’t sound too nekrophiliac :vampire:
I could e.g. imagine to setup rules that say that if the URL of a listed project becomes unavailable for a certain amount of time and the the maintainer does not react, the project is kicked out of the list. Just as an idea.

No, due to a lack of experience. This is the first project I am contributing to. But you sound like you could name a good negative example.

Do you mean this?

To me tags sound generally attractive. Having the ability to assign multiple tags might be helpful. A topic would still live inside a category. But, it seems that we can only tag topics, not posts, or did I miss something? I often search for posts where someone mentioned something in particular. Thus, in my opinion, it would help to tag an individual post within an otherwise untagged topic.


(Michael Richters) #12

I want to second @noseglasses’s opinion regarding the usefulness of Discourse as an information storage and retrieval system. I have found it quite difficult and time-consuming to find both topics and posts that I want to refer to if they’re not actively being discussed at the time, to the point that I often give up or don’t even try.

I do think it would be helpful if someone maintained a current list of plugins in a pinned topic on this forum, but having something that’s machine-readable would be even better, so I could list and search them from the command line, à la aptitude (I never use apt-get directly…).


(Ian Kelling) #13

I agree, especially for plugins that are in development and in the idea stage.


(Ian Kelling) #14

Alternatively, would turning on discourse’s ‘tags’ support help make it easier to find what you want?

New users will want to read through a list of all the noncore plugins people have published. Reading all the threads in this category seems to be the best bet right now, but half of the threads aren’t an announcement of a plugin, which is annoying and just feels like it should be fixed as the problem will grow. I suggest either: 1. enable tags and make one for plugin releases (i haven’t used the tag functionality, so not sure if this would work), 2. split this category into 2, one for announcements of plugins, one for everything else programming + plugins, and move the existing threads into the appropriate category. 3. Start a wiki page with a list of non-core plugins. 4. An alternate wiki for making a list of plugins is https://directory.fsf.org, just make a category for keyboardio plugins.


(kajsa.anderson) #15

+1 for a wiki page with a list of plugins, ideally with links to associated code and threads here


(Craig Disselkoen) #16

The original purpose of the Plugins sub-category within Programming was to have only plugin announcement threads. (See About the Plugins category and this other thread beginning at about post 8.) Although I’m totally in favor of having a wiki page with a list of plugins, brief descriptions, links to threads here, etc - another (stopgap?) measure we could take would be to kick out all the threads other than plugin announcement threads back into Programming instead of the Plugins sub-category. (Threads can be very easily recategorized, even retroactively.)


(Jesse) #17

…or to create another subcategory. do folks have name ideas?


(Craig Disselkoen) #18

I suppose we could let the non-plugin-announcement threads have the ‘Plugin’ category, and create a new category ‘Plugin Announcements’ or ‘Available Third-Party Plugins’ for plugin announcements. This would be clearer, as the original name ‘Plugins’ clearly isn’t communicating the intended purpose of third-party plugin announcements to average users.

Looking at the existing threads in Plugins, other than plugin announcements we seem to have:

  • Questions about usage of particular plugins
  • Questions about creating plugins
  • Questions about which plugin to use for a particular task
  • This thread, re: plugin database

As far as I’m concerned, I’m not sure there’s a meaningful difference between the threads listed above and threads that belong in the ‘Programming’ category in general. Indeed, sometimes the question about which plugin to use ends up being a programming question more than a plugin question.