Macros not working with v1.99.3 for Atreus and java.lang.NullPointerException when trying to get the 'master' branch installed in the Arduino IDE

Hello,

I’m using Chrysalis 0.8.4 and macros don’t seem to work for my Atreus if updating the firmware with v1.99.3. I’ve tried installing the ‘master’ branch in the Arduino IDE, but I get a java.lang.NullPointerException during the install process.

I was able to flash the Atreus with Kaleidoscope on a non-Windows computer using the ‘master’ and the macros work. I was using the Arduino IDE on a Windows machine for convenience.

Can someone help fix the issue in the Arduino IDE to allow it to install the ‘master’?

I’m ignorant of the issue, but assume the macros working versus not has something to do with this:

MASTER:
const macro_t *macroAction(uint8_t macro_id, KeyEvent &event) {

V1.99.3:
const macro_t *macroAction(uint8_t macroIndex, uint8_t keyState) {

Thank you!

Hello,

I use Arduino IDE on windows with no problem.
Could you send the entire error message and describe more which step of installation is in error ?

For me, macro works well with v1.99.3 firmware. But I never use Chrysalis for flashing.

Thanks.

Same problem here. Can’t install master branch on Mac.

Arduino errors log:

java.lang.NullPointerException
java.lang.RuntimeException: java.lang.NullPointerException
at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onInstallPressed$2(ContributionManagerUI.java:179)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at java.io.File.(File.java:360)
at cc.arduino.contributions.packages.ContributionInstaller.install(ContributionInstaller.java:160)
at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onInstallPressed$2(ContributionManagerUI.java:172)
… 1 more

Can you give us all steps you have done to come to this error in order to replay them ?

I followed official documentation on how to setup Arduino IDE.

I used stable release from

https://raw.githubusercontent.com/keyboardio/boardsmanager/master/package_keyboardio_index.json

and everything worked fine, could compile and flash firmware without any problem.

Today decided to try master branch (cause have some problems with configuring tap dance) and changed Additional Boards Manager Url to master branch. Error appears when try to install Keyboardio Board Manager.

I also have this error when installing master version. I think there is a problem with the packaging and arduino but can’t solve it.

A possible workaround is to download the master package :
https://raw.githubusercontent.com/keyboardio/arduino-kaleidoscope-master/master/builds//Kaleidoscope-2021.08.05182428.tar.bz2

Then uncompress the file in the users arduino directory, for windows it is C:\Users\username\AppData\Local\Arduino15\packages\keyboardio\hardware\avr\2021.8.5182428

It seems to work like this but I’m not sure it is safe.

I’m seeing the same issue on Archlinux as well, and was able to track it down. tl;dr and workaround at the bottom of everything, but check the next post for a better workaround.

I’ve tried to document everything (also as a note to myself). I took a closer look at the traceback I got:

java.lang.RuntimeException: java.lang.NullPointerException
	at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onInstallPressed$2(ContributionManagerUI.java:179)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at java.io.File.<init>(File.java:362)
	at cc.arduino.contributions.packages.ContributionInstaller.install(ContributionInstaller.java:160)
	at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onInstallPressed$2(ContributionManagerUI.java:172)
	... 1 more

This shows that it happens in Arduino’s ContributionInstaller.java on line 160:

File destFolder = new File(platformFolder, contributedPlatform.getParsedVersion());

This then calls into File.java on line 362:

public File(File parent, String child) {
    if (child == null) {
        throw new NullPointerException();
    }
    // ...
}

Thus, the crash happens because contributedPlatform.getParsedVersion() returned null. The source for that, in turn, says:

public String getParsedVersion() {
  Optional<Version> version = VersionHelper.valueOf(getVersion());
  if (version.isPresent()) {
    return version.get().toString();
  }
  return null;
}

thus, version.isPresent() was false. It’s coming from VersionHelper.valueOf, and taking a look at that reveals that either the version passed to it was empty or invalid:

  public static Optional<Version> valueOf(String ver) {
    if (ver == null) {
      return Optional.empty();
    }
    try {
      // [version parsing logic]
      return Optional.of(Version.valueOf(version + extra));
    } catch (Exception e) {
      return Optional.empty();
    }
  }

So something has either an invalid or missing version number. I really wish both Arduino and Java had some better error handling…

So let’s take a look at the master index JSON and compare it to the regular release JSON. Here’s the difference as generated by diff -Naur package_keyboardio_index.json package_kaleidoscope_master_index.json:

--- package_keyboardio_index.json	2021-09-02 20:18:48.769433980 +0200
+++ package_kaleidoscope_master_index.json	2021-09-02 20:18:54.128748865 +0200
@@ -1,451 +1,16 @@
 {
    "packages" : [
       {
-         "email" : "",
+         "email" : "jesse@keyboard.io",
          "help" : {
-            "online" : "http://keyboard.io"
+            "online" : "https://kaleidoscope.readthedocs.io"
          },
          "maintainer" : "Keyboardio",
-         "name" : "keyboardio",
+         "name" : "kaleidoscope_nightly",
          "platforms" : [
             {
                "architecture" : "avr",
-               [lots and lots of releases removed, but we only are interested in the differences to the newest release]
-               "archiveFileName" : "Kaleidoscope-1.99.3.tar.bz2",
+               "archiveFileName" : "Kaleidoscope-2021.08.05182428.tar.bz2",
                "boards" : [
                   {
                      "name" : "Keyboardio Model 01"
@@ -455,12 +20,12 @@
                   }
                ],
                "category" : "Contributed",
-               "checksum" : "SHA-256:eb75a6203962f77e9d18add88bdb3d6ddada45135d825a6c40e8bc492db21df7",
+               "checksum" : "SHA-256:fc17b1d9e9c208558d6c0b5507943c6d05ad5742b67161ed3fdbbe45d5bcaf4f",
                "help" : {
                   "online" : "https://community.keyboard.io"
                },
                "name" : "Kaleidoscope keyboards",
-               "size" : 679181,
+               "size" : 767144,
                "toolsDependencies" : [
                   {
                      "name" : "avr-gcc",
@@ -473,8 +38,8 @@
                      "version" : "6.3.0-arduino17"
                   }
                ],
-               "url" : "https://raw.githubusercontent.com/keyboardio/boardsmanager/master/builds//Kaleidoscope-1.99.3.tar.bz2",
-               "version" : "1.99.3"
+               "url" : "https://raw.githubusercontent.com/keyboardio/arduino-kaleidoscope-master/master/builds//Kaleidoscope-2021.08.05182428.tar.bz2",
+               "version" : "2021.08.05182428"
             }
          ],
          "tools" : [],

I can see a few possible culprits:

  • The version number changed from 1.99.3 to 2021.08.05182428. That does look like a valid version number to me, though, and the version parsing logic seems like it can deal with it just fine.
  • The download URL changed (as expected) from Kaleidoscope-1.99.3.tar.bz2 to Kaleidoscope-2021.08.05182428.tar.bz2. Perhaps something in those archives content is wrong? I took a cursory look with kdiff3, and all changes are in the code inside libraries/ - notably, boards.txt and platform.txt are equal, and those are probably the only things the Arduino IDE cares about at that stage.
  • There’s a double slash in the URL: […]/builds//Kaleidoscope-2021.08.05182428.tar.bz2 - maybe that trips something up?

At this point I uploaded the JSON file to my own website so I can experiment with changes. Note that I’d recommend backing up and removing ~/Arduino and ~/.arduino15 for those tests, as there are other issues related to the removal of packages resulting in a very similar NullPointerException.

Fixing the double slash did nothing, unfortunately. Then again, it’s part of the working JSON as well, so that was unlikely to be the culprit.

Next, I tried setting the version to 1.99.9999 and suddenly was able to install the package correctly. I haven’t tried to build it yet since it’s getting late here, but I assume it will work just fine. Here are the changes compared to the original:

--- package_kaleidoscope_master_index.json.1    2021-09-02 21:12:57.881159310 +0200
+++ package_kaleidoscope_master_index.json      2021-09-02 21:09:03.366922818 +0200
@@ -38,8 +38,8 @@
                      "version" : "6.3.0-arduino17"
                   }
                ],
-               "url" : "https://raw.githubusercontent.com/keyboardio/arduino-kaleidoscope-master/master/builds//Kaleidoscope-2021.08.05182428.tar.bz2",
-               "version" : "2021.08.05182428"
+               "url" : "https://raw.githubusercontent.com/keyboardio/arduino-kaleidoscope-master/master/builds/Kaleidoscope-2021.08.05182428.tar.bz2",
+               "version" : "1.99.9999"
             }
          ],
          "tools" : [],

So, let’s look at the version parsing logic again:

    try {
      // Allow x.y-something, assuming x.y.0-something
      // Allow x-something, assuming x.0.0-something
      String version = ver;
      String extra = "";
      String split[] = ver.split("[+-]", 2);
      if (split.length == 2) {
        version = split[0];
        extra = ver.substring(version.length()); // includes separator + or -
      }
      String[] parts = version.split("\\.");
      if (parts.length >= 3) {
        return Optional.of(Version.valueOf(ver));
      }
      if (parts.length == 2) {
        version += ".0";
      }
      if (parts.length == 1) {
        version += ".0.0";
      }
      return Optional.of(Version.valueOf(version + extra));
    } catch (Exception e) {
      return Optional.empty();
    }

At least there, I really don’t see anything which would be happy with 1.99.9999 but failing with 2021.08.05182428… Maybe there’s something I’ve missed which doesn’t like the leading zeroes in the date or something, but that’d be weird too. I have no idea, this is as far as I got.

tl;dr: It’s something about the Arduino IDE not liking the version number. Feel free to use my modified version which installs fine for me, but please note that it’s totally unofficial and - while that would be a very sad thing - I’m not responsible for any kittens being harmed or other problems with it. Check the next post for a better workaround.

1 Like

Another day, another chance of tracking this down - and I think I got to the bottom of it now! Again, tl;dr and a better workaround at the bottom.

Something I didn’t realize yesterday is that the version parsing is actually using Java SemVer to do the actual version parsing here:

import com.github.zafarkhaja.semver.Version;
// [...]
if (parts.length >= 3) {
  return Optional.of(Version.valueOf(ver));
}

so I edited the Arduino sources to stop hiding the real cause of the error by returning an empty version in that case:

diff --git i/arduino-core/src/cc/arduino/contributions/VersionHelper.java w/arduino-core/src/cc/arduino/contributions/VersionHelper.java
index bead8d4..738d4de 100644
--- i/arduino-core/src/cc/arduino/contributions/VersionHelper.java
+++ w/arduino-core/src/cc/arduino/contributions/VersionHelper.java
@@ -39,7 +39,6 @@ public class VersionHelper {
     if (ver == null) {
       return Optional.empty();
     }
-    try {
     // Allow x.y-something, assuming x.y.0-something
     // Allow x-something, assuming x.0.0-something
     String version = ver;
@@ -60,9 +59,6 @@ public class VersionHelper {
       version += ".0.0";
     }
     return Optional.of(Version.valueOf(version + extra));
-    } catch (Exception e) {
-      return Optional.empty();
-    }
   }
 
   public static int compare(String a, String b) {

and this finally reveals the actual traceback:

     [exec] java.lang.RuntimeException: Numeric identifier MUST NOT contain leading zeroes
     [exec] 	at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onUpdatePressed$1(ContributionManagerUI.java:150)
     [exec] 	at java.lang.Thread.run(Thread.java:748)
     [exec] Caused by: Numeric identifier MUST NOT contain leading zeroes
     [exec] 	at com.github.zafarkhaja.semver.VersionParser.checkForLeadingZeroes(VersionParser.java:479)
     [exec] 	at com.github.zafarkhaja.semver.VersionParser.numericIdentifier(VersionParser.java:407)
     [exec] 	at com.github.zafarkhaja.semver.VersionParser.parseVersionCore(VersionParser.java:287)
     [exec] 	at com.github.zafarkhaja.semver.VersionParser.parseValidSemVer(VersionParser.java:255)
     [exec] 	at com.github.zafarkhaja.semver.VersionParser.parseValidSemVer(VersionParser.java:195)
     [exec] 	at com.github.zafarkhaja.semver.Version.valueOf(Version.java:265)
     [exec] 	at cc.arduino.contributions.VersionHelper.valueOf(VersionHelper.java:53)
     [exec] 	at cc.arduino.contributions.DownloadableContribution.getParsedVersion(DownloadableContribution.java:70)
     [exec] 	at cc.arduino.contributions.packages.ui.ContributedPlatformReleases.add(ContributedPlatformReleases.java:66)
     [exec] 	at cc.arduino.contributions.packages.ui.ContributedPlatformReleases.<init>(ContributedPlatformReleases.java:55)
     [exec] 	at cc.arduino.contributions.packages.ui.ContributionIndexTableModel.addContribution(ContributionIndexTableModel.java:132)
     [exec] 	at cc.arduino.contributions.packages.ui.ContributionIndexTableModel.updateContributions(ContributionIndexTableModel.java:67)
     [exec] 	at cc.arduino.contributions.packages.ui.ContributionIndexTableModel.updateIndexFilter(ContributionIndexTableModel.java:58)
     [exec] 	at cc.arduino.contributions.ui.InstallerJDialog.updateIndexFilter(InstallerJDialog.java:293)
     [exec] 	at cc.arduino.contributions.ui.InstallerJDialog$5.actionPerformed(InstallerJDialog.java:341)
     [exec] 	at javax.swing.JComboBox.fireActionEvent(JComboBox.java:1258)
     [exec] 	at javax.swing.JComboBox.setSelectedItem(JComboBox.java:586)
     [exec] 	at javax.swing.JComboBox.setSelectedIndex(JComboBox.java:622)
     [exec] 	at cc.arduino.contributions.packages.ui.ContributionManagerUI.updateUI(ContributionManagerUI.java:116)
     [exec] 	at processing.app.Base$9.onIndexesUpdated(Base.java:1412)
     [exec] 	at cc.arduino.contributions.packages.ui.ContributionManagerUI.lambda$onUpdatePressed$1(ContributionManagerUI.java:144)
     [exec] 	... 1 more

So I was right with my guess above: Numeric identifier MUST NOT contain leading zeroes. This seems weird given that this is somewhat common, yet it’s exactly what the SemVer spec requires (emphasis mine):

  1. A normal version number MUST take the form X.Y.Z where X, Y, and Z are non-negative integers, and MUST NOT contain leading zeroes. X is the major version, Y is the minor version, and Z is the patch version. Each element MUST increase numerically. For instance: 1.9.0 → 1.10.0 → 1.11.0.

Thus, whatever script is generating that JSON is to blame. Looking at the git log, that seems to be Mr. “Automated release publisher <nobody@keyboard.io>” :smiley:.

I can’t find anything in the open source repositories which looks like that script, so I suppose that’s something @obra would need to fix somewhere (I hope it’s okay to ping you, apologies if not).

As a better workaround which doesn’t rely on my web server:

  • Download the master index JSON.
  • Remove all leading zeroes in the version number, e.g. 2021.08.051824282021.8.5182428.
  • Store it somewhere where it doesn’t bother you.
  • Add a file: URL to the edited file in the Arduino preferences, e.g. file:///home/yourusername/misc/package_kaleidoscope_master_index.json. On Windows, I suppose something like file:///c:/Users/yourusername/... should work. On macOS, probably file:///Users/yourusername/.... Those two are purely from memory, though.
  • Repeat the downloading/editing once in a while to update to new releases.

There still are errors when trying to build a sketch because the individual libraries carry the same version number:

Invalid version '2021.08.05182428' for library in: /home/florian/.arduino15/packages/kaleidoscope_nightly/hardware/avr/2021.8.5182428/libraries/Kaleidoscope-DynamicMacros
Invalid version '2021.08.05182428' for library in: /home/florian/.arduino15/packages/kaleidoscope_nightly/hardware/avr/2021.8.5182428/libraries/Kaleidoscope-LEDEffect-Rainbow
Invalid version '2021.08.05182428' for library in: /home/florian/.arduino15/packages/kaleidoscope_nightly/hardware/avr/2021.8.5182428/libraries/Kaleidoscope-LED-Palette-Theme
[etc.]

but it looks like those can safely be ignored :crossed_fingers:

2 Likes

Thank you for the excellent spelunking. I’ve just pushed afa32435 which should fix this.

3 Likes

Can confirm - it does fixed the problem. Thank you @ The-Compiler - you are great!

Thank you everyone. All issues are now resolved.