You say it stays lit as if it’s being held, which suggests the ActiveModColor plugin might be involved. There’s a test mode you can activate which should make it clear whether it’s a software or hardware problem. More info here: Kaleidoscope-Model01-TestMode documentation?
On the other hand, I’ve had situations where host computer hangs somehow caused the keyboard to stop updating LEDs, so the USB bus misbehaving is a possibility.
Newer firmware also has serial port control enabled by default, so it’s remotely possible something on the host side is causing it to light up. I wouldn’t count on this being the reason though.