I’ve been playing around with alternate ideas for storing layer data, and thinking through the implications. I’m finding that one thing really complicates the issue, and I think it’s of dubious value (and can be confusing to new users), and that is the concept of transparent keys on the layer stack.
So, I’m throwing the question out here for discussion: Is key transparency really worthwhile? It does have some advantages:
Keymap legibility — it’s easy to see which keys are important on a partial layer with the ___ macro.
It lets a user define a partial layer that can be used over several alternative base layers (e.g. a number pad on one side, leaving the rest as QWERTY, Dvorak, or Colemak, depending on while base layers are active).
One thing it doesn’t do is save space in progmem (or eeprom, if that’s where the keymap is), because Key_Transparent takes up two bytes, just like any other key.
Some people have now had a while with their keyboards; I’d like to know if people have found transparent keys to be genuinely useful in practice, or if their keymaps would have worked just as well without that feature.
It may turn out that significant program size reductions could be made by simplifying key value lookups, but only by dispensing with key transparency. Before I try any experiments with it, I’d like to know how valuable that feature is to people.
Good insight. I found myself not using them outside of the thumb keys / modifiers that I do want to persist in every layer. In every layer I put XXX wherever a new key was not defined.
I’m not sure about that. The main part of lookup is Layer.updateActiveLayers(), with its 154 bytes. Axing transparency would make this one irrelevant, but that’s about it. We could get rid of the caches (I’d still keep them, because they make lookup faster, especially when looking up from EEPROM), and save a handful of more bytes (and a lot of memory, mind you).
For EEPROM lookup, it is pretty much essential. Out of the factory, the EEPROM is filled with 0xff bytes, so when enabling the EEPROM-based lookup with PROGMEM fallback (which is the recommended thing to do, unless you have a way to fill up your EEPROM first), the EEPROM-override will be conveniently transparent by default. If we axe transparency, we’ll have to special-case this scenario, even though I’d expect it to be a pretty darn common one.
While my keymap on the Model01 would work just as well without transparency, that’s not the case for the Shortcut (or for the Raise): I have a QWERTY layer on those two (which I do not have on the Model01) for gaming purposes, a Dvorak layer for daily typing needs, and a numpad layer which I want to use with both: with Dvorak, for number entry; with QWERTY, bound to less frequently used actions. Without transparency, I’d need two numpad layers, wasting a lot of space, and duplicating the layer.
Perhaps we should not emphasize transparency all that much in the factory firmware then, and use it less there? That way, users aren’t introduced to the concept that early, but we’d still have the feature available for those who want or need it.
In addition to updateActiveLayers() itself, it would make the activeLayers[] cache (64 bytes of RAM on a Model01) irrelevant. But that’s just on the current version of Kaleidoscope, with only two different types of layers, and a lookup of every key on every scan cycle. I’m trying to experiment with changes to both of those things.
Starting from a discussion on IRC a while back regarding the Raise having (potentially many) game-specific layers, where I suggested a sparse layer map with just a “diff”, and someone (I think it was you, @algernon) mentioned the idea of what I’ve been thinking of as “shallow” layers, with just a one-byte keycode per key, without flags, I came up with the following independent axes for layer types:
PROGMEM vs EEPROM (and maybe even temporary SRAM layers?)
Full (one entry / key) vs Sparse (one {key_id,Key} pair per difference, plus a reference layer number)
Deep (Key entries) vs Shallow (keycode entries)
Altogether, that would be eight different types of layers, each with particular advantages and drawbacks. Each of them might be a good idea, but all of them together in one sketch is likely not a great idea. And each of those different layer types would have its own lookup() function.
So, to challenge one of your assumptions, a layer could be implemented to have a “reference” or “base” layer, such that if a key is transparent – or, in the case of sparse layers, absent – the key is looked up on that reference layer (probably non-recursively, to prevent infinite loops). This reference layer lookup could be done regardless of the state (active or inactive) of the reference layer itself (again, this makes the most sense for sparse layers). So, if you’ve got a layer in EEPROM, and Key_Transparent results in a lookup on a fallback reference layer in PROGMEM, that should deal with the initial EEPROM state issue without requiring the full key transparency behaviour.
The other, deeper change that I’m planning to experiment with is a complete rewrite of the main loop, which would result in doing lookups only when keys toggle on. This isn’t really so different from what happens now, were getKey() is called by updateLiveCompositeKeymap() whenever a key toggles on. In my system, I would still use one “cache”, which would be the equivalent of liveCompositeKeymap, but that would be the only one.
Do you use the keys from both base layers while the numpad layer is active? This is the essential question I’m getting at to determine if it’s really important to people to have the current version of key transparency.
Do you happen to know how long an EEPROM read takes, as opposed to a PROGMEM read or an SRAM read? I’ve been searching, and all I’ve found so far is info on EEPROM writes, which are, of course, very slow, but not really relevant.
(Will reply to the other points later, just a quick one now, before leaving work)
Yes, I do.
When gaming, I use both the numpad and the left half of QWERTY at the same time quite often. Like, I hold W on the left half, and mash some keys on the numpad on the right. I do this all the time.
When typing, I use the modifiers and punctuation I have on the left, and the numbers on the right. Punctuation is different between QWERTY and Dvorak, so I can’t get away with duplicating them on the numpad layer, as they’d conflict with letter keys on QWERTY.
IIRC, an EEPROM read is considerably more instructions. Insignificant when doing once, but adds up quickly, especially if we do it every cycle. If we don’t, then it doesn’t matter. I can’t recall the numbers, unfortunately, but it was a measurable amount, as far as I remember.
What if I want EEPROM to override the same layer in PROGMEM? As in, layer one in EEPROM would be overrides for layer one in PROGMEM? This could be accomplished by the EEPROM layer being a different layer, but that complicates a few things. For example, for an end-user, it wouldn’t immediately make sense. Probably even less so than the status quo. And then come layer keys… in PROGMEM, it may be ShiftToLayer(2), but in EEPROM, it must be, say, ShiftToLayer(5), because we have a three-layer offset between the two. That’s going to end up incredibly confusing.
So the current EEPROM-overrides-PROGMEM thing will need to stay…
My lack of familiarity with the EEPROM-Keymap plugin is going to show here, but that question seems totally nonsensical to me:
My intention is to make the Layer type a virtual base class. Kaleidoscope would have an array of Layer pointers, and the index in that array would be the layer number. There wouldn’t be any kind of offset for EEPROM layers; each Layer object would use its own lookup functions, which would be virtual functions in the base Layer class. Sparse layers (and/or layers with transparent keys with a reference layer) would inherit some of their values from a reference layer. No two layers in my system could share the same index number.
With this system, it’s even possible to change the order of the layers at run-time, though that’s probably not a great idea because it would mess with layer-shifting keys.
The EEPROM-Keymap defines a separate set of keymaps, that are stored in EEPROM, that exist in parallel to the keymaps in PROGMEM.
The transparency thing comes in to play with the plugin because you can have, say, layer 1 in EEPROM just override one key, leave the rest transparent, and it will act like layer 1 from PROGMEM but with that one key changed. That is, EEPROM layers are not another separate set of keymap layers, but can be thought of as modifying or supplementing the PROGMEM layers.
This implies that when using the plugin, the PROGMEM vs EEPROM layers aren’t distinct – layer 1 just means layer 1 from EEPROM, falling back to layer 1 of PROGMEM.
