The problem
A few times now, people have complained that layer switching doesn’t work (as expected). Usually, this is because they don’t understand how the layer stack works — they either expect just one layer to be active at a time, or that a layer activation will always result in the target layer coming to the top.
I’ve had a few ideas about how to ameliorate the confusion for new users, and I want to share them and see what everyone else thinks.
Layer key macro names
I’ve brought this up before, but I think it’s worth a second round of discussion — the names of macros that are used to define layer-switching keys can be misleading to some users, and I think they may be contributing to the incorrect assumptions that some new users make.
First, there’s ShiftToLayer(N), which suggests that the keymap is currently at one layer, and holding that key down will move the keymap to layer N.
Then, there’s LockLayer(N) & UnlockLayer(N), which like “flammable” & “inflammable” have the same meaning. While not as suggestive of the idea that only one layer is active at a time, when paired with ShiftToLayer(), they don’t contradict that notion, either. A more descriptive name would be ToggleLayer(). To me, that mildly connotes the actual functionality of the layer stack, though I don’t know to what extent that would assist new users.
We could use the names LayerShift() & LayerLock() (à la CapsLock), which seem more consistent to me than the current names, and might be less suggestive of different behaviour than actually exists, but I would expect further explanation to still be necessary for some new users to understand the layer stack.
Possible behaviour changes
I got an idea for a deeper solution to this problem from the fact that, in addition to the bitfield that stores the current layer state, Kaleidoscope also caches the index of the top active layer. It wouldn’t be difficult to change the way that a ShiftToLayer() key functions so that it would actually do what many people expect: shift to the target layer.
The layer shift algorithm would work like this: when a layer shift key, the target layer becomes the top active layer (highestLayer = N), and key lookups begin with that layer — but its state bit in layerState doesn’t get changed, and we don’t bother to check that bit when doing the first layer lookup.
This would mean that holding down ShiftToLayer(N) would actually do what’s generally expected, even if layer N is below the current top active layer. It would also mean that only one layer could be shifted to at a time. This would be a limitation that the current system doesn’t have, but it seems like a rather unusual thing to do, anyway, even with OneShot layers (if anyone is in the habit of using two simultaneous layer-shift keys, I’d like to hear about it!).
On release of a ShiftToLayer() key, highestLayer would once again be set from the active bits in layerState, and everything would return to normal. This would mean you could have a ShiftToLayer(1) key on layer 3 that would let you temporarily access keys on layer 1 without deactivating layer 3. I think this could be genuinely useful, even to people who already have a solid grasp of the layer stack and transparency, and the loss of the ability to simultaneously have multiple shift-activated layers seems like a small price to pay.
We would have to deal with two layer-shift keys being pressed concurrently, of course, and I would argue that the last one to be pressed should win. I think this could be accomplished by having pressed layer-shift keys check and see if highestLayer is different to their own layer, and if that’s the case, then mask themselves. Then there’s the question of what to do if a layer lock is activated while a layer shift key is held. I don’t think that’s so obvious, but I think the layer-shift should take precedence until it’s released. I’m imagining a layer-control layer, where the number keys are all layer toggles, accessed by a layer-shift key. I wouldn’t want the shift to this layer to be canceled by pressing one of those layer toggle keys, because I might want to use more than one before I’m done. (I’m also imagining using LEDs to indicate which layer are active…)
Last, maybe it would be worthwhile having a third type of layer-switch key defined, with another offset above LAYER_SHIFT_OFFSET, for a MoveToLayer(N) key type, that would result in a call to Layer.move(N), which is what at least a few people expect LockLayer(N) to do.
