Layer-switching confusion

The problem

A few times now, people have complained that layer switching doesn’t work (as expected). Usually, this is because they don’t understand how the layer stack works — they either expect just one layer to be active at a time, or that a layer activation will always result in the target layer coming to the top.

I’ve had a few ideas about how to ameliorate the confusion for new users, and I want to share them and see what everyone else thinks.

Layer key macro names

I’ve brought this up before, but I think it’s worth a second round of discussion — the names of macros that are used to define layer-switching keys can be misleading to some users, and I think they may be contributing to the incorrect assumptions that some new users make.

First, there’s ShiftToLayer(N), which suggests that the keymap is currently at one layer, and holding that key down will move the keymap to layer N.

Then, there’s LockLayer(N) & UnlockLayer(N), which like “flammable” & “inflammable” have the same meaning. While not as suggestive of the idea that only one layer is active at a time, when paired with ShiftToLayer(), they don’t contradict that notion, either. A more descriptive name would be ToggleLayer(). To me, that mildly connotes the actual functionality of the layer stack, though I don’t know to what extent that would assist new users.

We could use the names LayerShift() & LayerLock() (à la CapsLock), which seem more consistent to me than the current names, and might be less suggestive of different behaviour than actually exists, but I would expect further explanation to still be necessary for some new users to understand the layer stack.

Possible behaviour changes

I got an idea for a deeper solution to this problem from the fact that, in addition to the bitfield that stores the current layer state, Kaleidoscope also caches the index of the top active layer. It wouldn’t be difficult to change the way that a ShiftToLayer() key functions so that it would actually do what many people expect: shift to the target layer.

The layer shift algorithm would work like this: when a layer shift key, the target layer becomes the top active layer (highestLayer = N), and key lookups begin with that layer — but its state bit in layerState doesn’t get changed, and we don’t bother to check that bit when doing the first layer lookup.

This would mean that holding down ShiftToLayer(N) would actually do what’s generally expected, even if layer N is below the current top active layer. It would also mean that only one layer could be shifted to at a time. This would be a limitation that the current system doesn’t have, but it seems like a rather unusual thing to do, anyway, even with OneShot layers (if anyone is in the habit of using two simultaneous layer-shift keys, I’d like to hear about it!).

On release of a ShiftToLayer() key, highestLayer would once again be set from the active bits in layerState, and everything would return to normal. This would mean you could have a ShiftToLayer(1) key on layer 3 that would let you temporarily access keys on layer 1 without deactivating layer 3. I think this could be genuinely useful, even to people who already have a solid grasp of the layer stack and transparency, and the loss of the ability to simultaneously have multiple shift-activated layers seems like a small price to pay.

We would have to deal with two layer-shift keys being pressed concurrently, of course, and I would argue that the last one to be pressed should win. I think this could be accomplished by having pressed layer-shift keys check and see if highestLayer is different to their own layer, and if that’s the case, then mask themselves. Then there’s the question of what to do if a layer lock is activated while a layer shift key is held. I don’t think that’s so obvious, but I think the layer-shift should take precedence until it’s released. I’m imagining a layer-control layer, where the number keys are all layer toggles, accessed by a layer-shift key. I wouldn’t want the shift to this layer to be canceled by pressing one of those layer toggle keys, because I might want to use more than one before I’m done. (I’m also imagining using LEDs to indicate which layer are active…)

Last, maybe it would be worthwhile having a third type of layer-switch key defined, with another offset above LAYER_SHIFT_OFFSET, for a MoveToLayer(N) key type, that would result in a call to Layer.move(N), which is what at least a few people expect LockLayer(N) to do.


I forgot to mention one more side effect of my idea for the behaviour change in layer shifting:

Right now, if I tap LockLayer(2), then press and release ShiftToLayer(2), layer 2 gets deactivated. This is unlike an interaction between CapsLock and Shift, where pressing and releasing the latter doesn’t cancel the former (though the effect of holding Shift while CapsLock is active is platform-dependent, so the analogy. With my suggested change, layer 2 could remain active (deactivating it would be an additional step when a ShiftToLayer() key is released). Whether or not this is desirable is a matter of personal preference, of course. I’d rather not have a released layer-shift cancel a locked layer, because that would allow the following configuration to work:

  • On the base layer, both palm keys are ShiftToLayer(A)
  • On layer A, both palm keys are LockLayer(A)

If I want to shift to layer A temporarily, I hold down either palm key. If I want to stay there, I tap the other palm key. When I’m done with layer A, I tap either palm key, and it’s deactivated.

With the current system, the LockLayer(A) keys are non-functional, because as soon as the ShiftToLayer(A) key is released, layer A gets deactivated.

I’ve created a PR against Kaleidoscope, if anyone is interested in testing out my proposed change to the layer-switching behaviour.

I ended up checking the value of Layer.isOn(LAYERNAME) in my own layout switching macro, because I wanted the lights to change based on that state. And since I was already using that, I used Layer.on() and instead of LockLayer() anyway.

Presumably you could just have ShiftToLayer be a nonop if the target layer is already on, no?

In my proposed system, that would only be (effectively) a noop if the target layer was the top active layer. If a layer higher than the target layer was on, it would be hidden for the duration of the layer shift.

I think I didn’t quite understand the whole scenario when I made my comment. Now I think I fully understand, but this is still confusing. Is this summary correct?

CURRENT: the topmost active layer (highestLayer) is determined by selecting the layer with the highest index with layerState “on.” Using LockLayer(TARGETLAYER) or ShiftToLayer(TARGETLAYER) changes the value of layerState(TARGETLAYER). This has immediately visible effect if TARGETLAYER == highestLayer, but if TARGETLAYER has a lower index than highestLayer the effect is hidden until layers with higher indexes are made inactive.

PROPOSED: ShiftToLayer() would not change the stored value of layerState(TARGETLAYER) and would take precedence in setting highestLayer for the duration the shift modifier is in effect. When the shift modifier is released, highestLayer would revert to being set based on the highest layer index with layerState “on.”

Do I have that right?


Thanks for the rephrasing; you’ve got it exactly right, with one minor additional detail. In the current version, activating a target layer with an index lower than highestLayer can affect the live keymap, because transparent keys on the higher layers can allow keys from the target layer to “shine through”.

If it weren’t for the resolution of transparent keys I think I’d be 100% on board with this change. Making “shift” higher priority than “lock” seems like a consistent user experience.

I’m not qualified to address how this change might affect performance and response time (nor do I have the wherewithal to test it myself) but the proposed change does increase the complexity of documenting the behavior. Right now the logic is always “active layer with the highest index wins.” If the highest index layer has a transparent key, it just goes down the list in order of index. That’s pretty easy to understand and document.

The proposed change, however, increases the complexity to the extent I start to wonder if it’s worth the additional flexibility. For instance, assume layer 1 has a transparent key that the user expects to pass through to layer 0, but layer 2 is currently active. When the user shifts to layer 1, the pass through here would be to layer 2 (the active layer with the highest index), not to layer 0 (the layer that transparent keys in layer 1 would normally expose). That feels wrong to me, but the thing that feels right (ignoring all active layers with higher index than the shifted layer) also seems a bit counterintuitive. Personally I’d expect my transparent keys to pass through to layer 0 basically 100% of the time, but I’m not really doing the sort of compound stacking it seems like you’re targeting here.

In conclusion: ¯\(ツ)

1 Like

I didn’t describe the behaviour clearly enough, then. In your scenario, starting with layers 0 & 2 active, if the user presses a ShiftToLayer(1) key, any transparent key on layer 1 will look down the stack to layer 0, not layer 2; all mappings on layer 2 will be ignored until the shift is released, so the mapping will be the same as if layers 0 & 1 were active, but not layer 2.

Well then, it does what I expect, my bias is confirmed, and therefore there is no further interpretation that warrants discussion. But seriously, I’d love to hear if there’s anybody who would expect layer 2 to pass through in this usage. I’d expect that to be a non-zero number of people, but I really have no idea. I think you and I are on the same page.

But also to a point made in your original post, I agree that LockLayer and UnlockLayer are poorly named. I think instead of being a boolean toggle, LockLayer should always result in layerState on, and I’d argue that it should also result in the deactivation of higher layers. I think if somebody wants to insert an intermediate layer without unlocking a higher layer (to take advantage of transparency tricks) there should be a distinct command for it. (I mean, I guess you could use Layer.on directly for that usage, because it seems uncommon and a fair price to pay for that level of customization, but maybe an InsertLayer() function might make reasonable sense).

To wit, that would mean:

  • ShiftToLayer sets the active layer and temporarily ignores layers with higher indexes when resolving transparent keys;
  • LockLayer activates the selected layer (if it is inactive), and deactivates layers with higher indexes;
  • LockLayer on an active layer stops deactivating that layer;
  • UnlockLayer unlocks the selected layer and ignores higher layers;
  • All of the above instructions leave the activation status of layers with lower indexes untouched.

Then we could talk about things like ToggleLayer (to do what LockLayer actually does now) and maybe an InsertLayer (for transparency tricks). Does LockLayer() currently return the state of the target layer, or is it necessary to check Layer.isOn after use?

1 Like

Here’s another thing that’s a bit confusing: LockLayer() and ShiftToLayer() look like function calls, but they’re not. Those two preprocessor macros define two-byte Key objects with the correct flags set so that Kaleidoscope recognizes them as keys that should result in layer changes. Luckily, there is enough space available to add more layer-switching key types.

I would suggest just three:

  • ShiftToLayer() (as previously discussed)
  • ToggleLayer() (a new name for LockLayer())
  • MoveToLayer() (like ShifToLayer(), but permanently activates the target layer, and deactivates layers above)

I’m doing just that: my palm keys are one-shot layer keys, and both activated gets me to an empty layer, which I use to stop magic combos emitting random garbage. In other words, I have a number of magic combos, which all involve both palm keys, and a number of others. To stop the combo from inputting random keys while I’m setting up the hold of all the keys needed for it, the palm keys were set up to get me to an empty layer instead.

It is a bit of a hack, and an edge case, mind you.

I like that name too.


What would be best, in my opinion, is to lift out layer handling into a plugin. I quite like the way it works now, but I also understand that it is sometimes - and perhaps too often - incredibly confusing for those who aren’t used to it. It also has a few perhaps surprising limitations (like not being able to shift to a lower layer). Having an alternative - which can even be the one shipping with the factory firmware - would be great. But I’d love to have the current one around too.


With my proposed changes, you’d still be able to do it, but with a slightly modified gesture: tap one palm key, hold the other, then hold the first one, and you’ll be on that empty layer, with both palm keys held. Not as fast as your usual MagicCombo-activation hack.

Or, you could use a MagicCombo of the two palm keys together to get you to the MagicCombo layer (I’m guessing, since I’ve never used MagicCombo myself).

…or did you mean that your MagicCombos don’t use the palm keys? I’ll have to look at your sketch again…

Hmmm. That’s a good idea, would likely work too. Will try it later, thanks!

They do. All my magic combos are PALMS + stuff, where PALMS get me to the empty layer.

Do masked keys interfere with detection for MagicCombos? It’s been a while since I looked at the code, and I’ve never used it, but I seem to think it’s detection mechanism might side-step the key masking.

No, masking does not interfere, because MagicCombo looks at the scanner state, and does so in the loop hook, and as such is unaffected by masking. You remember correctly, it side-steps key masking (intentionally, I’ll add).


In that case, I think these changes wouldn’t interfere with your hack at all.

1 Like

That sounds great, will try it at home as soon as I can!

I do agree, by the way, that it would be good to move layer-switching to a plugin (in nearly is already, in a sense).