In Chrysalis I have the space key with secondary action “layer shift” to layer 1. There I have a key defined with “Alt-Tab”. This does work in principle, but the “Alt” does not stick and therefore making the function useless, because the window overview is not shown. This is how it should look like:
Similar with other combos like “Ctrl-Shift-Tab”. Other keyboard software like UHK’s Agent works as expected.
I don’t understand how that should work. If you have a single key that generates Alt-Tab then pressing that key results in pressing Alt and Tab and releasing that key releases Tab and Alt. That is exactly what I would expect. To support your use case you will likely have to implement something more complicated (maybe look into how UHK does it).
With the UHK software I define a secondary role exact in the same way like in Chrysalis. It makes absolute sense that a modifier acts as being hold when the mod-key is held which triggers whatever mod-key or mod-key combination (Ctrl, Alt, Win, Shift). This is something which needs to be implemented in the firmware and/ or configuration software. Alt-Tab is one prominent example where this would be very beneficial, but there are other key combos like this, for example Ctrl-Tab (next tab in browser) – also displaying the overview of possible tabs.
This is one example of my frustration with the model 100 where it does not work like it would be logical/ good. 
So it seems I am out of luck here.
Unless I misunderstand what you mean, I disagree.
If I press and hold a layer switching key and then press and release another key that enters a modifier (e.g. Alt) and another key it would be absolutely surprising and annoying for nearly all use cases if the modifier key (Alt in this example) would still be held. If I have one key in that layer assigned to Alt-Tab for switching tabs and another one in the same layer as F4 I certainly would not want the Alt to be still active when pressing F4.
I don’t know UHK and don’t know how it works (and what purpose it fills), but I am rather sure that it does not work that way. You are likely mixing things up when comparing UHK and Kaleidoscope.
I understand your use case and that it could be handy to you. But it is clear to me that this is rather special and will certainly need a bit of configuration and maybe even coding.
After thinking about this further I understand your point. It is indeed not as self explaining as I thought. Actually I have the F-keys on the same layer like my Alt-Tab combo. I never had any problems with this behaviour of the UHK. Trying your example with it I first noted that you always release the Alt-Tab combo – both with the original keys and as well with a mod-combo. Second when I want to provoke the “error” you mention as an example the following happens: The first keypress of the new key (4) quits the Alt-hold state, a second keypress then triggers F4. So IMO that situation is handled gracefully as well, but not a practical problem because one always releases all keys when firing a shortcut with a key combo. 
Edit:
from the UHK reference manual, the following applies to non-base layers and solves exactly the problem I described.
set stickyModifiers {never|smart|always} globally turns on or off sticky modifiers. This affects only standard scancode actions. Macro actions (both gui and command ones) are always nonsticky, unless sticky flag is included in tapKey|holdKey|pressKey commands. Default value is smart, which is the official behaviour - i.e., <alt/ctrl/gui> + <tab/arrows> are sticky.I find that smart indeed.
There is an example plugin named AppSwitcher which you might find helpful.
Thanks for the tip! I did not manage to make a custom firmware version (steps outlined in the docs did not work) and I have given up to try building custom firmware.