Using a key to both SHIFT_TO_LAYER and produce a key signal

I am using the neo layout under linux and use the xmodmap layout for Layer 4. The problem is that some applications (eg written with java swing) do not register control keys like arrows, enter etc when input with a (software-level) modifier.

What I would like therefore is to have the key I use as modifier to trigger both a shift to a layer (“firmware modifier”) and still send a keycode to the os (“software modifier”). That way I can input the control characters on Layer 4 by adjusting the firmware layout but still use the key as a software modifier for other characters and combination with other modifiers.