Research project on rollover (for Qukeys)

(Michael Richters) #1

I’m hoping to find some volunteers to help me with a little research project. A recent conversation on the forum gave me a few ideas for possible improvements to Qukeys. Unbeknownst to me when I started writing that plugin, a substantial number of people roll over from modifiers to other keys – or at least tend to release the two keys simultaneously when chording – rather than holding the modifier for the whole duration of the modified keypress. Qukeys works by deciding which keycode to use based on the order in which the two keys are released (if the subsequent key is released first, the qukey becomes a modifier). This means that “rollover-style chorders” often get the wrong keycode when typing, because they tend to release shift (for example) before T when typing T.

There is a system that is intended to help with this, but it hasn’t been working as well as it might, and I now have a new idea that might be better. I have a guess about how keys are often typed in rollover, but I would like to gather some data from real typists in order to test my hypothesis, and perhaps find a good default value for the configuration parameter that would be used. In order to do this, I’ve put together a new plugin: Kaleidoscope-RolloverLogger.

This plugin is a kind of keylogger. It writes one line to the serial port for every keypress, reporting the time it toggled on, the time it toggled off, and whether or not that key was a modifier (including layer shift keys). It does not (by default) log actual characters typed, nor keyswitch coordinates, so it should be extremely difficult for anyone to extract from such a log the actual input that was typed.

If you are interested in contributing some data, feel free to add the plugin to your sketch briefly (I strongly encourage you to remove it as soon as you’re done), set up whatever system for piping the serial port output to a file you find convenient, review the file, and send it to me: gedankenexperimenter@gmail.com. I’m happy to help anyone who has trouble setting up something to record the output.

There is more detailed information about the plugin in its README file, but I want to emphasize one point about it here: It is a keylogger. Please don’t type anything sensitive (e.g. passwords) while it’s installed in the firmware of your keyboard. I’ve done what I can to make the default output as useless as possible for malicious purposes, but RolloverLogger does have modes that output keyswitch coordinates and even key values to the serial port (feel free to use that functionality, if you have a use for it). I would not leave it installed in my keyboard’s firmware more than an hour or so at a time, and I would limit myself to doing things like typing practice in that time.

2 Likes
(Noseglasses) #2

I had to provide the VID and PID info to make capturing serial output work. For the Model01, e.g.

cat $(Kaleidoscope/bin/find-device-port-linux-udev 1209 2301) | tee /tmp/RolloverLogger.log
1 Like
(Michael Richters) #3

Oops! I forgot that those scripts aren’t quite the same as the the one that I use for debugging. Danke!

(Michael Richters) #4

I didn’t find the time yesterday to elaborate on my idea for the possible improvement(s) to Qukeys that constitute the raison d’etre of this project, so I’ll try now.

The Qukeys rollover problem

When I first wrote Qukeys, I assumed that people ~always hold modifiers for the full duration of the modified keypress (as I do). In other words, to send ctrl + S, I would press ctrl, press S, release S, then release ctrl. This is how Qukeys determines which state a qukey ends up in – by delaying both keypress events until one of them is released.

I then learned that my assumption was faulty, and there are plenty of people who frequently roll over from modifiers to other keys (i.e. press ctrl, press S, release ctrl, release S). Some people tend to release the two keys simultaneously, which results in a race condition. My solution was to introduce a configuration parameter, allowing the users to select a “grace period” for qukey release, so that if the S in the above example was released just a few milliseconds after the modifier, it would be treated as if it was released first.

That grace period idea helped, but it also made it possible for very fast typists to get unintended modifiers if they set up home-row qukeys. Typos from unintended letter keys can be frustrating, but unintended modifier keys have a much greater potential for destruction, so I always caution people to keep the release delay parameter set fairly low, and some users haven’t been able to find a setting that works well enough.

A new solution (or two)

It occurred to me after a recent discussion that I was not using all of the input data available, and that I might be able to do better. In a rollover situation, there are four points in time that can be measured:

  1. qukey press time
  2. other press time
  3. qukey release time
  4. other release time

…but I was only using two of them (the two releases) to decide whether the qukey should take on its primary or alternate Key value. In general, my goal with this research project is to get a data set showing how people use modifiers in combination with other keys – particularly when they roll over, and try to find the best algorithm to use all the available information to produce the intended output.

Specifically, I have one idea that I want to test, using just three of the timestamps above. This idea is to compare the duration of the overlap (between points 2 & 3, when both keys are held) to be at least x percent of the total duration of the second key press (between points 2 & 4). This would make the effective release delay dynamic, so it would be longer or shorter depending on how long the two keys overlap. I’m hoping that this will produce better results than the fixed-length release delay, and as a side benefit, it’s actually simpler in the code.

Of course, that only uses three of the timestamps available, and it’s possible that, for people who regularly roll over from modifiers (rather than those that release the two keys ~simultaneously), it might not provide a clear distinction between typing overlap and modifier use. In that case, there might still be a pattern that can be used based on the relative timing of the two key presses, or the duration that the first one is held, as well. And for people who really do roll over when using modifiers in the same way they do when typing normally, there still might be one possible distinguishing factor I could find by looking at the key press(es) preceding the modifier – I’m guessing that it’s more likely to have a gap before pressing a modifier key than a printable character key, though that would probably be somewhat less true of qukeys on the home row, especially ones with an alternate Key of shift.

A related problem

I’ve got one other Qukeys improvement in the works, which is related. A few times, people have reported rollover-related errors when using qukeys for what would normally be modifier keys (e.g. the thumb keys on the Model01). In these cases, I have usually recommended SpaceCadet instead, because it is intended for the purpose (using what is normally a modifier as a printable character if it is tapped alone), whereas Qukeys is the reverse (using printable character keys as modifiers if chorded). However, as a result of the release delay conversation, it occurred to me that a very small change would allow Qukeys to do both.

The idea is that if a qukey’s primary Key value is a modifier (including layer shift keys), it will be treated like a SpaceCadet key instead. It will always produce that modifier, unless it is tapped on its own. In other words, if there’s any rollover from a shift / X qukey, it will be shift, but for a X / shift qukey, its value will depend on release order and timing.

This might be useful because Qukeys offers more flexibility than SpaceCadet in defining which keys are affected. With SpaceCadet, if Key_LeftShift produces ( on tap, every Key_LeftShift will do so, regardless of where it is in the keymap. Qukeys specifies the exact coordinates of its keys, so you could have two different Key_LeftShift keys (probably on different layers), one of which has an alternate key value, and one that doesn’t.


One last note: If I can get a variety of contributors, I might even be able to produce a script that can be run on a future version of the data file produced by RolloverLogger that would recommend certain settings for the users to minimize Qukeys errors.

Again, I will be very grateful to anyone who helps me out by contributing some typing data so I will be able to make some better choices in the design of Qukeys. Even if you’re not a Qukeys user, it could still be very helpful to get typing rollover data.

Unexpected characters using Qukeys
(Michael Richters) #5

I implemented my new Qukeys variable rollover delay protocol in Kaleidoglyph, and my initial impression is that it might work even better than I hoped, even with a minimum overlap requirement of only 50%. I’d still like to get some real-life rollover data, but I’ll port my changes to Kaleidoscope when I get the time, so other people can give it a try.

(David Aue) #6

I love qukeys modifiers on the home row but this is a big problem for me.

I often get two characters when I wanted a shifted char like: js where j has a shift modifier and I wanted S

I also get misapplication in the other direction like: jt where I tried to type ‘just’ but the ‘us’ part turned into a ctrl+u because ‘s’ has a ctrl modifier on it. I what’s interesting here is I’m certain I’m hitting the qukey ‘s’ after ‘u’ but it still applies.

I also have these settings:
Qukeys.setTimeout(250);
Qukeys.setReleaseDelay(20);

I have installed your logger but I’m not sure what to collect for you. Just a lot of normal typing, or short logs of specific problems. Also do you know a way pipe a com port to a file on Windows?
type COM3: >> kb.log
doesn’t work even though COM3 is legit.

(Michael Richters) #7

This is exactly the problem that I think I’ve got a better solution for.

The fact that you also get this problem (I suspect that what you’re getting is ctrl + s, not ctrl + u) is not promising. If you get errors in both directions, it might be impossible with any algorithm to reliably get the intended output every time for you. Still, I do hope that my newer technique will at least help.

Unfortunately, I haven’t got a Windows machine to experiment with. I’m hoping someone else can provide a reasonably simple way to capture serial port output on Windows. I’ll try to come up with something tomorrow, though.

Mostly what I’m trying to capture is normal typing. And the more I think about it, the less hopeful I am that it’s going to provide all the information I want, unless you’re willing to send me data collected in unsafe-mode, especially if you’re using qukeys on the home row, which might show a different usage pattern than dedicated modifier keys. It doesn’t really matter if the data shows errors, as long as I’m able to discern whether a modifier or a non-modifier was intended.

Knowing that someone is using home-row qukeys and experiencing these problems is a great motivator for me to get my new algorithm ported from Kaleidoglyph to Kaleidoscope. I’ll do that soon, regardless.

(David Aue) #8

Great. I can use the serial monitor in the arduino IDE for logging and I’ll turn on unsafe and send you some logs.

I also think my second case may really be me hitting the keys in the other order which means I just need to improve my typing.

(Michael Richters) #9

Upon re-reading your description, I think this is the most likely explanation. If you’re really pressing the keys in the correct order (U before S), there should be no way to end up with the output you described, and I have never seen that bug before. Either way, what I suggested above is complete nonsense.

As for the Arduino serial monitor, I think I remember finding some problem with it, possibly that it only allowed me to select one screenful of text at a time, making it extremely tedious to copy and paste the text to a file.

#10

@merlin Have you seen this study on keystroke analysis?

Observations on Fyping from 136 Million Keystrokes

2 Likes
(Michael Richters) #11

No, I had not found that study yet – thanks, @kejadlen!

I’m going to inspect their data when I get a chance, but I suspect that they did not actually capture what I need for the analysis I want to do, which is to look for different patterns between rollover from one letter key to another vs rollover from a modifier to a letter. If they did actually capture that, it will be very helpful.

(Michael Richters) #12

I am delighted to discover that my guess was wrong; the data they collected does include modifiers (shift, at least), so I will be able to use it! Thanks again!