Improve drawing / mouse gestures to allow for classic Opera-style mouse gestures

Feature Requests
, ,
1

Hi, I'm the developer of an old mouse gestures app called xGestures. It allows adding the kind of mouse gestures pioneered by the classic Opera browser and rocker gestures anywhere in macOS.

I made the app in my spare time back in 2004 after switching from Windows to mac. I had become accustomed to using StrokeIt in Windows and found that I couldn't live without system-wide mouse gestures any longer. And while for the most part I'm not actively developing xGestures any longer, I continue to maintain it to keep it running on modern versions of macOS, because as far as I'm aware, it's still the only app that allows for this style of mouse gestures, and I still can't live without them.

To be specific, this style of mouse gestures work in the following way:

  • A specific button on the mouse (or perhaps a key on the keyboard) is designated as a gesture button. Typically this is the right mouse button.
  • The user presses and holds the modifier button
  • They move the mouse in a combination of movements, each of which is one of the four cardinal directions (so only left, up, right and down).
  • They release the gesture button

Then depending on the combination of movements, a certain action is performed. It doesn't matter how long each of these movements are, or what kind of image they make on the screen. All that matters is the order of discrete directions performed. So a "left right" gesture is distinct from "right left", "left", or "left right left right" even though they produce the same horizontal line on screen.

The gesture button is still usable as normal for clicking, too. Pressing and releasing the button without performing any movement will still produce a click, though the down and up events for the button would be delayed until after the button is released.

xGestures also recognizes rocker gestures, which is when the user presses and holds down the left mouse button, and then clicks the right button, or vise versa. (This was also in the original Opera web browser.) xGestures only allows this with the left or right mouse buttons, but some other implementations of rocker gestures allow any combination of two mouse buttons.

Currently BetterTouchTool doesn't have a feature for these. Its drawing feature, from my experience, is intended for a different kind of mouse gesture that's focused on recognizing specific shapes.

Of course there's nothing wrong with that style of gesture and it satisfies a different use case, but the advantage of Opera / xGestures' style of gestures is that detection is very accurate and consistent. I almost never have issues with mouse gestures being misdetected.

I'll keep maintaining xGestures as long as I can and is necessary, but over the years I've thought about rewriting it from scratch. (A lot of poorly written code from back in 2004 is still in there!) I haven't done so, though, because a lot of the features I might add to such an app would basically be recreating functionality that's in BetterTouchTool.

I'd love it if BetterTouchTool added this style of mouse gestures. At that point I think I'd be okay with letting xGestures fade away into the annals of software history.

So to make this feature request more discrete and specific amid the above ramblings, I'm hoping that BetterTouchTool might eventually have these specific features:

  • The ability to define a different kind of mouse gesture that's a combination of only left, right, up and down movements
  • Allowing the use of the right mouse button without modifier keys for drawing a gesture
  • Clicking a gesture button, whether it's the right mouse button or otherwise, would still produce a (deferred) click when the button is clicked without performing a gesture
  • Allow for rocker gestures, where the user presses and holds one mouse button and then clicks another

(edit: Just realized that two of those features are already in BTT!)

These sorts of mouse gestures are certainly niche and somewhat anachronistic in the age of trackpads and the Magic Mouse, but xGestures still has a very dedicated user base (which includes myself) of people who can't live without mouse gestures like these.

(Quick aside, Andreas: we've had a few email correspondences over the last several years, one of which was about adding this style of mouse gestures to BTT. But I figured I'd make a public and more official posting here and possibly inspire discussion.)

2

Hi!

I agree that the mouse gestures need to be improved. The current algorithm that I use is based on the $1 Unistroke Recognizer, which has some limitations - especially with straight lines.

My current plan is to switch to a Core ML based recognizer, which should solve most of the issues (and Apple provides good sample code for this, so it's very simple to integrate). This will also work good with straight lines according to my early evaluation.

For rocker gestures: Maybe you can achieve this already with the new "advanced trigger conditions". E.g. you can define that "button 3" should only trigger if also button 4 and 5 are pressed:

image
image1920×1145 128 KB

These conditions are very new, I haven't tested them a lot so far.

3

I didn't realize you had added advanced triggers -- those look quite powerful! I tried them out for left and right mouse button rocker gestures, and it seems to work for holding left and click right, but not holding right and clicking left, partly because BTT doesn't allow the left button to be used without a modifier even when an advanced condition is in place (and perhaps rightly so) (edit: once again I got this wrong), and partly because the context menu triggered by the right-click interferes.

Rocker gestures are tricky since since, similar to mouse gestures, they generally require withholding the normal action of pressing the mouse button until the button is released without performing a gesture (with the exception of the left button of course). And then when a rocker gesture is performed, sometimes it's necessary to post button-up events for the down mouse buttons before its action it executes, otherwise the down mouse buttons might interfere. I think that's how xGestures works, but I'd have to re-examine some rather ancient code to be sure.

4

@Andreas_Hegenberg I'm an extremely intense user of BTT. It's woven into a few hundred python scripts, AppleScripts, and macOS Shortcuts that run across two servers, etc.

xGestures somehow has impeccably good gesture recognition and is an incredibly stable base of code.

It's code that brings mouse gestures up to the "it just works" level of fluid reliability.

Could you possibly collaborate with the xGestures developer below, maybe you could incorporate that code as module with a simple API that interacts with BTT triggers, all under the hood?

Not anachronistic IMHO!

  • I think that the Magic Mouse has ergonomics issues and other problems (hover-finger strain) that have prevented it from being widely adopted. Unlike AirPods and iPhones, most of the market remains using traditional mice with buttons. That's the mainstream.
  • If you're not using the magic mouse, that leaves you with a normal mouse and a trackpad. You have 2 options:
    a) You constantly move your hand back and forth to the track pad to do things like change spaces, pan right or left on anything with horizontal scrollbars, etc.
    b) Make very complex mouse + keyboard triggers in BTT, which means you're now pressing keyboard buttons while using the mouse.

xGestures - it allows you to do a lot of things with just the mouse. You can replicate EVERY gesture you can do on the trackpad by just holding down your right finger and moving the mouse around.

Andreas, it's really superlative work that hasn't been replicated. I've scoured everywhere to find anything comparable and have tried hard to use BTT gestures, but they're ultimately too unreliable to be effortlessly useful like the simple xGestures program is.

It'd be unbelievably great if you could pull the code base into BTT as a module under the hood - if that's something you two would even consider.

1 Like
5

I just want to echo everything that Joe has said. I'm a longtime user of xGestures (with a trackball) and the straight line gestures are the real workhorses. I've recently decided it would be best to transition my gestures over to BTT in order to take advantage of the more flexible advanced trigger options, but was stunned to find out I couldn't use simple straight line gestures. I just don't "get" the BTT style of curves and shapes, but can see that perhaps trackpad users find this more useful.

Incidentally it was Brian who first introduced BTT to me years ago when emailing about a bug. Thanks, Brian for the tip-off, and for maintaining xG for so long, it is amazing work and has improved things for me for so many years.

6

I completely agree!

I hope most problems will be solved with the new recognition algorithm I'm planning to add. But of course @bribri if you were willing to share some code I'd be happy to integrate that (but I completely understand if that's not an option)

1 Like
7

I'd be willing to share some code. It's been a long time since I revisited the bit in xGestures that does the gesture detection. Looking at it now it could certainly stand to be cleaned up, and there's some parts of it that I'd have to think about to figure out precisely what they're doing.

I'm not particularly possessive of the xGestures codebase, especially now that it's been freeware for a few years, though I have been hesitant to open source it mainly because it's full of bad code that I wrote a long time ago back before I even really understood what good code is. At the very least I don't want people to look at it and say "this is the sort of code this fool will produce!"

8

I know that feeling very well :smiley:
Do you think the gesture recognition part could easily be put in a framework/library so you could still keep it closed source? Ah but on the other hand, I can't really include closed source stuff in BTT - not having the possibility to check for any bad things (of course I don't expect anything bad in it)

9

If I were going to put in the effort to release it as a framework, I'd just use that same effort to clean up the gesture detection code and release it fully. That wouldn't take very long, since it's a small part of the overall codebase. If you do think you'd like to incorporate it into BTT, I'll see about doing that at some point soon.

10

That would be really amazing! I would definitely incorporate that.

11

Okay! I can't promise exactly when I'll have the time to do that, but I'll try to get to it soon.

Do you think you'd be willing to add rocker gestures as well?

12

For rocker gestures BTT would need to wait a little bit before forwarding the right-click to the system, right? So that the right click does not happen if it is followed by a leftclick down immediately. I think there is already a (complicated) way to set this up in BTT. I'll give it a try and if it works I can make it easier to enable.

// edit I think when configured like this it can work as rocker gesture (just make sure to check the "allow to bind left-click without modifier keys" checkbox for the leftclick).

//edit2: unfortunately didn't fully work, it would still trigger a left click when left mouse comes first followed by a right mouse down. Are you preventing/delaying the left-mouse-down getting through to the system?

14

What I don't fully understand: what's the advantage of rocker gestures over just "right-mouse-down" drawing?

15

In the case of "rock right", which is pressing and holding the left button and then pressing the right, no, it doesn't delay the left click, since of course that would cause the system to become very difficult to use!

However, a "rock left" gesture (i.e. holding right and then clicking left) does withhold the right click since otherwise it would trigger a context menu on button down. If I were to implement rocker gestures involving the other mouse buttons (which is a feature request I received several times for xGestures) then I would do the same for those other auxiliary mouse buttons as well.

In the case of a "rock right" gesture, though, since the left mouse down event doesn't get withheld, that means that the system will consider the left button as being down when the rocker gesture is completed and it's time to execute an action. The trouble there is that the left button being down can prevent or interfere with certain gesture actions (as you may well know). One example is using "rock right" for history navigation in the Finder by setting it to perform the keystroke Cmd+]. If the Finder sees the left mouse button as being down then no keyboard shortcuts will work. So it's necessary to post an artificial event to release the left mouse button before executing the gesture action.

So in summary, a rocker gesture works like this:

  1. The user initiates a rocker gesture by pressing down one of the mouse buttons configured as the first button in the rocker gesture
  2. If it is the left button, do not withhold the button down event. If it is any other button, do withhold it.
  3. If the user releases the button without pressing any others, then they didn't perform the rocker gesture. For mouse buttons other than the left button, perform the withheld click (i.e. post the button down and up events). For the left button, the button down event wasn't withheld, so just allow the regular button up event to fire normally.
  4. If the user performs a rocker gesture by pressing down on another mouse button:
    • If the first button was the left button, post an artificial left button up event so that the system will think it's released. This may generate a click, and that's okay. Wait until the system has processed the event before proceeding to the action
    • Trigger the action associated with the gesture
    • Regardless of which buttons are involved, suppress the natural mouse up events that will occur from the user physically releasing the buttons -- we don't want a mouse up event for a mouse down that never occurred!

That's a bit longer of an explanation than I originally intended, but hopefully that makes sense!

16

Ah then what I posted before editing it away might have worked as intended. I'll post them again later.

But one more question: Do you only use these to start drawing the mouse gestures? Or are they used to trigger arbitrary actions?

17

Rocker gestures can execute arbitrary actions, at least in xGestures, and I know users of xGestures have assigned them to a variety of different actions.

The advantage of them is having another convenient and very quick way of triggering shortcuts with a standard mouse. They're more accurate and faster to perform than a drawing-based mouse gesture, even ones that just involve moving in a single direction. And since it has a left / right paradigm to it, it means you can have another set of gestures that map to a left / right space in your brain on top of the drawing gestures.

The way I use them is for navigation: being able to use rock left and rock right for moving backwards and forwards in navigation history, be it in a web browser, Finder, or wherever, is so ingrained into my muscle memory now that I break into a cold sweat on computers that don't have it. I originally picked up that habit from Opera nearly 20 years ago now.

I haven't really thought too much about what you could do with them if they could work with other mouse buttons, but an obvious thing is that you could practically make a mouse button become a mouse-specific modifier. So for example, holding down the auxiliary mouse button that's right under your thumb and then clicking any other mouse button could have their own actions, effectively doubling the number of things you can do just with mouse clicks.

18

Hi! What's the state of gesture development?

My trackpad has started behaving erratically, replacing would be to get a new one so I want to use this time to test a mouse. I had a lot of custom gestures on the trackpad so I'm looking to do the same. Since most of them were up/down/left/right movement, mouse gestures (gesture button + cursor movement) could make the transition way easier in paper.

I'll check xGestures for now, really looking forward to have this integrated in BTT.

Edit: It looks gestures are supported, but straight lines are not. However with a slight curve, up/down/left/right seems to be properly detected it's not really 100% consistent. Looking forward for a better detection implementation.

19

@Andreas_Hegenberg a quick note to say thanks for adding straight-line capability to the mouse gestures, it's great!

1 Like