Quick update since this new setting came along:
In combination with the trigger condition "fingers_touching_trackpad is 0" it works kind of like expected, if I'm only working with a touchpad.
I guess it's still not possible to only trigger for specific mouse devices unfortunately, as also requested back here: Differentiate between two different mice like you can for keyboards? Or?