Unfortunately it turned out this is a known issue with CGEventTaps on macOS (an API BTT and various other apps need to function). There currently doesn't seem to be a way to prevent this.
It seems to happen when macOS is busy and skips an event. I'll post another bug report to Apple about this.
In general remove all 2 finger pinch or rotate gestures from BTT, but that doesn't seem to fix it completely unfortunately.