ESC key very intermittent Macbook M1 pro

Describe the bug
The physical ESC key is very intermittent when using VMware Horizon. I use Horizon to connect to my work PC and then mostly a unix environment using Oracle SGD. The ESC key takes a long time to work, I just did a series of tests and it took 13, 13, 11, 10, 13, 14 keypresses of ESC before it registered, on both windows and the unix enviroment under it. Disabling BTT makes it work fine. Never had a problem on older macs until I got this M1 Pro, but it also has Big Sur so that could be the change too. I found other posts via google that pointed me to BTT being the culprit. I need BTT though to use gestures to middle-click to paste in the uniux side so can't just disable BTT on Horizon.

Affected input device (e.g. MacBook Trackpad, Magic Mouse/Trackpad, Touch Bar, etc.):
Keyboard

Device information:

  • Type of Mac: Macbook M1 Pro
  • macOS version: Big Sur 11.2.1
  • BetterTouchTool version: 3.557 (1697)
  • VMware Horizon: 8.1.0

Additional information (e.g. StackTraces, related issues, screenshots, workarounds, etc.):

On unix desktops I can hit CTRL-[ to get an escape key but this is painful when you have to use it a lot