Just had BTT halt and noticed it using a lot of memory, ~900MB. I had just been playing with excluding a DxO PhotoLab window from the window switcher.
After restarting BTT I repeatedly activated the window switcher, mainly with Cmd-Tab. Watching BTT in htop it's grown from ~70MB when freshly restarted to ~270MB now. It showed a gradual upwards trend that would pause when I stopped activating the window switcher.
I'm afraid it doesn't seem to have helped. About 5 minutes after repetitive window switcher activation I've currently got BTT using ~1GB and it's not showing any signs of giving it up.
Where are you checking the memory? Does your Mac still have a lot of memory free (in that case macOS in some cases does not release it until necessary)
OK, I have 2 VMs running which has resulted in 370MB of swap.
BTT then went from 1GB (from previous test) down to ~330MB ('Real' as reported by Activity Monitor) but still at 1GB in Activity Monitor's 'Memory' column. I didn't have Private Memory showing at that point.
I've recently had problems with BTT freezing far too often (I /think/ related to delayed actions (1ms delay), tried switching from sync to async delays but issue remains, will make proper report later) so had to restart BTT and repeat the test.
With the 2 VMs running I've managed to get BTT up to 467MB Memory, 517MB Real, 366MB Private. Purge has nudged 'Real' down slightly to 490MB.