The configuration pane opens up by itself randomly. I cannot figure out why, since it happens while I am using the computer (typing or moving the mouse), but also when I am not doing anything and the screen is simply on.
BTT version 3.553 (1693)
MacBook Pro (15-inch, 2016)
BigSur 11.2.3 (20D91)
I have the same problem; it seems to have begun in the last week or so - possibly since I plugged in an Elgato Stream Deck (although I haven't yet configured it to do anything that would specifically invoke BTT).
BTT version 3.560 (1700)
Mac Mini (M1, 2020)
Big Sur 11.2.1 (20D74)
While I don't know what's causing this (most likely some third party software), here is a workaround: Configuration window appears out of the blue - #19 by Andreas_Hegenberg