unfortunately I never figured out what causes this. It seems to happen on (very few) systems, since macOS 10.12. I assume something on the system is trying to launch BTT again while it is already running, which will cause the prefs to show.
The only workaround is this:
Configuration window appears out of the blue - Bug Reports - BetterTouchTool Community (folivora.ai)