Application's Configuration window appears after waking Mac from sleep

Often times I'll unlock my Mac using the fingerprint reader and I'll find that BTT has opened its Configuration window. It doesn't do it all the time but fairly often.

Then when I click to close it, it'll act like there are multiple windows open. I have to click the red x about 3 times for it to close.

I get this frequently as well, multiple instances of the config window are opening on their own