@syjefroi you might want to look into Keyboard Maestro
It has the ability to read a specified rectangle of the screen - like "From 1080, 600 to 1200, 700" (returning a rectangle of 120 by 100 pixels) and then OCR any text found in that area. Then it can do logic against that text.
Based on that, you might be able to make a macro that scans Finale and works out what Tool is active, by reading the screen where the active tool is displayed (assuming it is), and then making decisions based on that.
Or, if no text is available, you could use the "Find Image On Screen" action and provide it with an icon to look for - which might be able to detect a pressed-in icon versus a not-pressed-in, if you know what I mean. Like, in Photoshop for example, the difference between:
Not pressed icon:
So in that example, I'd search for the second image and if it was found, I could conclude that the Brush tool is active. Or actually, a better option at least in Photoshop, would be for me to scan the toolbar at the top of the screen, which shows the active tool - so if I saw the brush icon at the top of the screen, I'd know that brush was active:
You can trigger a Keyboard Maestro macro via any BTT gesture - either by having BTT send a keyboard shortcut, or by using Apple Script or other scripting to trigger KM. Likewise you can have Keyboard Maestro trigger BTT actions using BTT's Scripting API, eg with Apple Script. KM has more advanced menu-making options, so that's something I'm looking at KM for right now: making menus of actions, some of which include BTT actions like screenshot capture.
Note I've never actually done this myself I only just bought Keyboard Maestro and am still learning it. I also don't use Finale so don't know what it looks like. So it is just an idea to investigate, I can't promise it works in your case specifically.