Accessibility API implementation in Transform and Replace action

Discussion & Questions
1

Hey Andreas!

I love BTT and I have been using it for the past 2 years now. It's incredible how many features are crammed in :blush:

I noticed that you have a new "Transform & Replace Selection With ChatGPT" action and I wanted to ask about it.

I am developing an app that helps users with text - and I need to access the user's selected text. For this I am using the AXAccesibility API of course but there are some apps that I quite cannot get to work - like Firefox (GeckoNSApplication) or Microsoft Word. The API simply does not return the element or the tree.

Even though I am trying to set the AXEnhancedUserInterface and even the AXManualAccessibility to true. I have went through many forums, and bug report submissions in Chromium, Mozzila etc. but haven't found a definitive answer. It works in native & most Electron apps.

When I try the "Transform & Replace" action from BTT, it works anywhere - it obtains the user text, and then pastes it back and does it flawlessly. Like if there was a VoiceOver active on the system (in that case my app also works flawlessly)

Would you be willing to provide me with some insight into your implementation? Because I have spent a few tens of hours on this and didn't find any definitive solution.

2

The most simple way. BTT just does send the cmd+c shortcut because as you say, there is no other reliabe way as far as I know. Afterwards it copies the data feom the clipboard.

(Maybe there would be better ways uaing services, but cmd+c works good enough for me)

1 Like
3

That's genius, thank you Andreas.

And could I ask on how you are simulating the CMD+C/CMD+V action?

I am using this but I found it kind of unreliable as it does not paste sometimes

let eventDown = CGEvent(keyboardEventSource: nil, virtualKey: 0x09, keyDown: true)
eventDown?.flags = .maskCommand
eventDown?.post(tap: .cghidEventTap)
let eventUp = CGEvent(keyboardEventSource: nil, virtualKey: 0x09, keyDown: false)
eventUp?.post(tap: .cghidEventTap)
4

And I also found out that if the user invokes the Transform & Replace action in an empty field, it pastes modified response from what was already in the clipboard, so you might consider that a bug report :D.