I have been playing with BetterTouchTool with Apple Shortcut App for couple of days. It is really nice to see that you have added actions for GUI automation The automation that I am trying to build really becomes clunky with several "Send Keyboard Shortcut". in order to reach to a particular GUI elements, one after the other. with Key Code 48 (Tab). And it comes somewhat slow.
I wonder if you could add more advance actions for GUI automation. I wonder if you could create additional "actions" that would allow users to to target specific GUI Elements.
In the tutorial video above, pfiddlesoft's UI browser is used to identify the element code of a checkbox. In conjunction with Apple Script and Keyboard Maestro is use to automate a task. Despite Keyboard Maestro being powerful application, it is not a user friendly application.
With Apple's Shortcut App, automation becomes more user friendly and accessible without knowing complex Apple Script coding.
Could you add a series of actions into Apple's Shortcut where users could copy/paste GUI Element code ID from pfiddlesoft's UI browser. That way, Apple Shortcut would directly trigger or go to that element in one step. Without this, users would need to add Key Code 48 (Tab) "Tab" several times in order to reach that particular GUI element.
Hope these improvements can be added soon.
Thank you.
Additional request -
Instead of users adding Key Code Values manually, It is possible to have an action for the selecting the entire keyboard. Users would not know the values of Tab, Enter, Home, Page Up, Page Down, and etc. Maybe there can be a dropdown for selecting all the keystrokes?
Unfortunately that's really hard to do (which is probably why Apple has not done it)
You can however use the automator app on macOS to record such workflows, if you then select the actions created by automator and hit cmd+c & cmd+v you'll get the raw Apple Script which you can then modify without needing to know much about Apple Script
The UI Browser is able to create AppleScript for the UI elements (click, get and set value etc.), which can then be included in a BTT action. Often, however, the script must be adapted. Without rudimentary AppleScript knowledge, UI scripting is hardly possible.
I have been trying very hard to create GUI automation using UI Browser. But unfortunately, every attempts has been unsuccessful. I don't possess the knowledge of AppleScript. Hence, it is somewhat difficult for me. I am unable to achieve the automation that I need.
Till now, there has not been a simple drag and drop app for automation. I feel Apple's Shortcut could be the application to open up GUI automation.
I understand that it can be hard to achieve. Can there be a middle ground? Could BTT provide actions needed for simple UI elements? Maybe be you could start with few action actions for set value in text field and selecting specific items in drop-downs. With more user's feedback. It could grow over time.
Hope you could consider this.
Also could you add these improvements for the current actions?
Instead of users adding Key Code Values manually, It is possible to have an action for the selecting the entire keyboard. Users would not know the values of Tab, Enter, Home, Page Up, Page Down, and etc. Maybe there can be a drop-down for selecting all the keystrokes?
Say I need to tab 10 times, instead of have 10 Send Keyboard Shortcut, can there be a option for a multiplier in the Send Keyboard Shortcut action?
The shortcut becomes slow and sluggish when more BTT actions are added. Hope it can be improved.
UI Browser: You have to move the mouse over the target element in the screen reader and click the shortcut (default: ctrl + cmd + s). Then select AppleScript at the bottom right and put the code into the following block:
tell application "System Events"
-- UI Browser code
end tell
For example:
tell application "System Events"
click radio button 1 of radio group 1 of group 1 of group 1 of toolbar 1 of window 1 of application process "Photos"
end tell
The problem is that the UI elements do not always have a fixed ID. They are addressed either by name (click radio button "Years") or by the number in the respective listing (click radio button 1) or you have to search for the element before:
clickMenuBarItem("com.apple.menuextra.wifi", "Hotspot") -- activate personal hotspot
on clickMenuBarItem(barItem, menuItem)
tell application "System Events"
tell application process "ControlCenter"
repeat with tElement in menu bar items of menu bar 1
if (exists attribute "AXIdentifier" of tElement) then
if value of attribute "AXIdentifier" of tElement = barItem then
set theBar to tElement
tell theBar
click
end tell
set tElements to entire contents of window 1
set tElements to entire contents of window 1 -- needed (UI bug in macOS 12)
repeat with i from 1 to count of tElements
set tElement to item i of tElements
if (title of tElement contains menuItem) then
tell tElement
click
end tell
exit repeat
end if
end repeat
tell theBar
click
end tell
end if
end if
end repeat
end tell
end tell
end clickMenuBarItem
This is what makes UI scripting so complicated, plus it changes with each version and is language dependent.
If it were as easy to implement as you would like, there would already be apps in this direction. The UI Browser is currently the best tool for this or you use the recording function of the Automator as suggested by Andreas.
@Dirk It is indeed complicated. Thanks for the explanation. Really appreciate it!
@Andreas_Hegenberg Is possible to address the other improvements of the current BTT actions in shortcuts? I also notice as soon as if there are more then 5-6 BTT actions in the shortcut, it becomes externally slow and sluggish. And shortcut crashes at times.
I am unable to achieve the automation that I need.