Similar image actions to Keyboard Maestro

[Brahim]

I'm trying to have all my macros inside BTT, and I use find image and click a lot in my KM macros.

This is BTT:

This is KM:

BTT is mainly missing:

• Move & click in the same action
• Auto move back after click (Restore Mouse Location)

In my case, I want to start by migrating this KM macro to BTT:

It searches for the save button and clicks it. If not found, it looks for the Create field button. Timeout for each is 20ms.

This is what I have with BTT so far:

The main problem is that it clicks even when the button is not found, and it doesn't return the cursor to the original position.

[Andreas_Hegenberg]

currently not on my mac, but in BTT you‘d use the „if image visible on screen“ action/condition combined with the save/restore mouse position action (and an else branch for the other image)

[Brahim]

Thanks for the quick reply

I have this setup, testing the restore mouse location with only 1 button for now, it's very slow compared to KM for some reason, maybe because it's multiple actions..?

Would love all of this to be consolidated into a single action, which I think would make it much faster, simpler and easier.

[Andreas_Hegenberg]

no that doesn't make a difference, BTT always uses reusable building blocks for its actions. Bundling them into one action wouldn't make a difference.

The image recognition speed can be influenced by many things, do you have set the mouse move animation duration to 0?

Searching for smaller pictures can also speed up the search by a lot.
If you know the approximate position of the image, the recofgnition speed can also be increased by a lot by specifying the search region.

[Brahim]

Finding the button is quite fast, but one annoying issue is the return to mouse position feature - it doesn't move until I manually move the mouse cursor, and only then does it restore the position.

[Andreas_Hegenberg]

Could you export that trigger via right-click and post it (or send to andreas@folivora.ai)? The mouse move , click & restore should be so fast that you can't even see it because the system doesn't update the rendered cursor if it is happening so fast. If that's not the case something is wrong/broken.

[Brahim]

Looks like adding a 0.1sec wait right before the restore mouse position action fixed it

This is the trigger

[
  {
    "BTTActionCategory" : 0,
    "BTTLastUpdatedAt" : 1751849222.1733479,
    "BTTTriggerType" : 0,
    "BTTTriggerClass" : "BTTTriggerTypeKeyboardShortcut",
    "BTTUUID" : "023460BC-68DA-47AE-B82E-990F750EBB34",
    "BTTPredefinedActionType" : 366,
    "BTTPredefinedActionName" : "Empty Placeholder",
    "BTTAdditionalConfiguration" : "1048584",
    "BTTKeyboardShortcutKeyboardType" : 2302,
    "BTTTriggerOnDown" : 1,
    "BTTNotesInsteadOfDescription" : 1,
    "BTTLayoutIndependentChar" : "RETURN",
    "BTTEnabled" : 1,
    "BTTEnabled2" : 1,
    "BTTShortcutKeyCode" : 36,
    "BTTShortcutModifierKeys" : 1048576,
    "BTTOrder" : 1,
    "BTTAutoAdaptToKeyboardLayout" : 0,
    "BTTAdditionalActions" : [
      {
        "BTTActionCategory" : 0,
        "BTTLastUpdatedAt" : 1751849228.3430769,
        "BTTTriggerParentUUID" : "023460BC-68DA-47AE-B82E-990F750EBB34",
        "BTTIsPureAction" : true,
        "BTTTriggerClass" : "BTTTriggerTypeKeyboardShortcut",
        "BTTUUID" : "9404C26C-48E5-4927-BAE3-790D95EC6980",
        "BTTPredefinedActionType" : 154,
        "BTTPredefinedActionName" : "Save Current Mouse Position",
        "BTTGenericActionConfig" : "saved_mouse_position",
        "BTTKeyboardShortcutKeyboardType" : 0,
        "BTTEnabled" : 1,
        "BTTEnabled2" : 1,
        "BTTShortcutKeyCode" : -1,
        "BTTOrder" : 0,
        "BTTAutoAdaptToKeyboardLayout" : 0
      },
      {
        "BTTActionCategory" : 0,
        "BTTLastUpdatedAt" : 1751848682.0392909,
        "BTTTriggerParentUUID" : "023460BC-68DA-47AE-B82E-990F750EBB34",
        "BTTIsPureAction" : true,
        "BTTTriggerClass" : "BTTTriggerTypeKeyboardShortcut",
        "BTTUUID" : "D22D5C76-94E1-416F-A0EC-513BD75217A4",
        "BTTPredefinedActionType" : 419,
        "BTTPredefinedActionName" : "Wait For Image To Become Visible On Screen & Move Mouse",
        "BTTFindPositionOfImageImage" : "iVBORw0KGgoAAAANSUhEUgAAACsAAAAZCAYAAACo79dmAAAEDmlDQ1BrQ0dDb2xvclNwYWNlR2VuZXJpY1JHQgAAOI2NVV1oHFUUPpu5syskzoPUpqaSDv41lLRsUtGE2uj+ZbNt3CyTbLRBkMns3Z1pJjPj_KRpKT4UQRDBqOCT4P9bwSchaqvtiy2itFCiBIMo+ND6R6HSFwnruTOzu5O4a73L3PnmnO9+595z7t4LkLgsW5beJQIsGq4t5dPis8fmxMQ6dMF90A190C0rjpUqlSYBG+PCv9rt7yDG3tf2t_f_Z+uuUEcBiN2F2Kw4yiLiZQD+FcWyXYAEQfvICddi+AnEO2ycIOISw7UAVxieD_Cyz5mRMohfRSwoqoz+xNuIB+cj9loEB3Pw2448NaitKSLLRck2q5pOI9O9g_t_tkXda8Tbg0+PszB9FN8DuPaXKnKW4YcQn1Xk3HSIry5ps8UQ_2W5aQnxIwBdu7yFcgrxPsRjVXu8HOh0qao30cArp9SZZxDfg3h1wTzKxu5E_LUxX5wKdX5SnAzmDx4A4OIqLbB69yMesE1pKojLjVdoNsfyiPi45hZmAn3uLWdpOtfQOaVmikEs7ovj8hFWpz7EV6mel0L9Xy23FMYlPYZenAx0yDB1_PX6dledmQjikjkXCxqMJS9WtfFCyH9XtSekEF+2dH+P4tzITduTygGfv58a5VCTH5PtXD7EFZiNyUDBhHnsFTBgE0SQIA9pfFtgo6cKGuhooeilaKH41eDs38Ip+f4At1Rq_sjr6NEwQqb_I_DQqsLvaFUjvAx+eWirddAJZnAj1DFJL0mSg_gcIpPkMBkhoyCSJ8lTZIxk0TpKDjXHliJzZPO50dR5ASNSnzeLvIvod0HG_mdkmOC0z8VKnzcQ2M_Yz2vKldduXjp9bleLu0ZWn7vWc+l0JGcaai10yNrUnXLP_8Jf59ewX+c3Wgz+B34Df+vbVrc16zTMVgp9um9bxEfzPU5kPqUtVWxhs6OiWTVW+gIfywB9uXi7CGcGW_zk98k_kmvJ95IfJn_j3uQ+4c5zn3Kfcd+AyF3gLnJfcl9xH3OfR2rUee80a+6vo7EK5mmXUdyfQlrYLTwoZIU9wsPCZEtP6BWGhAlhL3p2N6sTjRdduwbHsG9kq32sgBepc+xurLPW4T9URpYGJ3ym4+8zA05u44QjST8ZIoVtu3qE7fWmdn5LPdqvgcZz8Ww8BWJ8X3w0PhQ_wnCDGd+LvlHs8dRy6bLLDuKMaZ20tZrqisPJ5ONiCq8yKhYM5cCgKOu66Lsc0aYOtZdo5QCwezI4wm9J_v0X23mlZXOfBjj8Jzv3WrY5D+CsA9D7aMs2gGfjve8ArD6mePZSeCfEYt8CONWDw8FXTxrPqx_r9Vt4biXeANh8vV7_+_16ffMD1N8AuKD_A_8leAvFY9bLAAAAOGVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAACoAIABAAAAAEAAAAroAMABAAAAAEAAAAZAAAAAJb1jiQAAANCSURBVFgJ7VVrSFNhGH7Ods50zm1uTt3ygvOSRlJWFlL9SbxminQDU9EKhTToRxRSGYXUj5BFKRhCEFlG0b9SRLqCPyI16QIm5rxmYS3npc1tZ67vnKVQxDoD_SHs_XHOd97zfO95vud9vu9QuoP9LqySEK0SnjxNH9mV6pZPWZ+yRAFaqAoBShqH84KxWS9B_2cHbrabYZqYFzp9WXCUkHM2UM3g7fVo+NEUxqec0ClFEIsobK8Zx+iAZVmICCkiaIOV5KghZSgknxxF6rFBrK0YwpzNicpclfsbFJCZrkZ1uQ55WcEQ0e6y+_ZokJgkX+KRQMYH8jX8s0Qm5rFVxWEIi5QuYTwNBNlg3r7A1yjOCEL9vUnY5lgklg4u1W2r02NjOAOTxQVNOoWPuxVIOzGEU7lK0GIgpWKWxzYfD8UC+V8+emHGu3o9AvwozM47cTZPieLGSTwjeU8hSNnmxz9g_M7idI4CY7fj0N0Ui_LCEL4uRSo8+WBFxqUv2HBkAHXtM0jUMqD9xWh8Oo01SjECVAz8FTQiVO7c+VItTzSxcgjrygbRPWxHQ1mwJ578O0HKslYndlYZEZMgQ9EuJQpTpbhYoEJUMIOahgm8Mc7DUKaBVqlFqJx4goSEqNbSOoXL+9U4lBUEB+uCi6jK5brqY0Asj9eGKB4bJCXykxCJRVhwurvIJ_66CCKbsk0BuVSE5y_NqO3_idobQN+tWGQnSXEt3B8tlaHoIupU3zchf4sMBZvcHnSQRfaM2FGyQw4n6X_vmB1c7us0i5BABqWN33g6UgkFi83lkSgHFGSDzGQZ7hJCRXtDEBTmh+wMNRSkzb3jdkSQlnPR8d6Knj4LtkZL+OfFy9U2M+JDaSSEMTC0uj354NUcKKJsjFaCTyM2HE1T4krR_20g6OhiSJseXoj8g8iwiUX6mTFYZxzoMOixXucmPTDJ8uTiKoywEAU5Tw_fiee5RxcPwPW7y03nIpFLOsORthOLcBuss3N6cY3_vAsiuziTJqSj9f4YG7XxJ8Jinrv7yWk4HS6wZHcLDW4hgWTzzZocgqZ4RVZQxRUECfLsCn7fq9I+sl7J5QXYp6wXYnkF9SnrlVxegH8Bdiv9m6msgPUAAAAASUVORK5CYII=",
        "BTTGenericActionConfig2" : "{\"BTTFindImageSearchOn\":5,\"BTTFindImageSearchRegion\":0,\"BTTFindImageUseDifferentDarkModeImage\":false,\"BTTFindImageSquareSize\":80,\"BTTFindImageUseColor\":true,\"BTTFindImageTreshold\":0.85000002384185791,\"BTTActionWaitForConditionsTimeout\":0.20000000298023224,\"BTTFindImageSaveToVariable\":false,\"BTTFindImageBTTMouseMoveDragType\":0,\"BTTFindImageContinueActionExecution\":false,\"BTTFindImageMoveMouseTo\":2,\"BTTFindImageBTTMouseMoveDuration\":0,\"BTTActionWaitForConditionsInterval\":1}",
        "BTTKeyboardShortcutKeyboardType" : 0,
        "BTTEnabled" : 1,
        "BTTEnabled2" : 1,
        "BTTShortcutKeyCode" : -1,
        "BTTOrder" : 1,
        "BTTAutoAdaptToKeyboardLayout" : 0
      },
      {
        "BTTActionCategory" : 0,
        "BTTLastUpdatedAt" : 1751848648.1082439,
        "BTTTriggerParentUUID" : "023460BC-68DA-47AE-B82E-990F750EBB34",
        "BTTIsPureAction" : true,
        "BTTTriggerClass" : "BTTTriggerTypeKeyboardShortcut",
        "BTTUUID" : "A6CDD08C-839E-44A3-A022-32C607F29EF9",
        "BTTPredefinedActionType" : 3,
        "BTTPredefinedActionName" : "Left Click (At Current Mouse Position)",
        "BTTKeyboardShortcutKeyboardType" : 0,
        "BTTEnabled" : 1,
        "BTTEnabled2" : 1,
        "BTTShortcutKeyCode" : -1,
        "BTTOrder" : 2,
        "BTTAutoAdaptToKeyboardLayout" : 0
      },
      {
        "BTTActionCategory" : 0,
        "BTTLastUpdatedAt" : 1751849229.4385118,
        "BTTTriggerParentUUID" : "023460BC-68DA-47AE-B82E-990F750EBB34",
        "BTTIsPureAction" : true,
        "BTTTriggerClass" : "BTTTriggerTypeKeyboardShortcut",
        "BTTUUID" : "42FFD5FB-AB1A-40ED-BF9D-5EF3A5281968",
        "BTTPredefinedActionType" : 155,
        "BTTPredefinedActionName" : "Restore Saved Mouse Position",
        "BTTGenericActionConfig" : "saved_mouse_position",
        "BTTKeyboardShortcutKeyboardType" : 0,
        "BTTEnabled" : 1,
        "BTTEnabled2" : 1,
        "BTTShortcutKeyCode" : -1,
        "BTTOrder" : 3,
        "BTTAutoAdaptToKeyboardLayout" : 0
      }
    ],
    "BTTGestureNotes" : "(BTT actions - Click Airtable Save Button)"
  }
]

[Andreas_Hegenberg]

Ah indeed, there is a little problem with the "Wait for image to become visible" action - the standard "Find/Search Image On ScreenAnd Move Mouse" action already contains a workaround for this, I'll add the same to the wait for image and the if image visible actions with the next alpha.

Do you need the "wait for image" action? If not I'd recommend to use the standard "Find/Search Image" action as it is a little less complex

[Brahim]

Switched it and it worked without needing the wait action!

I don't need the wait for this specific case, since I trigger when the button is already visible. The wait for image then move is very useful for my other multi-step processes, where it needs to wait for the UI to update before clicking the next element.

Thanks for the help Andreas!

[Brahim]

Would it be possible to choose which occurrence of the image to find? I used to do this in Microsoft Power Automate.

In my example, I always want it to find the bottom edit icon. I have the option to only look at the bottom half of the screen, but that section still has multiple similar buttons.

Another solution would be to search from bottom to top, similar to the 'find text' action.

Thanks!

[Brahim]

@Andreas_Hegenberg

The restore mouse position doesn't work anymore

I'm on

[Andreas_Hegenberg]

ah sorry this is currently a work in progress in the current alphas, best downgrade for now (Index of /releases )

I‘m currently adding some new options

[Brahim]

Always looking forward to new options

[Andreas_Hegenberg]

I have added "Search Direction" and "Skip First X" options in 5.529 alpha:

However I wasn't able to reproduce the mouse restoration issue you mentioned, it still seems to work fine here

[Brahim]

I think the mouse restoration issue is now fixed

This is great!

Tested the search direction, it sorta works
The occurrence setting is not working for me, it's very inconsistent, but also hard to figure out with the search direction options.

Could it be "Top to Bottom, Bottom to Top, Left to Right, Right to Left"? I think these would be the most useful instead of diagonal options.

Also, I think instead of 'skip', it could be simplified to the actual order number I want the action to target, So if I want it to find button #2, I'd just type 2 intuitively, instead of 1.

[Andreas_Hegenberg]

I think the skip option works fine, however it is important to note that all matches must reach the defined threshold.

The search needs to search the whole screen and not just a column / row, thus it needs a direction it searches in. Top to bottom would not be specific enough (if you say top to bottom, does it start searching at the top left, at the center or maybe at the top right?).

//edit: I think I have an idea to make the configuration for this easier. Will try something tomorrow!

[Andreas_Hegenberg]

In 5.535 I have updated the search algorithm a bit more, I think might now behave more like one would expect given the direction & skip settings

[Brahim]

This is beautiful!!

It works exactly as expected!

Many Thanks!

[Brahim]

Can this have the option to be based on a variable?

I want to be able to put "{{loop_run_index}}" as the value so I could process similar items one after the other in a loop

Having variable interpolation across BTT would be huge, If it's a normal number, it works normally, but if it has curly brackets, it gets the variable value first then proceeds, which would make BTT more modular. When there's a new field somewhere, it always has the option of being a variable.

This implementation would also simplify this to a single field

With '{{nb_repeats}}' as the value instead of having a dedicated field for variable name.

I know this might cause some problems with data validation in some cases, but can easily be resolved.

[Andreas_Hegenberg]

I will add variable support for this one! Unfortunately I can't have a generic variable interpolation in BTT right now, but you can always right-click an action and convert it to java script if you need more flexibility over variable usage.