move mouse to a specific ui element

hello! i am relatively new to btt and am experimenting with some settings and actions.

i work with logic and wonder if it is possible to use a shortcut to move the mouse to a specific ui element. for example: i mainly use an eq called fabfliter pro-q 4, which has a ui element which moves within the plugin window.

is it possible to search with btt for a specific ui element and create an action to move the mouse cursor there?

i dont think that this can be achieved with a relative position, cause the element within the plugin window is constantly moving around, depending on the frequency of the eq band.

best wishes. r

you can try the „find image on screen & move mouse“ action. It can be tricky to configure depending on the element you are searching for.

Better would be using Apple Script, but it’s often not easy to access a UI element via AS either

Ah thank you. I found the option to take a screenshot and place the mouse pointer to the element. i wanna control 3 elements. Sadly, when the mouse pointer gets placed there, a hover on menu gets popped up right beneath the control element and therefore it is not possible to reallocate the ui elements unless i place the mouse pointer away. which is a bit cumbersome. so i will try it with an apple script, although i have no clue how to do that.

doesn't the hover menu go away when you click? How does it behave if you move your mouse manually there to click?


it looks like this...

when the mouse cursor is on one of the knobs, the little hover over menu is there. when i click, it stays there. but when i move the mouse away, its gone.

Here's an implementation that uses OpenAI's new computer-use model with their new Responses API – it's pretty cool.

See docs here: https://platform.openai.com/docs/guides/tools-computer-use

I wrote this script quite quickly so I'll add improvements later... but it works!

In this demo, you type in a description of the element on your screen that you want to click on and then using vision AI your computer will automatically click on it.

Here's the script:


(async () => {
	let OPENAI_API_KEY = await get_keychain_item("BTT_OPENAI_API_KEY");

	let screenWidth = await get_number_variable({
		variable_name: "focused_screen_width",
	});

	let screenHeight = await get_number_variable({
		variable_name: "focused_screen_height",
	});

	let screenShotScript = `screencapture -cC`;

	await runShellScript({ script: screenShotScript });

	let clipboardBase64 = await get_clipboard_content({
		format: "public.png",
		asBase64: true,
	});

	// Get the prompt from BTT or use a default
	let prompt =
		(await get_string_variable({
			variable_name: "CLICK_ON_ELEMENT",
		})) || "the Apple logo";

	// Calculate target dimensions (similar to Python implementation)
	const MAX_WIDTH = 1728;
	let targetWidth = screenWidth;
	let targetHeight = screenHeight;

	if (screenWidth > MAX_WIDTH) {
		const scaleFactor = MAX_WIDTH / screenWidth;
		targetWidth = MAX_WIDTH;
		targetHeight = Math.round(screenHeight * scaleFactor);
	}

	// Prepare the API call
	const url = "https://api.openai.com/v1/responses";

	const headers = {
		Authorization: `Bearer ${OPENAI_API_KEY}`,
		"Content-Type": "application/json",
	};

	// Check for organization ID
	const OPENAI_ORG = await get_keychain_item("BTT_OPENAI_ORG");
	if (OPENAI_ORG) {
		headers["Openai-Organization"] = OPENAI_ORG;
	}

	// Prepare the input with screenshot and prompt
	const inputItems = [
		{
			role: "user",
			content: [
				{ type: "input_text", text: `Find and click on: ${prompt}` },
				{
					type: "input_image",
					image_url: `data:image/png;base64,${clipboardBase64}`,
				},
			],
		},
	];

	// Tools configuration
	const tools = [
		{
			type: "computer-preview",
			display_width: targetWidth,
			display_height: targetHeight,
			environment: "mac",
		},
	];

	// Prepare the payload
	const payload = {
		model: "computer-use-preview",
		input: inputItems,
		tools: tools,
		truncation: "auto",
	};

	try {
		const response = await fetch(url, {
			method: "POST",
			headers: headers,
			body: JSON.stringify(payload),
		});

		if (!response.ok) {
			throw new Error(
				`API Error: ${response.status} ${await response.text()}`
			);
		}

		const responseData = await response.json();

		// Extract click coordinates from the response
		try {
			const output = responseData.output || [];
			for (const item of output) {
				if (item.type === "computer_call") {
					const action = item.action || {};
					if (action.type === "click") {
						let waitMouseMoveDuration = 0.2;
						let moveMouseActionDefinition = {
							BTTActionCategory: 0,
							BTTIsPureAction: true,
							BTTPredefinedActionType: 153,
							BTTPredefinedActionName: "Move Mouse To Position",
							BTTAdditionalActionData: {
								BTTMouseMoveX: action.x,
								BTTMouseMoveAnchor: 0,
								BTTMouseMoveY: -action.y,
								BTTMouseMoveDuration: waitMouseMoveDuration,
							},
						};

						await trigger_action({
							json: JSON.stringify(moveMouseActionDefinition),
							wait_for_reply: true,
						});

						let waitActionDefinition = {
							BTTActionCategory: 0,
							BTTIsPureAction: true,
							BTTPredefinedActionType: 345,
							BTTPredefinedActionName:
								"Pause Execution  or  Delay Next Action (async  or  not blocking)",
							BTTDelayNextActionBy: String(
								waitMouseMoveDuration + 0.5
							),
						};

						await trigger_action({
							json: JSON.stringify(waitActionDefinition),
							wait_for_reply: true,
						});

						let clickActionDefinition = {
							BTTActionCategory: 0,
							BTTIsPureAction: true,
							BTTPredefinedActionType: 3,
							BTTPredefinedActionName:
								"Left Click (At Current Mouse Position)",
						};

						await trigger_action({
							json: JSON.stringify(clickActionDefinition),
							wait_for_reply: true,
						});

						// Return the coordinates if found
						returnToBTT(JSON.stringify(action));
						return;
					}
				}
			}
			// If we reach here, no click action was found
			console.log(`No click action found in response for: ${prompt}`);
			returnToBTT(
				JSON.stringify({
					success: false,
					error: `Could not identify ${prompt}`,
				})
			);
		} catch (e) {
			console.error(`Error extracting coordinates: ${e.message}`);
			returnToBTT(
				JSON.stringify({
					success: false,
					error: `Error extracting coordinates: ${e.message}`,
				})
			);
		}
	} catch (error) {
		returnToBTT(
			JSON.stringify({
				success: false,
				error: error.message,
			})
		);
	}
})();

Here's the BTT JSON config:

[
  {
    "BTTActionCategory" : 0,
    "BTTLastUpdatedAt" : 1741948279.2131031,
    "BTTTriggerType" : 0,
    "BTTTriggerClass" : "BTTTriggerTypeKeyboardShortcut",
    "BTTUUID" : "FAC970B7-47DE-4F73-85F9-6002E9EEBD77",
    "BTTPredefinedActionType" : 366,
    "BTTPredefinedActionName" : "Empty Placeholder",
    "BTTAdditionalConfiguration" : "8388608",
    "BTTKeyboardShortcutKeyboardType" : 2302,
    "BTTTriggerOnDown" : 1,
    "BTTLayoutIndependentChar" : "F4",
    "BTTEnabled" : 1,
    "BTTEnabled2" : 1,
    "BTTShortcutKeyCode" : 118,
    "BTTShortcutModifierKeys" : 8388608,
    "BTTOrder" : 10,
    "BTTAutoAdaptToKeyboardLayout" : 0,
    "BTTAdditionalActions" : [
      {
        "BTTActionCategory" : 0,
        "BTTLastUpdatedAt" : 1741948188.1062689,
        "BTTTriggerParentUUID" : "FAC970B7-47DE-4F73-85F9-6002E9EEBD77",
        "BTTIsPureAction" : true,
        "BTTTriggerClass" : "BTTTriggerTypeKeyboardShortcut",
        "BTTUUID" : "41899D96-6ACA-45B5-B8ED-06CE2A0B2D95",
        "BTTPredefinedActionType" : 403,
        "BTTPredefinedActionName" : "Ask For Input (Save To Variable)",
        "BTTAdditionalActionData" : {
          "BTTActionAskForInputOnlyHideOnEnterOrEsc" : 1,
          "BTTActionAskForInputVariableName" : "CLICK_ON_ELEMENT",
          "BTTActionAskForInputPrompt" : "What do you want to click on?"
        },
        "BTTKeyboardShortcutKeyboardType" : 0,
        "BTTEnabled" : 1,
        "BTTEnabled2" : 1,
        "BTTShortcutKeyCode" : -1,
        "BTTOrder" : 1,
        "BTTAutoAdaptToKeyboardLayout" : 0
      },
      {
        "BTTActionCategory" : 0,
        "BTTLastUpdatedAt" : 1741949678.9966969,
        "BTTTriggerParentUUID" : "FAC970B7-47DE-4F73-85F9-6002E9EEBD77",
        "BTTIsPureAction" : true,
        "BTTTriggerClass" : "BTTTriggerTypeKeyboardShortcut",
        "BTTUUID" : "9778EB56-8A76-4D86-A071-62F3C6D4B1C8",
        "BTTPredefinedActionType" : 281,
        "BTTPredefinedActionName" : "Run Real JavaScript",
        "BTTAdditionalActionData" : {
          "BTTScriptFunctionToCall" : "someJavaScriptFunction",
          "BTTJavaScriptUseIsolatedContext" : false,
          "BTTAppleScriptRunInBackground" : false,
          "BTTScriptType" : 3,
          "BTTAppleScriptString" : "(async () => {\n\tlet OPENAI_API_KEY = await get_keychain_item(\"BTT_OPENAI_API_KEY\");\n\n\tlet screenWidth = await get_number_variable({\n\t\tvariable_name: \"focused_screen_width\",\n\t});\n\n\tlet screenHeight = await get_number_variable({\n\t\tvariable_name: \"focused_screen_height\",\n\t});\n\n\tlet screenShotScript = `screencapture -cC`;\n\n\tawait runShellScript({ script: screenShotScript });\n\n\tlet clipboardBase64 = await get_clipboard_content({\n\t\tformat: \"public.png\",\n\t\tasBase64: true,\n\t});\n\n\t\/\/ Get the prompt from BTT or use a default\n\tlet prompt =\n\t\t(await get_string_variable({\n\t\t\tvariable_name: \"CLICK_ON_ELEMENT\",\n\t\t})) || \"the Apple logo\";\n\n\t\/\/ Calculate target dimensions (similar to Python implementation)\n\tconst MAX_WIDTH = 1728;\n\tlet targetWidth = screenWidth;\n\tlet targetHeight = screenHeight;\n\n\tif (screenWidth > MAX_WIDTH) {\n\t\tconst scaleFactor = MAX_WIDTH \/ screenWidth;\n\t\ttargetWidth = MAX_WIDTH;\n\t\ttargetHeight = Math.round(screenHeight * scaleFactor);\n\t}\n\n\t\/\/ Prepare the API call\n\tconst url = \"https:\/\/api.openai.com\/v1\/responses\";\n\n\tconst headers = {\n\t\tAuthorization: `Bearer ${OPENAI_API_KEY}`,\n\t\t\"Content-Type\": \"application\/json\",\n\t};\n\n\t\/\/ Check for organization ID\n\tconst OPENAI_ORG = await get_keychain_item(\"BTT_OPENAI_ORG\");\n\tif (OPENAI_ORG) {\n\t\theaders[\"Openai-Organization\"] = OPENAI_ORG;\n\t}\n\n\t\/\/ Prepare the input with screenshot and prompt\n\tconst inputItems = [\n\t\t{\n\t\t\trole: \"user\",\n\t\t\tcontent: [\n\t\t\t\t{ type: \"input_text\", text: `Find and click on: ${prompt}` },\n\t\t\t\t{\n\t\t\t\t\ttype: \"input_image\",\n\t\t\t\t\timage_url: `data:image\/png;base64,${clipboardBase64}`,\n\t\t\t\t},\n\t\t\t],\n\t\t},\n\t];\n\n\t\/\/ Tools configuration\n\tconst tools = [\n\t\t{\n\t\t\ttype: \"computer-preview\",\n\t\t\tdisplay_width: targetWidth,\n\t\t\tdisplay_height: targetHeight,\n\t\t\tenvironment: \"mac\",\n\t\t},\n\t];\n\n\t\/\/ Prepare the payload\n\tconst payload = {\n\t\tmodel: \"computer-use-preview\",\n\t\tinput: inputItems,\n\t\ttools: tools,\n\t\ttruncation: \"auto\",\n\t};\n\n\ttry {\n\t\tconst response = await fetch(url, {\n\t\t\tmethod: \"POST\",\n\t\t\theaders: headers,\n\t\t\tbody: JSON.stringify(payload),\n\t\t});\n\n\t\tif (!response.ok) {\n\t\t\tthrow new Error(\n\t\t\t\t`API Error: ${response.status} ${await response.text()}`\n\t\t\t);\n\t\t}\n\n\t\tconst responseData = await response.json();\n\n\t\t\/\/ Extract click coordinates from the response\n\t\ttry {\n\t\t\tconst output = responseData.output || [];\n\t\t\tfor (const item of output) {\n\t\t\t\tif (item.type === \"computer_call\") {\n\t\t\t\t\tconst action = item.action || {};\n\t\t\t\t\tif (action.type === \"click\") {\n\t\t\t\t\t\tlet waitMouseMoveDuration = 0.2;\n\t\t\t\t\t\tlet moveMouseActionDefinition = {\n\t\t\t\t\t\t\tBTTActionCategory: 0,\n\t\t\t\t\t\t\tBTTIsPureAction: true,\n\t\t\t\t\t\t\tBTTPredefinedActionType: 153,\n\t\t\t\t\t\t\tBTTPredefinedActionName: \"Move Mouse To Position\",\n\t\t\t\t\t\t\tBTTAdditionalActionData: {\n\t\t\t\t\t\t\t\tBTTMouseMoveX: action.x,\n\t\t\t\t\t\t\t\tBTTMouseMoveAnchor: 0,\n\t\t\t\t\t\t\t\tBTTMouseMoveY: -action.y,\n\t\t\t\t\t\t\t\tBTTMouseMoveDuration: waitMouseMoveDuration,\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t};\n\n\t\t\t\t\t\tawait trigger_action({\n\t\t\t\t\t\t\tjson: JSON.stringify(moveMouseActionDefinition),\n\t\t\t\t\t\t\twait_for_reply: true,\n\t\t\t\t\t\t});\n\n\t\t\t\t\t\tlet waitActionDefinition = {\n\t\t\t\t\t\t\tBTTActionCategory: 0,\n\t\t\t\t\t\t\tBTTIsPureAction: true,\n\t\t\t\t\t\t\tBTTPredefinedActionType: 345,\n\t\t\t\t\t\t\tBTTPredefinedActionName:\n\t\t\t\t\t\t\t\t\"Pause Execution  or  Delay Next Action (async  or  not blocking)\",\n\t\t\t\t\t\t\tBTTDelayNextActionBy: String(\n\t\t\t\t\t\t\t\twaitMouseMoveDuration + 0.5\n\t\t\t\t\t\t\t),\n\t\t\t\t\t\t};\n\n\t\t\t\t\t\tawait trigger_action({\n\t\t\t\t\t\t\tjson: JSON.stringify(waitActionDefinition),\n\t\t\t\t\t\t\twait_for_reply: true,\n\t\t\t\t\t\t});\n\n\t\t\t\t\t\tlet clickActionDefinition = {\n\t\t\t\t\t\t\tBTTActionCategory: 0,\n\t\t\t\t\t\t\tBTTIsPureAction: true,\n\t\t\t\t\t\t\tBTTPredefinedActionType: 3,\n\t\t\t\t\t\t\tBTTPredefinedActionName:\n\t\t\t\t\t\t\t\t\"Left Click (At Current Mouse Position)\",\n\t\t\t\t\t\t};\n\n\t\t\t\t\t\tawait trigger_action({\n\t\t\t\t\t\t\tjson: JSON.stringify(clickActionDefinition),\n\t\t\t\t\t\t\twait_for_reply: true,\n\t\t\t\t\t\t});\n\n\t\t\t\t\t\t\/\/ Return the coordinates if found\n\t\t\t\t\t\treturnToBTT(JSON.stringify(action));\n\t\t\t\t\t\treturn;\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\t\/\/ If we reach here, no click action was found\n\t\t\tconsole.log(`No click action found in response for: ${prompt}`);\n\t\t\treturnToBTT(\n\t\t\t\tJSON.stringify({\n\t\t\t\t\tsuccess: false,\n\t\t\t\t\terror: `Could not identify ${prompt}`,\n\t\t\t\t})\n\t\t\t);\n\t\t} catch (e) {\n\t\t\tconsole.error(`Error extracting coordinates: ${e.message}`);\n\t\t\treturnToBTT(\n\t\t\t\tJSON.stringify({\n\t\t\t\t\tsuccess: false,\n\t\t\t\t\terror: `Error extracting coordinates: ${e.message}`,\n\t\t\t\t})\n\t\t\t);\n\t\t}\n\t} catch (error) {\n\t\treturnToBTT(\n\t\t\tJSON.stringify({\n\t\t\t\tsuccess: false,\n\t\t\t\terror: error.message,\n\t\t\t})\n\t\t);\n\t}\n})();\n",
          "BTTActionJSRunInSeparateContext" : false,
          "BTTAppleScriptUsePath" : false,
          "BTTScriptLocation" : 0
        },
        "BTTRealJavaScriptString" : "(async () => {\n\tlet OPENAI_API_KEY = await get_keychain_item(\"BTT_OPENAI_API_KEY\");\n\n\tlet screenWidth = await get_number_variable({\n\t\tvariable_name: \"focused_screen_width\",\n\t});\n\n\tlet screenHeight = await get_number_variable({\n\t\tvariable_name: \"focused_screen_height\",\n\t});\n\n\tlet screenShotScript = `screencapture -cC`;\n\n\tawait runShellScript({ script: screenShotScript });\n\n\tlet clipboardBase64 = await get_clipboard_content({\n\t\tformat: \"public.png\",\n\t\tasBase64: true,\n\t});\n\n\t\/\/ Get the prompt from BTT or use a default\n\tlet prompt =\n\t\t(await get_string_variable({\n\t\t\tvariable_name: \"CLICK_ON_ELEMENT\",\n\t\t})) || \"the Apple logo\";\n\n\t\/\/ Calculate target dimensions (similar to Python implementation)\n\tconst MAX_WIDTH = 1728;\n\tlet targetWidth = screenWidth;\n\tlet targetHeight = screenHeight;\n\n\tif (screenWidth > MAX_WIDTH) {\n\t\tconst scaleFactor = MAX_WIDTH \/ screenWidth;\n\t\ttargetWidth = MAX_WIDTH;\n\t\ttargetHeight = Math.round(screenHeight * scaleFactor);\n\t}\n\n\t\/\/ Prepare the API call\n\tconst url = \"https:\/\/api.openai.com\/v1\/responses\";\n\n\tconst headers = {\n\t\tAuthorization: `Bearer ${OPENAI_API_KEY}`,\n\t\t\"Content-Type\": \"application\/json\",\n\t};\n\n\t\/\/ Check for organization ID\n\tconst OPENAI_ORG = await get_keychain_item(\"BTT_OPENAI_ORG\");\n\tif (OPENAI_ORG) {\n\t\theaders[\"Openai-Organization\"] = OPENAI_ORG;\n\t}\n\n\t\/\/ Prepare the input with screenshot and prompt\n\tconst inputItems = [\n\t\t{\n\t\t\trole: \"user\",\n\t\t\tcontent: [\n\t\t\t\t{ type: \"input_text\", text: `Find and click on: ${prompt}` },\n\t\t\t\t{\n\t\t\t\t\ttype: \"input_image\",\n\t\t\t\t\timage_url: `data:image\/png;base64,${clipboardBase64}`,\n\t\t\t\t},\n\t\t\t],\n\t\t},\n\t];\n\n\t\/\/ Tools configuration\n\tconst tools = [\n\t\t{\n\t\t\ttype: \"computer-preview\",\n\t\t\tdisplay_width: targetWidth,\n\t\t\tdisplay_height: targetHeight,\n\t\t\tenvironment: \"mac\",\n\t\t},\n\t];\n\n\t\/\/ Prepare the payload\n\tconst payload = {\n\t\tmodel: \"computer-use-preview\",\n\t\tinput: inputItems,\n\t\ttools: tools,\n\t\ttruncation: \"auto\",\n\t};\n\n\ttry {\n\t\tconst response = await fetch(url, {\n\t\t\tmethod: \"POST\",\n\t\t\theaders: headers,\n\t\t\tbody: JSON.stringify(payload),\n\t\t});\n\n\t\tif (!response.ok) {\n\t\t\tthrow new Error(\n\t\t\t\t`API Error: ${response.status} ${await response.text()}`\n\t\t\t);\n\t\t}\n\n\t\tconst responseData = await response.json();\n\n\t\t\/\/ Extract click coordinates from the response\n\t\ttry {\n\t\t\tconst output = responseData.output || [];\n\t\t\tfor (const item of output) {\n\t\t\t\tif (item.type === \"computer_call\") {\n\t\t\t\t\tconst action = item.action || {};\n\t\t\t\t\tif (action.type === \"click\") {\n\t\t\t\t\t\tlet waitMouseMoveDuration = 0.2;\n\t\t\t\t\t\tlet moveMouseActionDefinition = {\n\t\t\t\t\t\t\tBTTActionCategory: 0,\n\t\t\t\t\t\t\tBTTIsPureAction: true,\n\t\t\t\t\t\t\tBTTPredefinedActionType: 153,\n\t\t\t\t\t\t\tBTTPredefinedActionName: \"Move Mouse To Position\",\n\t\t\t\t\t\t\tBTTAdditionalActionData: {\n\t\t\t\t\t\t\t\tBTTMouseMoveX: action.x,\n\t\t\t\t\t\t\t\tBTTMouseMoveAnchor: 0,\n\t\t\t\t\t\t\t\tBTTMouseMoveY: -action.y,\n\t\t\t\t\t\t\t\tBTTMouseMoveDuration: waitMouseMoveDuration,\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t};\n\n\t\t\t\t\t\tawait trigger_action({\n\t\t\t\t\t\t\tjson: JSON.stringify(moveMouseActionDefinition),\n\t\t\t\t\t\t\twait_for_reply: true,\n\t\t\t\t\t\t});\n\n\t\t\t\t\t\tlet waitActionDefinition = {\n\t\t\t\t\t\t\tBTTActionCategory: 0,\n\t\t\t\t\t\t\tBTTIsPureAction: true,\n\t\t\t\t\t\t\tBTTPredefinedActionType: 345,\n\t\t\t\t\t\t\tBTTPredefinedActionName:\n\t\t\t\t\t\t\t\t\"Pause Execution  or  Delay Next Action (async  or  not blocking)\",\n\t\t\t\t\t\t\tBTTDelayNextActionBy: String(\n\t\t\t\t\t\t\t\twaitMouseMoveDuration + 0.5\n\t\t\t\t\t\t\t),\n\t\t\t\t\t\t};\n\n\t\t\t\t\t\tawait trigger_action({\n\t\t\t\t\t\t\tjson: JSON.stringify(waitActionDefinition),\n\t\t\t\t\t\t\twait_for_reply: true,\n\t\t\t\t\t\t});\n\n\t\t\t\t\t\tlet clickActionDefinition = {\n\t\t\t\t\t\t\tBTTActionCategory: 0,\n\t\t\t\t\t\t\tBTTIsPureAction: true,\n\t\t\t\t\t\t\tBTTPredefinedActionType: 3,\n\t\t\t\t\t\t\tBTTPredefinedActionName:\n\t\t\t\t\t\t\t\t\"Left Click (At Current Mouse Position)\",\n\t\t\t\t\t\t};\n\n\t\t\t\t\t\tawait trigger_action({\n\t\t\t\t\t\t\tjson: JSON.stringify(clickActionDefinition),\n\t\t\t\t\t\t\twait_for_reply: true,\n\t\t\t\t\t\t});\n\n\t\t\t\t\t\t\/\/ Return the coordinates if found\n\t\t\t\t\t\treturnToBTT(JSON.stringify(action));\n\t\t\t\t\t\treturn;\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\t\/\/ If we reach here, no click action was found\n\t\t\tconsole.log(`No click action found in response for: ${prompt}`);\n\t\t\treturnToBTT(\n\t\t\t\tJSON.stringify({\n\t\t\t\t\tsuccess: false,\n\t\t\t\t\terror: `Could not identify ${prompt}`,\n\t\t\t\t})\n\t\t\t);\n\t\t} catch (e) {\n\t\t\tconsole.error(`Error extracting coordinates: ${e.message}`);\n\t\t\treturnToBTT(\n\t\t\t\tJSON.stringify({\n\t\t\t\t\tsuccess: false,\n\t\t\t\t\terror: `Error extracting coordinates: ${e.message}`,\n\t\t\t\t})\n\t\t\t);\n\t\t}\n\t} catch (error) {\n\t\treturnToBTT(\n\t\t\tJSON.stringify({\n\t\t\t\tsuccess: false,\n\t\t\t\terror: error.message,\n\t\t\t})\n\t\t);\n\t}\n})();\n",
        "BTTKeyboardShortcutKeyboardType" : 0,
        "BTTEnabled" : 1,
        "BTTEnabled2" : 1,
        "BTTShortcutKeyCode" : -1,
        "BTTOrder" : 2,
        "BTTAutoAdaptToKeyboardLayout" : 0
      }
    ]
  }
]

Nice, I really need to look into this model!

I used it to click on the "Reply" button for this post :joy:

1 Like

@R_S maybe instead of searching for the knobs, you could search for this:
image

Move the mouse there, then add another "move mouse" action that moves the mouse relative to the (new) "current mouse cursor position":

ah thank you. i will definitely try this out.

very cool. do i need to buy openAI tokens for that?

the problem is, that when i change the frequency of the EQ band, the Menu floats vertically away.. left or right..

Yes, the script I wrote works by using OpenAI's API which is a paid service.

Here's the cost breakdown:

Input:
$3 USD / 1 million tokens

Output:
$12 USD / 1 million tokens

You can get an API key here: https://platform.openai.com/api-keys

What's great about this new API is that given a screenshot, the response from the API returns an output that contains either a computer_call item, just text, or other tool calls, depending on the state of the conversation. Examples of computer_call items are a click, a scroll, a key press, or any other event. This is perfect for BTT!

yep that sounds great! I'll try to integrate it into BTT next week.

1 Like