AI agent inside BTT ?!

[Emm_Gr]

Would it be too complicated to implement AI agent functionality within BTT? That could be awesome!

GitHub - mediar-ai/MacosUseSDK: Library to traverse and control MacOS

[fortred2]

I’ve been exploring the MacosUseSDK project and integrating it with BTT. I’m still at the exploration stage – I’ll share my code if I produce something functional.

In the meantime, take a look at this script I wrote that integrates BTT with OpenAI’s new Responses API. The script enables users to click on any element on your screen simply by describing the element in natural language.

move mouse to a specific ui element

Here's an implementation that uses OpenAI's new computer-use model with their new Responses API – it's pretty cool. See docs here: https://platform.openai.com/docs/guides/tools-computer-use I wrote this script quite quickly so I'll add improvements later... but it works! In this demo, you type in a description of the element on your screen that you want to click on and then using vision AI your computer will automatically click on it. Here's the script: (async () => { let OPENAI_API_KEY =…

[Emm_Gr]

that's awesome, im sure that can be an amazing functionality