h@llo.ai - Upcoming AI Features

Configuration Help & General Discussion
1

June will be BTT's month of AI additions. I will use this thread to collect ideas and discuss betas.
I call this feature h@llo.ai - a few years ago I got the llo.ai domain because I thought llo looks like a hanging sloth :sloth: (need a better logo)

image
image924×526 26.7 KB

Sometime next week I will start a beta with a simple AI Chat Interface. The fancy thing about it is that it gives the AI full control over all of BTT's functionality (or for whatever functionality you allow it to use). Basically it will be able to control anything on your Mac (but of course also be usable as a simple Ai Chat Interface).

The AI Chat Window will allow you to:

  • show / hide via action, the action allows you to configure (individually for every chat window you configure):
    • use custom API keys and custom provider URLs. This means you can use local llms or chatgpt or claude etc.
    • select which of BTT's tools shall be available
    • set custom system prompts per Ai window to instruct the Ai how to behave. I'll provide a sample system prompt that works well for controlling your Mac
  • save a specific prompt as action
  • create custom actions the AI can use via named triggers
  • allow you to provide context to the Ai (e.g. details about the active window but also functions so the AI can query that stuff itself)
  • add MCP servers you want to use
  • configure BTT triggers, floating menus etc. for you

Simple example:

image
image1920×1500 113 KB

4 Likes
3

I've been waiting for you to make this post for a while :wink:

Sign me up for the beta – I'll QA and provide user feedback.

4

Oh my... Very impressive!

5

Great!

6

Awesome!

May I suggest a concurrent discussion about your desktop icons situation?

8

not anymore! :joy:

1 Like
9

I would love to beta test this with/for you! I absolutely hate subscription services but I would totally pay a subscription for this functionality.

10

I don't plan to provide a subscription for now, instead users will need to use their own API keys or a local LLM. If there is demand for a subscription that can be purchased through BTT I'll think about it.

3 Likes
11

I would personally not pay for a subscription, although integration with LMstudio does sound cool

1 Like
12

Amazing! Cannot wait to test this out.

BTW, is there a post where you talk about the story behind the name/sloth obsession?! BTT makes us the total opposite of it so have always been curious!

13

My goal was always to make using my Mac as efficient as possible, so even a sloth (me, lazy) could get some work done :wink:
(And I always wanted to run a sloth business)

2 Likes
14

Haha! If necessity is the mother of all inventions, laziness must be the father :wink:

15

I would like a MCP server of BTT. In a way that we at least can trigger some actions trough text. Even better if we can actually create/update some popular items withou having to manually set them.

For instance, I have many keys mapped into right option + something to switch between my most used applications.

Sometime I have to bulk update each of the actions manually when I want to adjust some things, like when I recently changed my main browser and IDE.

It would be great to ask for an AI connected to BTT MCP to make the changes I wanted, since they were simple but many...

Exposing tools to fetch information is equally, if not more, useful as well... BTT already access MANY system informations, used for conditional triggering etc. We could choose to expose some of those informations for the AI to reason about specific conditions.

This would allow endless possibilities of complex rules written in plain text to be followed by the AI. Not only inside BTT UI but also other MCP compatible UIs.

Along with this suggestions, I strongly advocate in favor of being as open as possible about these, and avoid vendor lock in at all costs. That means, not only having the option to bring external AI providers to BTT, but seeking the other way around is a must have nowadays.

We're going to a path where we have more chat UI being created per day than JS frameworks.

Letting the user choose to interact with their favorite UI, regardless of what it offers, is way more attractive than yet another UI, IMHO.

Since BTT is made for Mac OS, I would also like if we could not only use Ollama as local provider, but LM Studio as well, since they provide better hardware optimizations for local LLM usage in Metal.

16

yes, I have recently worked on exposing everything via MCP. Still figuring out the best setup, but will definitely come soon!

2 Likes
17

Wow, that was a quick response, I edited my previous comment. Sorry for the length, I'm somewhat excited hahaha

18

as long as they support standard api's any provider will be able to be used. I built everything to be maximum flexible and not locked in. I want to be able to switch myself whenever necessary.

However having the built-in chat interface (in addition to MCP) also makes sense. Things like saving prompts as BTT actions etc. will be much easier that way.

1 Like
19

For BTT's UI:

What about templating BTT variables as part of the System Prompt (or even the actual prompt)?

In addition to that, having a tool in the BTT UI for RAG on the official documentation would be great for beginners.

I'm really excited with this update hahaha

20

yes that's already supported (even with the existing AI actions)

About RAG, the docs will be built into BTT for fast access, so the AI doesn't need to send queries to my webserver.

1 Like
21

It seems that you have everything already very well though then, great work. Can't wait to see it in action.

After I don't even know how many years of using BTT, you never ceases to impress.