This thing's much faster and probably more performant than audio switcher widgets that use applescript, because it doesn't depend on opening and then closing the system preferences window in the background.
Setup is fairly simple:
brew install switchaudio-osx. This is needed to bypass the preference pane and select audio devices via shell script.
The buttons refresh each other when you press them to show which audio devices are selected, and you may need to update some UUIDs in an additional 'run applescript' action attached to each button.
The preset comes with slots for three different devices; you can easily add more by copying and pasting the buttons and editing a number in each of their shell scripts.
All scripts in the preset are annotated to point out where to make these edits.
Thanks to @burni for the idea!