Is there a way to assign a shortcut to smart invert colors?

It's the top setting in Accessbility-> Display

Will this do the trick or you need shortcut to be in BTT?

Yes, it does. Right after I posted, I saw I could do this. I actually configured a separate shortcut to trigger that shortcut, because MacOS for some reason wouldn't let me assing Ctrl+Opt+Cmd+down arrow as a shortcut that's what I wanted.