How can I hold a Siri Remote button down and activate a modifier key?

As far as I can tell, I can only press and hold a button on the Siri Remote and use a modifier key DOWN or UP but not have it toggle based on the state of the button press.

Or to put it another way: when I press on a Siri Remote button, I can only have the modifier key stay down, even after I let go of the Siri Remote button.

How can I have the Siri Remote button hold down the keyboard modifier key when I am holding down the button, and release the modifier key when I let go?