Are these SF Symbol icons? By default BTT uses the image size given by the image, but you can set a custom size. But maybe something is wrong with the auto-invert for some images.
The old one won't go away if you prefer to specify in text.
Also don't see a option to set an icon size. there is one back in the main preference window for "Menubar Item: " but not one for the entries into the customer menu
A bit confused; In my last screenshot, one of the items ( at the top, the Named Trigger: Screenshot.. )is clicked on but there is no icon size option, just Icon selection and other options.
Oh wait, how did you get the triggers to show up on the left? I think the current alpha does not yet allow to paste them from somewhere else. I'll look into that soon.
Nevermind; I see what might be the issue. I copied the items/actions from the (OLD) custom menu, and pasted them into the (NEW) one.. but they are different "items" (not sure what to call them) .. Instead of "Named Triggers", its a "Context Menu Item"
You select some item in macOS (e.g. text in a text editor), then trigger the action. BTT will try to open the context menu for that selected item then.