Swiping on touch bar also does trackpad trigger

I have three finger swipe on the touch bar set to change the brightness but it's also snapping my windows left and right because that's what I have my trackpad set up for. Is there a way to turn this off?