The three finger swipe in BTT only triggers if the systems reports exactly 3 fingers touching the trackpad.
So maybe you are releasing the fourth finger at some point but continue swiping?
I'd recommend to check via live view, that often helps.
In general BTT does not know anything about the system gestures, thus unfortunately the only way to make everything work in combination is to disable the system gestures.