What do you mean with 'custom button'?
Did you use the {modal} tags to create a link styled like a button?
Or are you using an actual <button> element?

Could be cache.

Do you have this online so I can take a look?

You have another modal script running on your website that is conflicting with Modals.
Slimbox is loaded by the com_fabrik.

Well, you cannot use these 2 modal scripts on the same page. As they are trying to do the same thing and conflicting with each other.

You can try to change the modal class in the Modals system plugin settings. Maybe that prevents Slimbox from trying to handle the Modals links.
Or see if you can disable Slimbox in Fabrik.

Ah, sorry, ignore that. That option no longer exists and shouldn't matter anyway.
Modals gives its links the class "modal_link".
So not sure why Slimbox is causing errors for that modal.
I recommend you ask Fabrik about this.

If you check your browser error console, you can see a bunch of errors when you open the modal.
All that is outside the control of Modals.