Turns out the secret to beating Bayle with magic damage only is getting one stagger (Ghostflame Ignition deals magic with frost buildup, no physical damage)