Its not that the Gorosei/Imu are weak, its just that Joy Boys haki is just THAT strong, Admirals/BB/Mihawk etc etc would've gotten killed if they took that Haki