The simple answer is that it you listen at the same loudness level no matter what amp you are using, it doesn’t matter if the amp is 15 watts or 500 watts. That said, if you listen too loud with an underpowered amplifier and cause that amp to clip, you can do damage.
If you are getting this new amp in order to allow yourself to increase your volume to louder than before levels, then you have to be aware that at a certain point, you could potentially damage your speakers and yes, single drivers, generally speaking will not handle as much power as traditional speakers.