The power handling figure on a speaker is misleading in many ways. Generally, they do not tell you the specifics....100 watts at what fequencies? 100 watts into what ohms? 100 watts for what duration? Things like that. For example, tweeters tend to fry if a lot of power is being pushed through them for a long duration....as in several seconds. Yet they can withstand several hundred watts for fractions of a second. Since a speaker is generally made of several components (tweeter, woofer, crossover, etc.), a figure like maximum power handling of 100 watts really doesn't tell you a lot.
However, you should know that speakers generally get fried when overdriven by an underpowered amp. When the underpowered amp get pushed too far, its signal gets distorted, or clipped. Such distortion can easily destroy a speaker. When one listens to music at typical volume, an amp is usually producing no more than 5 to 10 watts, if that much. Only when the dynamics of the music need the power will the amp be called to deliever the extra power. If the amp is underpowered to reproduce that dynamic as requested, it clips and produces distortion. Think of it as owning a 300 horsepower sports car....your engine is not producing 300 horsepower all the time, right?