It is partly contextual. If you are at a gym, with music going, machines whirring, and so on, the level of volume you need to overcome all that is probably 5-10 dB over ambient. You may very well be at levels exceeding 80 dB, meaning, 1 hour of exposure is a maximum.
In a quiet living room (30-40 dB ambient), you should be able to use a much lower volume and still hear the music well, and prolonged exposure is less of a concern. Anytime you are concerned that you may be listening too loud, take a 5 minute break. I gaurantee you will twist the knob down and be OK with it when you return.
With In-Ears, however, I try to limit myself to 1 hour not matter the volume. The In-Ears always heat up my drums, and moisture can build. If they are not kept clean, it is easy to get infections this way. Not to mention, they hurt my ears.