Best Answer - Chosen by Voters
For any amplifier and speaker combination the volume setting to deliver a certain loudness from the speakers is basically fixed, but will depend on a few variables. Assuming you are using adequate speaker wire and there are no poor connections, the biggest variable is that the larger the room the higher the volume setting required. Speaker placement can also have some effect ... particulalry re. bass.
That said, your issue is probably that your amplifier is underpowered for the speakers. Note that speakers vary widely in efficiency. The same power input into one speaker may generate a much louder output than another. Your speakers may be low efficiency. Look for a spec in the form "89 dB @ 1 watt" for an indication of efficiency. The higher the number the louder it will be.
Finally, unless you have the volume turned up past about 80-90% you are unlikely to adversely impact anything.