Its not the speed that determines the damage, but the rate of deceleration (which relates to your mass and velocity and how long everything takes to come to a rest).

The rate of deceleration is determined by what your mass & velocity is (i.e. what your kinetic energy level is) and how long (in time/distance) it takes to reduce your velocity (and hence kinetic engergy) to 0.

Its the rate of change, not the change itself which is the issue here.

When you apply your brakes to stop your car from say 20MPH you take up to 2 seconds/many tens of feet to come to a stop - and you experience a gentle level of deceleration.
Hit a solid wall as 20MPH and you come to a stop in a fraction of a second (and about 5 feet of distance) - and you experience a much greater level of deceleration as a result.
The total kinetic energy "loss" (from braking or from hitting the wall) is pretty much the same for both cases - just the rate of change is vastly larger for the second compared to the first case.

Running a car at any speed into a solid (practically immovable) concrete bridge abuttment will reduce the cars kinetic energy to zero faster than if you run the car into a properly designed collision barrier with crumple zones (for example).

While the degree of damage to the car may be the same for both collisions, the deceleration experienced by the vehicle (and generally to the same extent - its occupants/empegs etc) will be much reduced in the second collision.

Its on this principle that safety devices like Airbags, Seat Belts, car crumple zones, collision barriers etc are designed to work.

These are all designed to give the objects in the collision more time to reduce their kinetic energy. Increasing the time over which the deceleration occurs, must, for the same kinetic energy level, reduce the (average) deceleration rate.