There was a great study by a couple of Harvard University researchers that answered this question.
The answer basically comes down to proximity of who you live around. While it is true there are more poor white people than poor black people in America, most poor whites don't live in concentrated communities. They are spread out and in many cases live near more affulent white communities. This leads to a cultural interaction between rich and poor whites thus reducing the chances a poverty induced culture of violence arises. Poor whites by living beside more affulent whites are able to see that it's possible to advance in this society without resorting to violence.
This same is not true for poor blacks. Ulike poor whites that don't live in concentrated communities, poor black people live in concentrated communities called "ghettos". These poor concentrated communities are segregated from more affulent communities. Because these communities see no way out of the poverty they experience, this allows a culture of violence and desperation to develop.
And if you think this stuff is all theory, the same thing happened to whites in the early 1900s. Early white migrants from Europe lived in concentrated ghettos in large American cities like New York and Boston. This led to the development of large American gangs. Remember the movie Gangs of New York starting Leonardo DiCaprio? That basically shows what life was like when poor white people were concentrated into ghettos.
The government responded to the issue of white gang violence from European immigrants by creating the suburbs. This allowed poor whites to leave a cycle of violence in the inner city and move to the middle class.