Why prosperity is problem

Yes, prosperity not just can be but is always a problem for the those who are nor prosperous! Imagine this scenario:
Instead of much of India, whole of India with an exception of very few services were to be engaged in agriculture, income disparity would much be much less. In other words becasue a select few can not affoard to buy just becasue they can; this would not let inflation raise making life much easier for the rest of the consumers. I will not elaborate.