Corporate dinning rooms, healthier life styles, the automobile, and the number of people who work at home have all led to their demise.
I don't agree in part. Americans eat out and take out more than ever.
You mention healthier life styles and that may be a big factor, along with other changes in the American diet.
Many diner owners seem trapped in a time warp. The owners utterly ignore trends in low carbs, thumb their noses that those who don't want fried foods, wouldn't stoop to serve a vegan, and on and on.
There's a wonderful roadfood place in OK City that really does it right called Classen Grill: http://www.roadfood.com/Reviews/Writeup.aspx?ReviewID=3512&RefID=3606
You can have traditional "triple bypass" specials like its biscuits debris, but also many healthy choices of salads, fresh OJ, a vegan pasta, etc., etc. It also knows that most Americans these days like spicier choices like its Mexican egg scramble.
Moreover, unlike so many diners, it controls food costs by having a more limited menu, not seemingly the 100's of items of the diner nearest me. There's no way that they're accurately tracking food costs or keeping all that absolutely fresh.