This is and extremely astute observation and I think has a lot to do with it. In 1980 the Under 18 age group made up 28.1% of the population and the 45-64 group made up 19.6% (according to the U.S. Census). By 2010, that had shifted to 24.2% under 18 and 26.1% 45-64. That trend isn't slowing down. We are just getting older as a society and it shows up in many places.

I have also told many people in my life that I felt there was a "racing gene". I am a fan because my father was a fan. He was a fan because my grandfather was. My son is a fan because I am. I honestly don't know a whole lot of race fans whose parents weren't also fans. And I think this is true for racing way more than for any other sport. So when you stop taking your kids to races, you are giving the gene a chance to die out. And once it's gone, it's difficult for it to regenerate.