Horrible racing is the biggest reason imo.

I live in the northeast and there used to 5-6 200 lap afternoon races for the dirt mods every fall, usually packed with fans. Then it just got to be obvious that 1) the racing at night was much better 2) the promoters would keep you there for hours on end, and the 200 lappers are down to 2.

Plus, I am not sure people really like getting home on the later end of Sunday, I know I don't go to Sunday races very much anymore unless i know I have Monday off.