I think SUPERDUKE and Offy22 nailed it: Lack of promotion and track prep.
In the later years it seemed Earl didn't want the track heavy for the non-wingers (for safety reasons, one has to assume), even though there are generally more, if less spectacular, crashes on a slick track. I really thought that would change when Stewart bought it, but it hasn't. I think people are tired of seeing the thing slick off during qualifying or heat racing.
I can't speak to the lack of promotion too much, but if Duke lives in Sidney and didn't hear a word about it, that doesn't bode well. Seems the days of a guy really going out and getting a crowd are over. Just my $.02.