4 downs, without substantial other changes, would be a mistake IMO.
With all of the receivers in forward motion, the bigger field, huge endzones, and the 12th man, you'd have games that regularly end up with likely somewhere around 100 points scored per game between the two teams.
I agree there's too much punting, and that is of course because the 3 downs... A team can go for a 5 yard run, have 1 good defensive play against them, or simply drop a pass, and you're punting. Maybe 4 downs is the answer, but a lot would need to be done to decrease the average yards per successful play.
The CFL falls victim to a few factors...
One being that in many of it's markets, it is far from the most "important" team. Vancouver, Calgary, Edmonton, Winnipeg, Ottawa, Montreal and Toronto all share their market with NHL teams. Vancouver, Montreal and Toronto share with MLS teams, and in the case of Toronto, Baseball & Basketball as well. Only in Saskatchewan is the CFL the biggest game in town. It's hard to create a culture when you're not the first thing that sports fans talk about.
Second, is the schedule. Take Toronto's home schedule for example, they played:
Thursday at 7:30pm
Monday at 7:30pm
Saturday at 7:00pm
Sunday at 5:00pm
Saturday at 7:00pm
Friday at 7:30pm
Saturday at 4:00pm
Saturday at 2:00pm
One of the reasons that the NFL is so successful, is because of the religion they've built up around Sundays -- and the all-day festivity that goes with it. CFL has nothing like that.