Like most posters here, I'm an avid college football fan. I like to watch ball.
My viewing habits are going to be the one or two "must see" teams that I support. After that, I'm going to want to watch a "relevant" game. It's that "relevance" that I feel is the single most important factor (almost the only factor) driving viewership numbers.
"Relevance" in this context means "how does it affect the national picture?". Does at least one of the teams have a chance to be at the top of the rankings? What are the odds of a national championship race changing loss? That, as I see it, drives viewership more than anything.
A couple of my supported teams have ultra-large fan bases. That's a good place to start, but if they're playing cruddy ball and have three losses by mid-October, their viewership numbers are unremarkable. The ultra-large fan base just can't move the needle when compared to the entire college football fan world. They don't tune in when the game is nationally irrelevant.
The broadcaster doesn't matter all that much. If undefeated Alabama plays undefeated LSU in November, we're all watching even if the game is played on FS2.
Becoming nationally relevant is therefore the trick. It's obviously better to be born on top. If you're born on top, you just have to win. For those not born on top, they have to schedule, play, and win games against those on top. Easier if your conference has those top teams built in. Otherwise, getting non-con agreements can be tough.