If nobody was going millions in the hole for athletics there would be no comparisons to make, so no brain drain.
The entire Ivy League (first and foremost a football conference, and the dominant conference in the 1st half of the 20th century) downgraded athletics decades ago and suffered no brain drain. The University of Chicago, the original Monsters of the Midway, the people who built the first nuclear reactor in a squash court, abolished football in 1939 (three years prior to the reactor). I don't think there has been a brain drain at UC.
What if the money spent on athletics were, instead, spent on academic pursuits? Would that be just as useful?
I am not a proponent of abolishing college sports. But as far as I can tell, the way they are run today, the ONLY beneficiary is ESPN who pay less than it costs to operate the sports to carry them for a profit on TV. Well, that's not quite true. The NFL benefits from not having to run a Minor league, "sports administrators" and coaches earn millions of dollars, and college presidents get to act like professional sports franchise owners, flying to games in private jets and sitting in luxury boxes to watch games.
Meanwhile we are asked to dig deep to support the program.
Illigitimus non carborundum