You only have to stop the other team from scoring...as much as you score. Again, since the determining factor of winning is scoring more points, defense is simply a supporting dynamic to scoring more points than the other team. Thus, being more efficient on offense allows more room for error.
And yes, if you're strictly looking for elite 3 pt shooters then sometimes it is tough to find solid perimeter defenders as well (although that is exactly what Baylor currently has). But I was just referencing scoring and shooting in general. Plethora of ways to put the ball in the whole but you need guys who can create and put their selves or their teammates in positions to score. Over emphasis on length and athleticism to bolster shot blocking or defense at the cost of guys who are efficient on the offensive end is how you get the rock fight style and constantly in close games. It can work but offensive efficiency makes everything else easier.
Sidebar: I'd argue our over emphasis on shot blocker and blocking shots from the help defender gives up just as many weak side wide open put-backs or second attempts even if we "lead the nation in clocked shots". If the on ball guy can challenge block it then great but I'd rather everyone else block out and secure the miss. Help D should just be for if you're sliding over bc the other man is beat, not to have 2 (sometimes 3) guys all challenging the same shot like we do on a consistent basis.