To be fair, they have the second most fans in the state, although they are pretty close with NC State.
http://www.publicpolicypolling.com/pdf/2015/PPP_Release_NC_40615.pdf
They have 21% of fans in NC. To compare that, Kentucky has 66% of fans in KY.
http://www.publicpolicypolling.com/pdf/2015/PPP_Release_KY_62915.pdf
At first, you'd think UK has a crazy amount more fans from their home state than does Duke. But the population of NC is roughly 10.09 million compared to Kentucky with roughly 4.43 million.
10.09 * 21% = 2,118,900 Duke fans in NC.
4.43 * 66% = 2,923,800 UK fans in KY.
Yes, UK does have roughly 800,000 more in-state fans than Duke, but it's much closer than you'd think given the percentages. Also, given that Duke, as well as UK, both have sizable national followings and I'd say that Duke does in fact have a ton of fans.