Get rid of "conference refs". Everybody goes into one pool of refs.
Have a group of 10 or so former players, coaches, and refs grade them...
Divide them into 3 groups:
Power 5 (which get paid the most), Mid-Majors, and small.
They get evaluated every month and can be raised or lower level...