It should be. The payer is supposed to get off the field in a timely fashion, but the rules state the ref can't spot the ball until he does.
I stand corrected...my understanding of the rule was that once they felt the defense had enough time to sub, that the ref just came off the ball and allowed the offense to run a play. However, here is the actual rule (which does call for a penalty)...
When Team A sends in its substitutes, the officials will not allow the ball to be snapped until Team B has been given an opportunity to substitute. While in the process of substitution or simulated substitution, Team A is prohibited from rushing quickly to the line of scrimmage with the obvious attempt of creating a defensive disadvantage. If the ball is ready for play, the game officials will not permit the ball to be snapped until Team B has placed substitutes in position and replaced players have left the field of play. Team B must react promptly with its substitutes.
PENALTY—Dead-ball foul. Delay of game on Team B for not completing its substitutions promptly, or delay of game on Team A for causing the play clock to expire. Five yards from the succeeding spot [S21]. (AR 3-5-2-V)