So I decided to bring my vast experience with Tivo into this discussion. Pausing, slo-mo, all that stuff, except watching the sidelines instead of the players.
From the start, Ash was actively involved with the defense. He had the big-*** card. Again, from the start, so there was never any need to supposedly rip it away from Buh. He already had it. It was attached to his hip and often in his hand. Buh, also from the get-go, had a smaller set of notes, ones that he could place in his back pocket. That was it. Ash moved up and down the sidelines, at times waiving and yelling to the D while they were lining up, card in hand. Mind you--this is the very first series. Buh was also roaming the sidelines, much less animated, but engaged with the D. On what I believe was the first timeout--again, during the first series--the defense came to the sidelines and Ash walked into the middle of the huddle, Buh right behind him, and began talking.
This was the same throughout the first 3 series--and the 4th.
Offense was different. Anytime you saw the sidelines, it was McNulty with the card, waiving, talking, yelling.
In short, my vast Tivo expertise tells me that Ash was intimately involved with the defense from the very start and nothing changed after three series.