The quickest way to get disciplined, including losing your job, is to refuse a reasonable request from your manager. That's called insubordination, or disobedience of authority, and typically it's only excusable under extreme situations such as when a manager requests that you perform an unsafe or illegal act, for example.
Working in another department for two hours doesn't seem that unreasonable on the face of it. Even when you're typically assigned to a given work area or department, sometimes business needs require that a manager reassign an employee for a short period of time, as it seems he tried to do here. You have to be a little flexible, right?
The only real choice he had when you refused was to send you home for insubordination, although he probably should have clarified. Saying something like the following would have made the consequences super clear for you: "Are you refusing my management direction by not going to the X Department? Because if so, that's insubordination, and I'll have no choice but to send you home and have you come back tomorrow to talk with the big boss. Typically, insubordinate employees are fired. Do you want a minute to decide?" You could then have weighed your options. I'm betting you would have made a different choice.
Just because he did not spell it out, however, doesn't mean you should not have known. Remember, we all report to someone.