There's no anger at being called out, just disappointment over that game being done for now. His head doesn't seem to have taken any damage from his hits, but again, he's a Caster and while there are some who are physically strong Casters, Merlin is not one of them, so it's not as if he can do any damage.

"Saber? How interesting, if the tables were turned, you might have hurt me if you'd done the same as I." There's the slightest hint of a smile on his lips as he doesn't answer Saber quite yet, catching the frisbee with some minor difficulty before throwing it back without a word of direction. "Why, would you like to put your trust in me, Saber? That's quite the bad idea, trusting a stranger blindly."