Good points, Graham, I always made sure in a group class that everyone knew what was expected of whatever we were doing at that point -- and what was out-of-bounds -- of course, my group classes were rarely more than ten people at a time, all of whom knew that we were doing a recreational martial art and that there was some risk of accidental injury.
On the other hand, we sometimes trained with people from other like-minded clubs and what you suggest was always a risk in such venues. Some people don't like to lose face or appear to lose face when doing something supposedly friendly and tempers sometimes got lost. It's up to the teacher and/or senior students to take control of such situations before they escalate.
I suppose you could also argue that running the risk of getting punched or thrown by a stranger just makes the push-hands or applications training a little more relevant to self-defense; but people should also stick to fair-play and whatever rules are enforced if it's an organized event.
I used to "push" with everyone who showed an interest when I was younger; now I'm fussy about who I touch hands with... sadly, not everyone is as sweet as I am.