The Reds were leading thanks to Kwesi Appiah's header 31st minute goal when Ashley Nadesan was sent off for a second bookable offence in the 43rd minute.
Nadesan was booked for a foul in the 22nd minute before he was adjudged to have dived when brought down by keeper Cameron Dawson.
Footage shows there was not a lot of contact, but Yems said in his post-match interview with the club 'if he hadn't have jumped over the keeper's legs, he would have been sent to Redhill hospital'.
Three goals in the second half from Matt Jay, Nigel Atangana and Sam Nombe gave the Grecians the win and condemned Yems' side to a second successive home defeat.
But Yems was pleased with the effort and performance from his side and thought it would have been a great game if both sides had remained with 11 men.
When asked how much he would like to come out after a game and not talk about the match officials, he said: "Absolutely, absolutely, absolutely but unfortunately again the biggest topic of conversation is the officials.
"People don't pay to watch a performance like that."
SEE ALSO Crawley Town 1, Exeter City 3: Nadesan sent off, Kwesi scores - but who stood out for the Reds? Player ratings
Yems was also left frustrated with the explanation after a tussle in the Reds area involving Tony Craig and George Ray ended up with Jake Hessenthaler and Atangana being booked.
"To come out and say that you booked two players because you didn't know the two culprits..." he said. "There were three of them standing there like the Three Amigos without a clue and they picked the wrong people.
"Everyone in the crowd could see it.
"I had the fourth official telling me ‘you keep whinging’, well get things right. If I don’t get it right I get sacked, they get to go on to their next game. Useless.
"They must get them out of Lidl’s or something."
But Yems feels nothing is going his team's way at the moment despite the effort being put in by the players and the brilliant fans
He said: "The players played well and the crowd stuck with us. The crowd were again fantastic but it feels like I have shot an albatross of something because everything that could go wrong has gone wrong.
"We have just got to fight through it. I never feel sorry for myself. I come off best when people are writing me off and we have have our backs against the wall."