James Tilley's second-half stunner brought the Reds level after a poor first half, which saw the visitors take the lead through Oliver Hawkins.
Yems said: "We huffed and puffed and didn't blow the house down again.
"We got what we deserved in the end.
"We had the opportunities and didn't take them.
"People have got to start learning and understanding what we are doing. If they don't, things have to change.
"They were a team of men and we weren't, simple."
Yems pointed to a lack of experience amongst his players.
"It's not just the budget, it's the experience they've got," he said of Mansfield. "We haven't got that.
"I don't want to talk about it as it's getting on my nerves."
Yems said he was disappointed that his side couldn't 'finish the game off' after drawing level so quickly into the second-half.
"We threw it away," he added. "We were on top.
"We didn't stop believing, we just didn't follow it through. They did a good job on us."
Crawley now have a ten-day break before their next game away at Walsall.
This won't be a chance for the players to rest though, according to the manager.
"We won't be breaking," Yems said. "We will be training and then we have a friendly next week.
"No one will be resting."