Jay Spearing's deflected free-kick won it for the high-flying visitors in a close game.
But Yems struggled to find the positives in the defeat against a side who are second in League Two.
Yems, who had the well-publicised hump after Tuesday's defeat at Stevenage, said: "We had a chance to have a go at them, we were 1-0 down. We were the home side and we didn’t seem to want to get at them.
"We changed it, we put Lynchy [Joel Lynch] up top, how many crosses did we get in?
"We bring Nadders [Ashley Nadesan] on, how many balls did we get over?
"It’s the first time I have felt really disappointed because they were there to have a go at.
"They are a good side, don’t get me wrong. They are clued up. They knew when to get on the floor and get a free kick, they knew everything. We just didn't compete the way I know we have got to compete if we have got any ambition to go forward.
"I don’t think their keeper had a shot to save, I don't think we tested them.
"We never gave the crowd anything to get behind.
"For me it wasn’t frustrating, it was the worst afternoon I have had as a manager here. That’s how disappointed I am.. I honestly thought we were set up to have a go at them. We didn’t and I have let the boys know that."
The Reds are back in action on Saturday when they travel to Bradford City.