It's a second defeat in four days for Danny Bloor's men, yet things had looked so good just after the interval after goals by Chris Whelpdale, Charley Kendall and Greg Luer put them into a 3-1 lead.
Wanderers, one of the sides fancied to win promotion this season but who have had a mixed start to the season, hit back strongly and scored three goals in 22 minutes to ruin Borough's hopes of an awayday success.
In the Isthmian premier, Worthing went top with a 2-1 home win over Merstham but had to come from behind to see off the in-form Surrey visitors.
In front of another crowd of 1,000-plus, John Ufuah gave Merstham a first-half lead but Ollie Pearce and Reece Meekums struck in a four-minute spell early in the second half to turn it round. That's seven wins from eight in the league for Adam Hinshelwood's team.
Bognor looked to be heading for victory at home to Folkestone when Jordy Mongoy and Harrison Brook, with his first Rocks goal, put them 2-0 up within 17 minutes. But Adam Yusuff and Jordan Wright quickly made it 2-2 and a second-half David Wright double killed off Bognor, 4-2 the final score.
In the Isthmian south east there was a very rare home defeat for Hastings, beaten 3-1 by promotion rivals Ashford United. Lanre Azeez gave the Us the lead but Tariq Ossai, Tashi-Jay Kwayie and Roberto Ratti netted in the final 20 minutes for the Kent visitors, who had lost at he Pilot Field in the FA Trophy just a few weeks ago.
Lancing are improving as the Isthmian south east season continues and a Matt Daniel strike just after the interval looked to have earned them a 1-0 home win over Faversham, only for D'jean Spence to level three minutes from the end as the points were shared.
Whitehawk won 4-0 at Corinthian and East Grinstead were held 0-0 at home to Herne Bay.
In the Sussex Senior Cup Littlehampton Town, who can't stop scoring this season, beat AFC Uckfield 5-2 and Eastbourne Town put six past Midhurst, beating them 6-0. The Eastbourne goals came from Dan Rogers (11), Aaron Hopkinson (15), Aaron Capon (33), Leke Ademola (49), Jack Samways (60) and Fletcher Holman (68).
Still in the county cup, Pagham won 4-2 at Arundel, Billingshurst beat Newhaven 3-2 and East Preston won 3-1 at Crowborough, Worthing United defeated Forest Row 2-1 and Little Common won 2-1 at home to Peacehaven. Saltdean beat Hailsham 3-0, while Selsey went out 2-1 to Horsham YMCA after extra time at the High Street Ground. Broadbridge Heath lost 3-2 to Roffey after extra time after a 2-2 draw in the 90 minutes.