A fine strike by Tommy Leigh saw Rocks take the lead before an impressive Jordy Mongoy chipped goal saw Bognor two up.
But two penalties from Tom Richardson late in the second half saw the Blues steal a worthy draw after Joe Cook was sent off for his second bookable offence in the build-up to the second penalty.
Bognor had a full bench of Jake Flannigan, Bradley Lethbridge, Alfie Lis, Ashton Leigh and Kieran Rowe. Earning starts for Bognor were Ben Mendoza, Ethan Robb, Alfie Stanley and Sam Dowridge, with Jordy Mongoy starting up front.
Archie Jones smashed the first effort at goal bit it was straight at Amadou Tangara on seven minutes.
Bognor were having a nervy start and Keaton Wood had to head wide a dangerous Stortford cross from the right.
Correy Davidson went close to a Stortford opener a couple of times and Darren Foxley curled a shot at goal which clipped the bar on 23 minutes, a let off for Bognor.
A minute later Bognor went ahead - Stanley exchanged passes and Tommy Leigh had a dig from outside the area, curling it high into the net beyond the outstretched hand of Jack Giddens.
It was better work from Bognor after the goal as Mongoy and Stanley caused problems. Richardson hit a shot straight at Tangara after he gifted the ball to the home team.
Cook was given a yellow card on 38 minutes, seemingly for dissent. Ben Mendoza saw yellow on 41 minutes as Bognor gave the ball away and his challenge was late. HT 0-1
Goal number two came for the Rocks on 50 minutes as Dowridge played it in from the right to Mongoy, who with one touch turned his defender before chipping it left-footed into the net.
Tommy Leigh went into the book on 52 minutes for a challenge as Bognor lost possession again just outside the box. Sam Robbins picked up a caution for the hosts.
Jake Flannigan replaced Mendoza on 61 minutes for Bognor after the defence blocked another direct free-kick.
Bishops Stortford headed a chance wide on 65 minutes and were not giving up. Stanley set up Mongoy to curl an effort narrowly over the bar on 67 minutes.
Wood required treatment as it appeared that he lost his contact lenses during play and so the Rocks continued with ten men temporarily.
Mongoy went down holding his left leg but did return to play. Dowridge was replaced by Bradley Lethbridge on 74 minutes as tired legs started to show.
Harvey Whyte was deemed to have tripped Richardson in the area and a penalty was awarded on 79 minutes. Richardson dispatched it confidently high into the net.
Wood was back to head out another free-kick for the hosts but disaster struck minutes later as Cook received his second yellow for a challenge following a cross. A similar penalty kick by Richardson saw him net again on 83 minutes to draw the score level.
Ashton Leigh returned from injury and came off the bench to replace Stanley on 85 minutes. Both sides had late half-chances but had to settle for a point apiece.
Bognor fans will take some heart their team took a point on the road, though there will be disappointment they could not hold on to a 2-0 lead.
The Rocks turn their attention to FA Cup on Tuesday as they travel to South Park (7.45pm).
Rocks: Tangara, Whyte (c), Davies, Robb, Cook, Wood, Dowridge (Lethbridge 74), Mendoza (Flannigan 61), Mongoy, T Leigh, Stanley (A Leigh 85). Subs Not Used: Lis, Rowe