With the Hornets leading 2-1 with 86 minutes on the clock, Harry Bradbrook drew the visitors level before Jamie Odegah struck with a minute to go to spark delirium amongst the travelling Claret supporters.
Earlier, Adam Crowther had put Chelmsford ahead inside the first 10 minutes but the Hornets roared back in the second half.
Shamir Fenelon equalised for Horsham on 64 minutes before James Hammond fired the hosts in front from the penalty spot with ten minutes remaining.
But the Hornets could not hold on as Chelmsford conjured two moments of magic at the death to steal all three points.
Defeat see Horsham drop out of the National League South play-off places. They now sit eighth in the table, a point behind place-above Ebbsfleet United but having played four games more.
This Saturday sees the Hornets travel to former Football League club Scunthorpe United in the fifth round of the FA Trophy.
See McCartney Martin’s pictures from Horsham v Chelmsford City on this page and the ones linked – and get Dominic Di Paola’s view on the action in the West Sussex County Times, out on Thursday.
1. Horsham FC floored by late sucker punches in roller coaster defeat to Chelmsford - the match in pictures
Horsham FC snatched defeat from the jaws of victory as two goals in the final four minutes saw Chelmsford City secure a scarcely believable National League South win. Photo: McCartney Martin
2. Horsham FC floored by late sucker punches in roller coaster defeat to Chelmsford - the match in pictures
Horsham FC snatched defeat from the jaws of victory as two goals in the final four minutes saw Chelmsford City secure a scarcely believable National League South win. Photo: McCartney Martin
3. Horsham FC floored by late sucker punches in roller coaster defeat to Chelmsford - the match in pictures
Horsham FC snatched defeat from the jaws of victory as two goals in the final four minutes saw Chelmsford City secure a scarcely believable National League South win. Photo: McCartney Martin
4. Horsham FC floored by late sucker punches in roller coaster defeat to Chelmsford - the match in pictures
Horsham FC snatched defeat from the jaws of victory as two goals in the final four minutes saw Chelmsford City secure a scarcely believable National League South win. Photo: McCartney Martin