They left it late - very late - but Tom Kavanagh's superb 87th minute free kick flew into the top corner to see off the National League visitors.
Now Horsham fans can go to bed dreaming of a potential meeting with a Sheffield Wednesday, Sunderland, Portsmouth or Charlton in Sunday afternoon's draw.
See footage of the winner taken by our man Sam Pole in the video above.
Match report and reaction to follow...
Hastings didn't make it - going down 3-1 at Maidenhead. But there will be another Sussex team in the first round draw, as this is where League Two Crawley come in. So that's a 79/1 chance of a Horsham v Crawley match-up. Anyone fancy that...?
Apart from the glory of the day Horsham can also reflect on a nice bit of prize money. They get £9,375 for winning in this final qualifying round while if they can prosper in the next round they'll bag a further £22,629.
See news of Horsham's first round draw on this website on Sunday afternoon