It was their third 4-3 home league defeat of the season, to add to a 5-3 reverse, as they lost for the sixth time in seven home league games.
Tom Lawley hit his first hat-trick for Rebels but it was not enough to take anything out of the match as they gave away sloppy goals and trailed 4-2 at half-time. Worthing rallied in the second half and quickly got a goal back and then dominated as they pushed for an equaliser but Hythe keeper Dean Ruddy made some fine stops.
Worthing manager Lee Brace said: “We got off to a fantastic start but errors cost us.
“I don’t think it was a penalty as Matt Hurley said he didn’t touch their player and I’m disappointed with the officials. I don’t really like to talk about them but there were 200-odd people here who, I’m sure, made their mind up about them and my view is probably the same.
“The lads were fantastic, though, and we battered them for 45 minutes.”
Worthing are at Chipstead on Saturday and travel to Horsham in the Isthmian League Cup on Tuesday.