Raiders had won five successive games heading into the match but the hosts touched down a converted try after just two minutes.
Worthing responded immediately as Ben Loosmore fed Matt McLean, who went over in the corner.
Two penalties moved Taunton 13-5 ahead at half-time and another penalty early in the second half increased the hosts’ lead.
Worthing hit back, though, as Finlay Coxon-Smith touched down and Joe Marchant coverted, before Alex Nielsen went over on the hour to give Worthing a 17-16 lead.
A penalty try moved Taunton back ahead, before they sealed the win with a late converted try.
Raiders drop a place to sixth in the table and host third-from-bottom Lyndey next Saturday.