Top scorer Ollie Pearce gave the Mackerel Men the lead from the spot after Reece Meekums but two nice finishes by Lorent Tolaj helped Albion's U23s into a 4-1 lead by the midway point of the second half.
Joel Colbran netted Worthing's second but there was no cup at the end of the night to add to Worthing's Isthmian premier title.
But they can look back on their run to the final - and their gutsy performance on the night - with pride, and they of course have much to look forward to as they prepare for next season's National League campaign.
Having already won the Isthmian title, Adam Hinshelwood's team were keen to add the county cup to the trophy cabinet - but the manager had warned they will be 'massive underdogs' against the full-time youngsters from the Rebels' Premier League near neighbours.
The Sussex Senior Cup is the longest running cup competition in the county having first been played for in the 1882/83 season, shortly after the founding of the Sussex County FA.
It's been 23 seasons since Worthing last won the trophy - a 3-0 final win over Hastings in 1999 their last triumph. They've lost to Brighton (twice) and Eastbourne in finals since then.
No-one has won it since Bognor Regis Town lifted it in 2019 by beating Burgess Hill 2-1 in extra-time. The 19-20 and 20-21 competitions both fell victim to the Covid pandemic, the former competition abandoned after teams had reached the latter stages.
2 mins - Reece Meekums fires in the first shot of the evening after some early Worthing pressure but it's deflected over for a corner which Aarran Racine heads just wide. Good start from the Mackerel Men.
4 mins- Harrison Male makes a sharp save from Ben Jackson's header across goal following a Jack Spong free-kick.
9 mins - Pat Webber heads clear from just beneath the Worthing bar after Brighton's best move of the game so far has Male beaten.
20 mins - A lovely touch from Ollie Pearce gives Calum Kealy the chance to run at goal but his attempt to find a team-mate is cut out - but the move gets the Worthing fans in good voice.
24 mins -1-0 Worthing. Pearce flicks a lay-off into the path of Meekums and he's sent flying by defender Jack Hinchy. It's a clear penalty and after Hinchy is booked, Pearce sends James Beadle the wrong way from the spot.
25 - Male saves well when Lorent Tolaj is sent through one on one - and Jasper Pattenden heads the follow-up effort clear from a few yards out.
30 mins - 1-1 - Brighton are level thanks to a fantastic first-time strike by Lorent Tolaj after a pass by Zac Sturge. Male could do nothing about it.
33 mins - Albion have come back strongly after conceding and from a left-wing cross Todd Miller's head skims off the top of the bar. It's an extremely open game.
39 mins - Tolaj again! 2-1 to Brighton as the Swiss frontman, who has just spent three months on loan with League One Cambridge, runs on to a through ball and slots comfortably past Male. Great comeback from Albion after going behind.
44 mins - It's a lively end to the half as Pearce is booked then Joel Colbran fires over after taking a touch to control a Pearce cross to the far post.
Second half - Dayshonne Golding replaces Will Seager for the restart.
51 - Sub Golding should have done better with a header from a curling cross by Colbran. Worthing starting the second half on the front foot.
53 - Pattenden surges down the right, beats his man but shoots just high and wide of the near post when he may have done better to cross.
58 - Brighton get a third as Jack Spong powers into the area and squares to an unmarked Cam Peupion who squeezes a shot under Male. Harsh on Worthing to be 3-1 down after their bright start to the second half.
64 - Clinical:Andy Moran turns away from his man in the box and fire No4 past Male after a quick Brighton move down the right.
69 -Ollie Starkey replace Marvin Armstrong for Worthing.
72 - There's a Hinshelwood on the pitch. Not Adam but his son Jack, who is on for Albion in place of Jack Hinchy.
74 - A couple of half-chances that fall to Pearce are the best Worthing can do by way of a response to going 4-1 down.
75 - Benecio Baker-Boaltey replaces Todd Miller for Brighton. Two minutes later the sub completes miscues a shot from 12 yards after Tolaj unselfishly squares to him.
80 - Jesse Starkey is booked and Worthing's fans are calling for a penalty as Golding is manhandled when a ball goes into the Brighton box.
82 - Colbran gets one back for Worthing with a nice flick past the keeper when a corner is not cleared - 4-2 - is the comeback on?
83 - Alex Parsons - who has missed virtually the whole season with a serious injury - replaces Reece Meekums.
85 - Double scorer Tolaj is replaced by Marcus Ifill.
Injury time - Sad scenes as the stretcher comes out for what looks like it might be a serious injury to Albion's Baker-Boaltey. He is lifted on to it and taken off for treatment.
90 + 10 - Pattenden is red-carded after bringing down Ifill as he runs through on goa, conceding a free-kick just outisde the box. Spong's free kick is saved.
Worthing: Male, Colbran, Webber, Seager, Racine, Armstrong, J Starkey, Kealy, Pearce, Meekums, Pattenden. Subs: Budd, Cox, Golding, O Starkey, Parsons.
Brighton: Beadle, Peupion, Furlong, Turns, Jackson, Spong, Miller, Hinchy, Tolaj, Moran, Sturge. Subs: Baker-Boatley, Talley, Ifill, Hinshelwood, Chouchane.
Ref: Dan Austin