Pagham made a bright start and had a good chance on 12 minutes when Hisham Kasimu ran into the penalty area, but the striker dragged his effort wide of the target from close range.
But it was Brighton who took the lead on 17 minutes when James Tilley chested home George Cox's cross.
Dan Cashman doubled the lead on 19 minutes with a neat finish at the back post from Daniel Mandroiu's lofted pass.
The score remained 2-0 at half-time with Albion are enjoying the majority of possession at Culver Road.
Mandroiu pulled the strings in the centre of midfield, while Cashman and Aaron Connolly looked to use their pace to stretch the play.
Ten minutes into the second half, it was 3-0 as Mandroiu fired home from the corner of the penalty area into the bottom corner.
Jordan Davies then made it 4-0 to Albion from close range on 66 minutes before Davies grabbed his second with a simple finish to make it 5-0.
Brighton equalled Crawley Town's total against Saltdean United in the other semi final when substitute Viktor Gyokeres made it 6-0 when he tapped home at the back post from Henrik Bjordal's teasing cross from the left.