The forward netted 19 times for Worthing last season as he ended the campaign as the club's leading scorer.
Pamment appeared to have agreed on a stay at Woodside Road following discussions with boss Adam Hinshelwood last month but the frontman seems to have had a change of heart.
Strikers Ben Pope and Zack Newton have both left the club since the end of the season, with Pamment's departure leaving Worthing further short in the forward areas.