Two black men approached the 31-year-old.
One was 5ft 10in, thin and wearing a black hoodie with red piping and the other was 5ft 6in, stocky and wearing a white top with a square pattern.
Detective Constable Sarah Thurgood, from West Sussex CID, said: “Although the victim was not hurt, he has been left very shaken by the incident.
“If you were in Worthing town centre early last Saturday morning and saw these men or anything of what happened, please contact us.”
Anyone with information should call 101 quoting serial 260 of 14/12, email [email protected] or call the independent charity Crimestoppers anonymously on 0800 555.