Everton have been linked with making a move for the Brighton head coach following the sacking of their previous boss Rafael Benitez.
Potter, who is currently self-isolating after testing positive for Covid-19, has impressed at Albion since replacing Chris Hughton when he joined from Swansea in 2019.
The Seagulls have enjoyed progress under the forward thinking 46-year-old and this term are ninth in the Premier League table and have played some attractive football.
Potter, who is under contract with Brighton until 2025, was previously linked with Tottenham and Newcastle and also the Manchester United role.
Everton are keen to rebuild after recent underwhelming spells at the helm from Carlo Ancelotti and Benitez.
The Toffees have almost spent vast sums in the transfer market of late and are said to be impressed with the results Potter has achieved on a relatively small top flight budget at Brighton.
Potter also has a reputation for developing young players and giving them their chances in the first team.
Murray however feels Potter should resist any temptation to jump ship and continue his progress at Brighton.
"Everton are a bigger club and there is a lot of potential," said Murray on Radio Five Live. "But Graham Potter's job is secure at Brighton.
"He's worked well with the people there and he's built that squad.
"He's been there a long time and I think we are only starting to see the benefit from it now and there is more to come.
"I look at Everton and there is no structure, no hierarchy - and how long do you last there?
"For me I would stay and continue his progress and in the coming months and years bigger clubs will come knocking."