Albion’s divisional rivals are searching for a new manager after sacking Rafael Benitez following just one win from 13 top-flight games.
“I can’t comment on speculation, things that are out there in the newspapers,” said Potter, whose contract runs until 2025.
“I’m focused on my job here, very happy here at, in my opinion, one of the best-run football clubs in the Premier League, with a fantastic owner, chairman, CEO, sports director.
"It's a great environment to work in, we are an ambitious club and I will continue to do my best.
Former Ostersunds and Swansea boss Potter has been connected to a number of other clubs during his time on the south coast, including Man United, Tottenham and Newcastle.
The 46-year-old also came under pressure in the job last season after Brighton failed to win any of their opening 10 top-flight home matches.
While many within the game regard him as one of most talented English managers, Potter believes his growing reputation is down to a collective effort.
Asked if he was flattered by the rumoured interest, Potter replied: “No, not really.
“It’s just the speculation or people want to write something.
“The reality is I’m at a really good football club that helped me. I’m not here because I’m that good, it’s because I’m at a good club and I have good support.
“I have people that allow me to do my job, that give me support, give me trust and these things are important, there’s an organisation around the club.
“The reality of it is we’re not as good as people think; we’re probably not as bad as people think as well when things aren’t going so well.
“You just can’t help what people write or put out there.”
Of Benitez, Potter added: "It is not nice when someone loses their job. I only met Rafa when we played Everton and he came across as a real gentleman. I obviously wish him well."
This is a good read: Brighton transfer window current state of play