Potter has emerged as the favourite to take over at Spurs, should the Italian depart this summer.
Conte is yet to commit his future to the north London club beyond the current campaign amid links with French champions Paris St Germain.
Potter has guided the Seagulls to their highest Premier League points tally this term and underlined his credentials with Saturday’s 4-0 thrashing of Manchester United.
Yet the 46-year-old, whose Albion contract runs until 2025, insists he is happy at the Amex Stadium.
“You guys (the media) sometimes put stuff in the newspapers that’s not exactly true,” said Potter.
“What can I say? It’s got to the point now where there’s not even a vacancy and you’re getting linked to it. I don’t know what to say.
“I’ve never said I want to leave at all.
“It wasn’t so long back that I was probably getting criticism and if you’d gone on some local fan sites, I bet I wasn’t the flavour of the month.
“But I’ve always had the support of Tony (Bloom, Brighton owner) and the board and they’ve given me chance to work, so I’m grateful for that and I enjoy it here.”
Potter was linked with Tottenham last summer following the dismissal of Jose Mourinho but the role eventually went to Nuno Espirito Santo, who was later sacked and replaced by Conte.
Despite suffering six straight defeats in February and March, ninth-placed Albion are closing in on maiden top-half Premier League finish after dismantling United courtesy of goals from Moises Caicedo, Marc Cucurella, Pascal Gross and Leandro Trossard.
The south-coast club have 47 points – six more than they have managed in any of their previous seasons following promotion in 2017 – ahead of fixtures against Leeds and West Ham.
Former Ostersunds coach Potter concedes it will be tough to improve on a record-breaking campaign.
“The reality of it is the leagues are determined by resources and finance, then you’ve got to think about how you can beat that,” he replied, when asked if there is a ceiling to what Brighton can achieve.
“And how you beat that is having an idea, aligning the idea to the resources you have, sticking together, working, trying to create something that is unique, that people can buy into and enjoy.
“That’s what we achieved in Sweden for example – to get Ostersunds to the last 32 of the Europa League, I would say miracle is downplaying that a little bit.”
Potter saw Ben White join Arsenal for £50million last summer before fellow defender Dan Burn moved to Newcastle in January.
Yet he is not concerned about the prospect of further departures.
“We’ve sold £65million’s worth of players in the last two windows and we’ve managed to improve, so it can be done,” he said.
“I can’t sit here and say we’re going to keep all of our players for the next three years because it’s impossible. We have to be brave in that.
“We’re not desperate to do anything but if people want to come and pay over the odds or big, big prices for our players then that’s life.”