Potter, 45, has impressed in the Premier League this season and it's little surprise to his name linked with Tottenham following the dismissal of their previous manager Jose Mourinho.
Potter has guided Brighton 10 points clear of the relegation places with four matches left to play and Barber insists Potter will be at the Amex for the next campaign and for many seasons to come.
“We have no desire whatsoever to lose Graham," said Barber, speaking to Brian Owen from the Argus. "Graham is very happy here.
"We are delighted with the work he is doing, we are delighted with the work the staff around Graham are doing.
“As far as we are concerned there is absolutely no way we would allow Graham to leave at this time.”
"I don't think there is any interest in Graham's part whatsoever either. He's happy doing what he is doing and we are delighted with what he is doing for us.
"Over the last two seasons, people have seen the progress we have made in terms of the style of football we have played against teams that have substantially more expensively assembled squads than us and a very significant part of that is down to Graham's coaching and his style.
"We are delighted with him and we expect him to be here for many years to come."
Potter joined Brighton from Swansea in May 2019 and was swiftly handed a two-year contract extension - six months after being appointed. His current deal expires in 2025.
"We are aware of the media speculation but nobody has been in contact with me and I would not expect them to," Barber added. "Everybody in football knows we secured Graham on a long term contract, when a lot of people were surprised we did that.
"But it's because we believe in Graham and we believe in the quality he brings to the club.
"He is not only an excellent football coach but he is a terrific human being. That is the culture of our club and from that point f view we are delighted with him."
Read also: Where Graham Potter ranks in latest next Tottenham manager odds | Graham Potter to Tottenham: why Harry Kane Ben White and Yves Bissouma must be considered