Potter was awarded a two-year contract extension in November 2019, just six months after joining Brighton on an initial four-year deal.
Last season, the Seagulls finished 15th with 41 points and six points clear of the relegation zone, however this season they have struggled at times.
They currently sit level on points with 18th-placed Fulham and are only above the dreaded drop zone on goal difference with just one top-flight home win this campaign.
“I certainly feel the support of the hierarchy at the club, as I’ve said before,” Potter said.
“I’m fortunate that I have a fantastic chairman, Tony Bloom is amazing, and the board, so I have no complaints whatsoever.
“I also know that it’s football and these guys have got a huge responsibility and they have to do what they think is the right thing all the time because the most important thing is the club that when you’re in my position you have to understand and respect that.
“But my relationship with them is brilliant, at the same time I understand that we are where we are.
“If results are like we are then of course there’s always conversation and pressure from outside, but there’s nothing you can do to control that.”
Brighton travel to Southampton on Sunday, and Ralph Hasenhuttl’s side only ended their own sequence of nine games without a win in the Premier League by beating Sheffield United last Saturday and have just seven more points than their south coast rivals.
When asked about the pressure he was under, Potter said: “Of course, but the pressure isn’t for my own personal situation, it’s more for the responsibility that you have for the team and for the club.
“I made a promise to myself and my family when I started this job – back when we left England for Ostersund – you can’t fear the sack, you can’t be fearful of losing your job as a football manager because then I don’t think you make the right decisions, I don’t think you approach it in the right way.
“I think you’ve got to understand that and what I’ve said before, that the chairman, board, owners, they have a huge responsibility because they are responsible for the club so whatever they decide, whatever decision is made you’ve got to respect it.
“I’ve always thought like that, and then my job is to do my best but that still doesn’t mean that you don’t feel pressure, you don’t feel the responsibility, you don’t feel that strain – that’s clear because you’ve got people you care about in the club that you want to do better for and help and that’s where the pressure is.”