The Seagulls have won only one of their last six games and put in one of their worst performances of the season when losing 3-0 at home to Burnley last week.
After an encouraging start under Steven Gerrard, Saturday’s visitors Aston Villa have hit a similarly sticky patch with just one win in seven.
Potter said: “As you can see from the league generally, apart from the top three, getting points in this league consistently is very, very difficult.
“Previous to the Burnley game we have had some good performances. We are Brighton and Hove Albion, we have no divine right to win in the Premier League.
“There are bigger and better clubs than us suffering for points as well.
“The margins you need in this league haven’t gone our way. Apart from the top three, who hasn’t struggled? It is the best league in the world and it is difficult to get points.
“It is the league and it is really competitive.
“You can see what Villa are trying to do, you can see the performances, the structure of the team, on another day they could have won the games they lost.
“But the margins of this competition, this is the highest level of football in the world, so by definition it is not easy to win.”
Gerrard’s first game in charge of Villa was against Brighton in November’s reverse fixture, which the Seagulls lost 2-0.
Potter added: “We are playing in a league where everybody you play against has huge quality, the coaches are all fantastic, they are all here because they have done well in their careers and Steven is no exception to that.
“I don’t need to speak about his career, it speaks for itself and he is on a really exciting managerial journey.
“He will want to come and play well and win the points and we have to do the same. We are excited, it’s a Premier League match against a fantastic football club and we will pit our wits against them.”
Scroll down and click through to see how Graham potter's team could line-up for this one
1. Rob Sanchez
The Albion No 1 was spoken to last week following his dust-uu with Shane Duffy but will be between the sticks this one
2. Tariq Lamptey
Not quite the sizzling form of late but still a live-wire down the right
3. Lewis Dunk
He was missed against Burnley and Albion's influential skipper will return for this one following suspension
4. Joel Veltman
I expect the experienced Dutchman to get the nod ahead of Shane Duffy for this one