The Seagulls beat Arsenal in their opening game of the restart and then held Leicester to a goalless draw, before they were on the end of a 3-0 defeat by Manchester United on Tuesday.
They head to fellow strugglers Norwich on Saturday lunchtime knowing a victory would stretch the gap to the bottom three to nine points, for a few hours at least.
Reaching 36 points with five games to go would also be a major achievement given that was Brighton's total at the end of last season.
"It's about perspective," said Potter. "Manchester United are a level above us. But if you'd have offered me four points from Arsenal, Leicester and Man United before Arsenal I'd have probably snapped your hand off.
"We were disappointed to lose, of course, but we are also positive because we know how hard points are to come by. We have to go to the next game feeling positive with the work we have done.
"It's an important game for both teams because of the stage of the season and where we are in the table. But that's not something we are surprised about.
"We've picked up four points. We were only two points above the bottom three and we've improved our position. The next game is the most important and that is Norwich.
"What points total will keep us up? I'm not thinking too much about that. If we get to 36 points I'll worry about it then.
"If we win on Saturday that gives us the same total we had last year, with 15 more points to play for. I suppose it depends on what your expectations are and where your perspective is.
"We know we've got to fight, we know we're in the Premier League, we know that every game is so tough and so difficult. It would be strange if I sat here and thought we should getting 50 or 60 points."
Potter tinkered with his line-up against United, but he denied having one eye on the Norwich game or their subsequent two fixtures, against Liverpool and Manchester City.
"It wasn't done with a view to this game, it was done with a view to getting a positive result against Manchester United. You can't write off any game in the Premier League," he added.
"It's an opportunity to take three points from a team fighting for their lives as well. We've got to play well and not worry about what may or may not happen."
A message from the Editor, Gary Shipton: In order for us to continue to provide high quality and trusted local news, I am asking you to please purchase a copy of our newspapers.
With the coronavirus lockdown having a major impact on many of our local valued advertisers - and consequently the advertising that we receive - we are more reliant than ever on you helping us to provide you with news and information by buying a copy of our newspapers.
Our journalists are highly trained and our content is independently regulated by IPSO to some of the most rigorous standards in the world. But being your eyes and ears comes at a price. So we need your support more than ever to buy our newspapers during this crisis.
Stay safe, and best wishes.