Manchester United star credits Brighton manager for saving his career
James, 21, has impressed since his move to United in the summer after a fine season in the Championship with Swansea.
But the winger admitted his confidence was at all-time low and was considering his future away from football after a disappointing loan spell at Shrewsbury.
Potter arrived at Swansea in June 2018 following their relegation from the Premier League and soon met with James.
"I think it was the down point in my career," said James, speaking to Sky Sports. "I went to Shrewsbury on loan, I came back within a month and hadn't played a game - injured and feeling sorry for myself. I didn't think I was going to make it as a footballer.
"I wanted to go back on loan but the gaffer who was there at the time - he's at Brighton now - said to me 'I feel like you've got a big chance to play here'. Coming out of that meeting, obviously I was delighted but was he just saying that to be nice? He really put the confidence back into me to play."
Suitably inspired, James went on to have an outstanding season with Swansea and has since scored three goals in the Premier League from his eight appearances at United. James has also scored two goals in his five appearances for Wales in 2019.
James added, "I still don't believe I'm a Manchester United player. You have to pinch yourself every night but you can't let that get to you. Obviously you're there for a reason, to do a job."