The Seagulls are playing in the quarter-finals of the FA Cup for the first time since 1986, while United have won the competition six times since then, the last in 2016.
The sides met in the 1983 final – the only time Albion have reached the final – and United lifted the trophy after a 4-0 win in the replay, following a 2-2 draw.
Manchester United won 1-0 at Old Trafford when the sides met in the Premier League in November and Albion took a number of positives from that match.
They created several opportunities and were only defeated by Ashley Young’s strike which flew into the far top corner after deflecting off Albion defender Lewis Dunk and went down as an own goal.
United go into the game as hot favourites but Duffy believes Albion can take confidence from their performance in the league game at Old Trafford and said: “You have to take confidence from every game but they’ll know from the last game we’re better than what they probably expected us to be.
“I always say no two games are the same but you can take confidence from it. They’ll probably be a bit more prepared and we’re obviously expecting a tough game but you’ve got to believe.
“It’s a great tie to look forward to. The lads are looking forward to it and there’s no pressure on us. We’ll go to United and try to get to Wembley.
“We know we’re up against it. It’s Man United at Old Trafford and not many teams beat them but there’s no pressure on us and we’ll go there and give it our all as always.
“We’ve got to believe we can go and beat them and we do. If we go back to how we played against Arsenal, there’s no reason why we can’t beat them.
“We’re not fearing anyone. We’ve got to respect them but on our day we can beat anyone but we can’t play like we did against Everton, otherwise we’ll get punished.”
Albion have beaten United just once in 16 meetings, a 1-0 victory in November, 1982, and the Red Devils have won the past three matches 1-0.
Looking ahead to the last-eight tie, Albion striker Glenn Murray added: “It’s a tough game but it’s got to be finished on the night and we’ll be going up there to try to get through.
“It’s a bonus away from our main focus in the league.
“If you’re going to places like that, you need a little bit of luck on the evening and hopefully we’ll be carrying that at the weekend.”