Glenn Murray scored the only goal on 29 minutes as Albion increased their unbeaten run to four games and moved up to eighth in the Premier League table.
The Seagulls are now seven points clear of the bottom three after 11 league games and have made an impressive start to life in the top flight.
Swansea, on the other hand, have just eight points from 11 games and have lost six of their seven home games in all competitions this season.
Clement said: "My summary is if we play at that level and that standard, we can't be expected to win a game.
"Until they scored, it was relatively even but it was very poor the way we defended the goal. It was a simple throw-in, we didn't deal with that, we didn't stop the cross into the box and then a player was unmarked. You then see a team that is really struggling with any kind of confidence at the moment.
"We've lost seven games now, six games by a goal. The margin of the results is tight but with the level of that performance I can't say we deserved anything. We had a couple of big chances - they (Brighton) defended for their lives on the line and then we had a good chance late on where wev'e hit the bar but overall the performance was not at the level required to get a point, never mind a win.
"We're professionals, we have to learn to deal with good times, bad times, times of indifference. Today, we're in a really bad moment, no question about it.
"Together, with me at the front, we have to find a way to get a victory. We might have to grind a result out somewhere, we might have to win ugly, just to stop the vicious cycle of poor performance, bad result, poor performance, bad result.
"A lot of the players in this group have been in this situation before. We have belief we can get out of it but we don't want this continuing on and on and on, we have to do something about it soon."
Swansea fans were vocal in their criticism of their team throughout the second half and boos rang out at full-time. Clement said: "No one is more disappointed and frustrated with the levels of performance than I am."