Seagulls head coach Graham Potter said: “Levi has enormous potential and has already shown his ability last season in the Championship and with England at various age group levels up to under-21s.
“I am really looking forward to working with him on the training pitch, and in matches over the course of the season.
“He plays mostly at centre-back, but has also played on the left as a full-back, and he is a player who is comfortable on the ball.”
Colwill has been with Chelsea since joining their under-9s squad over a decade ago and has come through the club’s academy.
Last June he signed a new four-year Chelsea contract before his move to Huddersfield.
He made 29 league appearances for the Terriers, helping them to a third-place finish in the Championship and to gain a place in an ultimately unsuccessful Championship Play-Off final.
Colwill, who will be ineligible for the two games against parent club Chelsea this season, has also captained England at under-19 level.