Ross Walters, 53, said he had gone for a swim during his lunch break on Tuesday (October 16) when he noticed the large animal keeping him company.
At first, he said he thought it was a buoy or another swimmer until it dropped below the water and resurfaced closer to the shore.
He said: "It took a few moments for the penny to drop. I thought it was a porpoise as I have seen a load of them while I was growing up in Wales. When I realised it was a shark, I swam out of there so quickly.
"It was huge. I cannot tell you how big it was.
"I'm aware that basking sharks are vegetarians but you only need to be wrong once. It's like an elephant - they're vegetarian, but they could still kill you."
Ross, who has lived in Hastings for ten years, said he was concerned nobody would believe him so once he returned to the shore he grabbed his camera to take pictures and video of the animal which can grow to as long as 40 feet.
He added: "I was the only person swimming at the time but there were loads of people pointing at it from the pier. Quite a few people would have seen it that day.
"I would say at one point it could only have been about ten to 15 yards from the shore."
In the video, the basking shark can be seen floating close to the surface of the water and remaining fairly stationary as the waves ride over it.
Ross added: "These sharks are the second biggest sharks in the world. They are essentially the aqua version of cows, moving slowly through the water grabbing the plankton. It had been moving that day though because that is when it shocked me.
"I wouldn't say the whole thing was especially traumatic but, being the only swimmer, it was a bit reminiscent of the classic scene in Jaws.
"I didn't sleep particularly well that night - it was so out of the ordinary."
The experience has not stopped Ross from swimming in the sea but he admits he has chosen to swim closer to the rocks to avoid another run-in with a large animal.