The Pointless star spoke out about his life in a candid interview broadcast on BBC Radio 4’s Desert Island Discs on Boxing Day.
The comedian and TV producer turned novelist told of his shock when his father suddenly left their home in Cuckfield, near Haywards Heath, when he was aged nine.
“It was a complete shock,” he said. “A complete surprise. There was no sign it was going to happen.
“He moved out instantly and I didn’t see him for a very long time.
“I shut a bit of myself down. It’s very painful, especially when you are a kid.”
Richard, who went to Warden Park School in Cuckfield before going on to study at Cambridge, also spoke of his pride in his mother, a teacher, and his brother Mat - bass guitarist with the chart-topping band Suede.
“I said to him: ‘You’re on Top of the Pops, mate.’ I was so proud of him.”
He described his mum Brenda as “an extraordinary woman who did extraordinary things.”
And, he said, the most important thing about his fame was that he had been able to pay back his mum.
“I was able to buy her where she now lives.”
The 51-year-old father of two also revealed that his mum was the inspiration for his debut novel The Thursday Murder Club.
And he told of his emotion after attending a book signing at a shop in Haywards Heath which he had frequented while growing up.
He also spoke of a life-long addiction to food which he strained to control, awkardness over his height - he’s 6ft 7in - and an eye condition which left him unable to see properly.
“It was like being in a fog all the time,” he said. “I couldn’t see the board at school.”
But he said that his unclear vision had an upside.
“On TV, I can’t see the autocue so I just make things up” - proving himself a master at ad-libbing.
“I’ve been able to use it to my advantage, but I’d rather be able to see well.”