The 31-year-old could face major surgery after jumping off of a table during rehearsals resulted in him damaging the anterior cruciate ligament in his right knee.
“If I have to miss Tokyo it would be devastating,” he told The Sun.
“I’m at the top of my career. I’ve played four major tournaments this season and I’ve won three gold medals. I’m the best player in the world at the moment.
“So this is just massively scary to have this injury. It’s only just starting to sink in.”
Bayley, who was born with Arthrogryposis – a condition which causes curved limbs, said he was in “agony” after the jump but was unaware of the potential severity of his injury.
“I can take pain, I’ve been in pain all my life, but I screamed in agony and the next day I couldn’t do anything, so alarm bells started ringing,” he said.
“It has hurt ever since. I went for a scan, and now the decision has been taken out of my hands.”
Bayley, who’s leg is in a brace, announced his departure from the show on Wednesday night’s It Takes Two. The Paralympian broke down in tears as he and professional partner Janette Manrara were shown footage from their short-lived time on the BBC ballroom show.
“It’s really gutting because I didn’t want it to end like this not properly ending it. Even if we went out next week or week four I just didn’t want it to end like this.”
Fighting back tears as he spoke about why the show meant so much to him, he said: “I just wanted to try and help people as well.”
Bayley is one of many Strictly contestants who have been forced to quit the show over injury, including British long-jumper Jade Johnson who after quitting the seventh series was also forced to pull out of the London 2012 Olympics.
Strictly Come Dancing is on BBC One, Saturday at 7pm