When I was a little kid in the 1960s, living in a council house in the West Midlands, I would sometimes get a gentle knock on my bedroom door in the middle of the night. It would be my dad, waking me up so we could go downstairs to the kitchen, put on the transistor radio and listen to the latest Cassius Clay fight, live from the USA.

My dad was a big boxing fan. He read The Ring magazine and was very excited about this mouthy newcomer, not only outclassing opponents but often predicting the actual round in which he would finish them off. Just getting up at three in the morning was exciting enough, but listening to the echoey voice of the commentator, the rise and fall of the crowd reaction, or jumping up whenever Cassius Clay, or Muhammad Ali as he later became, got the upper hand, was pure joy. I couldn’t sit too close to my dad as we listened because he bobbed and weaved and threw every punch the fighters threw.

Ali wasn’t like the other boxers. They always seemed to be battle-scarred street fighters who would hit and get hit. He was an artist, dancing and flicking out his lightening left jab, making them look slow and bewildered. The great boxing writer, AJ Liebling, described one of Ali’s unfortunate opponents as looking “like a man trying to fight off wasps with a shovel”.

As I grew up I watched every Ali fight, but it was more than the boxing. He was the funniest man on television. I never saw anyone delight a chat-show audience like he did. It’s hard to pin down those things that make us who we are but I was definitely thrilled by the idea of making a crowd laugh like that.

Ali knew how good he was and he loved sharing that knowledge with anyone who’d listen. He claimed he was so fast he could switch off his bedroom light and be in bed before it got dark. He said any fighter who dreamt of beating him better wake up and apologise. I loved him.

A lot of white people turned against Ali when he joined a seemingly scary organisation called the Nation of Islam and started spicing up his chat-show banter with outrage at the treatment of black people in America, but me and my dad stayed loyal.

I met Ali, as a trembling fanboy, three times in the 1990s: a book signing where he pointed out that he hadn’t been paid for use of his image on the T-shirt I was wearing, but still happily autographed it; a black-tie dinner with his former opponent, Henry Cooper; and at the premiere of a play about his life where he would only sign the pro-Islamic leaflets that he was handing out.

Frank Skinner & Larry Holmes

He died last June. I was just about to do my Saturday-morning radio show when my producer, who wasn’t aware of my depth of feeling for the man, casually mentioned that Ali was gone. Of course, I cried. I opened the show by playing a song about his great comeback fight against George Foreman: Johnny Wakelin’s In Zaire. I thought back to that council-house kitchen and my now-departed dad throwing all those nocturnal punches.

Then the BBC asked me if I’d like to make a documentary about my hero. Next thing, I was in Louisville, Kentucky, talking to his former boyhood neighbours. I wanted to know how segregation really felt. I found out in gut-wrenching detail when the lady who lived next door to young Cassius told me black people were allowed to buy clothes at the local department stores but they weren’t allowed to try them on. This was because white people wouldn’t shop at a store if there was any chance they might end up wearing a garment that had, even briefly, been worn by a black person. Ali’s sometimes shocking attacks on white society suddenly made horrible sense.

I interviewed Ali’s ex-wife, Khalilah, who told me how she met Ali when she was working in the Nation of Islam bakery and he went in to buy pastries. Nation of Islam ran a bakery! Those great bogeymen of the 1960s! It was like discovering that al-Qaeda sold stationery.

Holmes v Ali

I visited Larry Holmes, another great heavyweight champion, who fought Ali when the dancing and the lightning jab was just a memory. It was a fight most Ali fans could hardly bear to watch. Holmes loved Ali, too. He knew he had to beat the fading fighter but he didn’t want to do too much damage. He assured me he punched him with open fists, thus lessening the force of the blows. I remembered watching that fight. Those punches may not have hurt Ali but they certainly hurt me. At the end of our chat, Larry asked, “You wanna go see my statue?”, and then drove me to see the impressive bronze figure of himself, on a street about a mile from his house.

I also met the doctor who diagnosed Ali’s Parkinson’s disease. I asked him if boxing on for too long had caused the illness, a theory that most fans subscribe to. The doctor said it probably had, but the only way to know for sure was to perform an autopsy and Ali had refused permission. The doctor’s theory was that boxing had given Ali so much he didn’t want his own body to be used as evidence in the case against the sport.

Everyone I met spoke of Ali as a sparkling presence in their lives. When I visited his grave at Cave Hill Cemetery in Louisville, Kentucky, I felt some sadness but most of all I felt grateful. What a thrill it was to watch – and listen to – him fight, to see him entertain. At his last ever post-fight press conference, one journalist stood up and, instead of asking a question, said, “Thanks for giving us one hell of a ride.” I echo that sentiment.

Frank Skinner on Muhammad Ali is on Thursday at 9pm on BBC1