Even to a rookie, it was a shock on his first night on the job to discover he was with a sergeant whose knowledge of the game could be written on the back of a postage stamp.
"He announced to the Millwall elite that John Fashanu was lazy and white and then the sergeant was surprised, although I wasn't, that he was then nearly beaten to death," recalls James.
He delivers the anecdote deadpan, then starts laughing at the memory. His reaction comes from the if you didn't laugh you'd cry school of thought.
"What were some of the other situations I was in? I was with 14 of Millwall's absolutely hardest hooligans and we went into North Bank at Arsenal, in the Arsenal home end, and were surrounded by 8000 Arsenal.
"It doesn't matter how brave you are or whoever you are, that's a scary experience, especially when you're only there because you're trying to gather evidence to put the nasty people in prison."
Brutally honest, these shocking accounts of the two and a half years James spent in the 1980s as an undercover officer formed the story in his best-selling book, Running with the Firm.
He turned it into a one-man show and took it to the Edinburgh Fringe last year. After a slow burn - "on our first night we had one audience member, it was a work in progress, by the end of it we were sold out and people were queuing up" - he received five-star reviews and is now appearing at the Tron on March 19 during Glasgow Comedy Festival.
He says the performance crosses many divides and we're not talking team colours or good guys and bad guys. One of the biggest surprises has been the amount of women who go to the show.
"From 16-year-old girls to women in their 70s, they've been coming up and saying, it really wasn't what I expected," he explains.
"It's not surprising now but at the time it was. I thought my target audience was 20 to 45-year-old guys."
Despite the bravado of the title, the show isn't about being a football hooligan, it is the incredible story of a 21-year-old who lives an extraordinary life for two years.
"People are surprised when they come, they expect me to sit and read extracts from the book. It's a scripted one-man show."
This all makes perfect sense when you learn how James earned a living after leaving the force but we're getting ahead of ourselves. First, how did a man so young cope mentally and physically with the demands of the job?
"It was far too young, what got me thought was youthful arrogance," he reasons. "It's always nice to find something you're very good at, at a young age. I just wish maybe it had been something different.
"Ultimately you can't ever lose sight of the fact that it is a job. My job was to be a convincing football hooligan and I wasn't going to do that from the family enclosure.
"It's about not being seduced by what you're doing and remembering the fact that it is a job but at the same time be convincing; you're going to have to be up the front and effectively be around when certain things go on.
"What I never did was go looking for trouble, I wasn't the one that was leading the charge or running at people, but if somebody was running at me with a big baton and they were going to hit me then I think I was more than justified in hitting them before they hit me."
HE continues: "That happened on very rare occasions, there was lots of posturing, lots of talking, lots of making out you're going to do stuff but at the same time it does happen.
"And when it happens you need to be in a position where you have to defend yourself but at the same time not lose sight of the fact that you're there to do a job which is to uphold the law."
Frustratingly, no arrests were ever made after the operation James was involved in and he says he felt let down at the time. Though he admits now he could never have written the book if there were arrests. "I don't think it would be fair or appropriate on the people arrested."
After watching Gary Oldman in The Firm, Alan Clarke's 1989 acclaimed drama about football hooliganism, James decided on a new career path.
"I thought, I've just done that and convinced some of south London's finest that I'm exactly that.
"If Gary Oldman gets his lines wrong he gets to do them again, if I get mine wrong I'm in all sorts of trouble.
"So I thought I would use that to my advantage and that's why I became an actor."
James wrote the story for the film ID, starring Reece Dinsdale, ostensibly to give himself a role. But his part fell through two weeks before shooting began.
Other roles in film and television followed before James tried a new one as a successful property developer, featured in Channel 4's Grand Designs magazine and earning him a handsome profit.
"I had some businesses in the south-east which I sold in 2007 so I had an extortionate amount of money and then decided to become Richard Branson and set up a commercial airline.
"There's a show in that, I really would not recommend that to anyone.
"It was an amazing experience but not one I would wish to repeat as it cost me nearly everything.
"In fact it did, I had a pair of pants and that was it. So we had to start again. That was quite a sobering experience."
Now an ambassador for the Prince's Trust, the father of two girls encourages youngsters to follow their dreams.
"I've been very fortunate, I've lived more lives than most people will live in 100 lifetimes."
There's a reason for giving thanks he survived that first night on the job as a Millwall fan.
n James Bannon: Running with the Firm, Tron Theatre, March 19.