The funnyman who starred in television comedy The IT Crowd before becoming a director says there would "have to be an outbreak of influenza" for him to cast himself in one of his own projects. As for the fact he was once labelled the 'coolest man in London' by NME magazine, that just makes him wince.
"I think that was a typo - it was the coldest man in London, the most emotionally distant man," says the softly-spoken 36-year-old, looking not unlike his IT Crowd alter ego Moss, in a brown suit, patterned shirt, dark tie and thick-rimmed glasses is constantly pushing upwards.
He might be modest but Ayoade, whose film-making debut Submarine earned him a Bafta nomination and a British Independent Film Award for Best Screenplay, is clearly discerning about the work he takes on.
The producers of his latest project, The Double, had to stalk him to get him interested in working with them.
"I didn't think it would be possible to direct films at all. I hoped to write, and I guess that is still the thing I do mainly. This directing feels very unexpected," says Ayoade, the son of a Nigerian father and Norwegian mother. He studied law at Cambridge, despite knowing the degree was not for him.
"I'm an idiot and a coward. The two things - idiocy and cowardice - have accompanied me since pre-school. And fecklessness," says Ayoade, as to why he stuck with the course.
He did get to flex his creative muscles, however, and was president of the university's famous Footlights society before winning a Perrier Comedy Award at the Edinburgh Fringe in 2001. He went on to land roles in television comedies The Mighty Boosh and The IT Crowd, and made music videos for the Arctic Monkeys and Vampire Weekend before releasing Submarine in 2010.
The Double, his second directorial feature film, is based on Fyodor Dostoevsky's 1846 book of the same name. It stars The Social Network's Jesse Eisenberg as Simon, a timid man whose dreary world is turned upside down by the arrival of a confident, charismatic colleague, who happens to be his physical double.
"What I liked about it was that it was very funny and sad. The central idea felt very unusual, this idea of a lonely person who just becomes so invisible that a double appears and no one notices, and they are subsumed into that person," he says.
As the film progresses, Simon's co-worker ends up taking credit for all his work and, worse still, romances the woman he is infatuated with, played by Australian actress Mia Wasikowska.
The love story was not in Dostoevsky's original novella, but Ayoade - who also co-wrote the script for the film - believed it was an important addition.
"While I completely emotionally relate to Simon as a character, and his disintegration, what causes it does not quite feel like the same cosmology that we occupy, where you could unravel that much through status at work.
"I don't know that anyone feels that way about work," he adds. "I could be wrong, because I don't have a proper job, but it does not feel as important. Not being recognised by someone you love seems far more dangerous."
The film's cast are full of praise for their director, with Eisenberg noting his "unique sensibility" and Wasikowska impressed by his "really sensitive way of dealing with actors and getting the best out of us".
Did Ayoade's own experiences as an actor help him when he came to direct?
"What these actors do is a completely different thing to what I do. I am more or less waiting for someone to finish what I am saying and then I say something, whereas they are really doing something that is very complex," says Ayoade, ever self-effacing.
"But I am aware of how difficult it is to act, through not being able to do it. So you want to create a sympathetic environment, not have people talk during their scenes and not overtly shake your head after they have finished something - things you really do see directors do."
Making a film where the two male leads are played by the same actor proved tricky logistically, and shooting most scenes at night was tiring, but Ayoade is not one to complain.
"It was physically hard, but it is just dressing up," he says, shrugging. "It always seems absurd talking about some hard shoot, no matter what it is. It is like complaining about playing cowboys and Indians."
Nor would he want to sound ungrateful by moaning about the commercial pressures of making a film.
"Any frustration is counterbalanced by being very privileged to be funded to make something, so it is a bit like saying, 'I feel really annoyed someone has an interest in how I have spent this money that is not mine'.You are very lucky to make a film, so that outweighs it in terms of overall frustrations you might have."
The future looks very bright for Ayoade, who hopes to make more films and has also landed a three-book deal with Faber and Faber ("It seems like that shouldn't happen. It seems like even one is too much," he admits).
But has he ever had his thunder stolen, like poor Simon?
"I've no real thunder to steal," he says. "Someone's borrowed my drizzle, but I have precious little clang."
With that, the most reluctantly cool man in London readjusts his spectacles one last time.