'What strange places Glasgow and Edinburgh are': Richard J Williams on Scotland's cities

"In a way," the professor of contemporary visual cultures at Edinburgh College of Art, says, looking around the redevelopment of the former Royal Infirmary of Edinburgh site, "it's a failure so far. It's not filled up in anything like the way they were hoping. It's been going since 2008 and immediately they hit the financial crash. The developers sold the site for a fraction of what they were hoping to get for it."

It's the failure, though, that he rather likes. "It's very quiet. It's not like the rest of the city. You could be anywhere in a way. It's quite anonymous. I think a lot of people appreciate that and I'll be quite sorry when it does become a success, which it will. It has become something that nobody quite predicted it would be."

Cities – their complexity and unpredictability – are one of Williams's passions and for years now he has been bringing his outspoken, often provocative (some would say contrarian) opinions to bear on the built environment through a series of academic essays and books.

Back in 2013 he managed to anger both the then Edinburgh City Council leader Andy Burns and author Irvine Welsh when he decried Edinburgh as a "dystopian wasteland" that was in a state of "urban stagnation" in an essay published in Foreign Policy magazine.

His latest book, Why Cities Look The Way They Do, looks at modern cities through the prisms of money, power (the Scottish Parliament building inevitably features), work, war (think of those anti-terror barriers on the Royal Mile), sex and culture and examines how those factors can collude and sometimes collide.

It has a global focus but Scottish cities are part of the mix and in both print and in person, Williams remains an acerbic critic of the built environment birth of the border.

That's not to say his default position is negative. The transformation of Dundee in recent years – most notably the construction of V&A Dundee – is something to be applauded, he believes.

"Dundee is in a very interesting condition at the moment. It's a bit of a mess, but what's interesting is they're not quite so hamstrung by the past. The local political culture seems to be quite vigorous and quite sane. They're certainly doing some imaginative things.

"They're pretty late to the party," he admits. "The idea of building a museum now everyone would say is a 20-year-old idea. But all credit to them. They've got the visitors in, they've got all that coverage from the Wall Street Journal. They may evolve it into something new, and as a place to watch I think it's where there are more likely to be some interesting experiments."

He is less convinced by the contemporary state of Glasgow and Edinburgh. "What strange places they are. Glasgow obviously had this fantastic period in the early 90s where it was really in the vanguard of trends. And it wasn't so much the stuff that it built, but the way it was thinking about things, the way it really got into the culture debate.

"It seems to have gone off the boil. I don't know what happened. Weak leadership maybe. There is something about the political culture that is not very helpful.

"I think Scottish cities have had quite weak leadership in the last 20 years. In Edinburgh it's very noticeable – there's plenty of money, but its ability to act and make statements, build stuff and try things out is quite limited."

He is critical of the new St James development in the capital. "It represents a particular stage in retail/capital, with a marked shift towards eating and drinking. The hotel is vulgar," he argues.

But he's approving of the newly announced plans for a metro system for Glasgow. "Some bold thinking there. Let's hope they manage it. Infrastructure is important."

The only question, he adds, is why has it taken so long?

Edinburgh has a unique problem, Williams argues, in terms of the strength of its conservation bodies.

"For me, the power of the conservation lobby is a cause for some regret here. It's an important voice, but it's one of a number of important voices.

"The new town looks beautiful if you like that sort of thing, but it's scarcely inhabited. So there's an imagination of the city which is basically a sculpture park rather than a place that's actually lived."

When you say not inhabited? "The density of habitation is incredibly low. You walk around the new town, who's there? Is anybody there? It's like a film set. I wouldn't want the whole city built in that image, in the same way I wouldn't want the whole city built in the image of the festival."

Williams has been based in Edinburgh since the turn of the century but his own urban history has taken in everywhere from Washington DC where he was born (his father was an astrophysicist) to Manchester, where he grew up, London, Madrid and Oxford.

(There was also a month spent living in a pub car park in Winchester when he was a teenager. "I had some friends who were crusties - we didn't use that word then. I was going to the Stonehenge festival.")

The city as a physical and cultural idea has changed dramatically in his lifetime. The Manchester he grew up in in the 1970s was still largely a bomb site, he recalls.

"It was absolute rubble in the 70s. It's what I remember; miles of cleared landscape and emptiness and rubble. Literally a bombsite, with these fragments of intense culture going on in them, club culture. I went to all the wrong clubs. Me and my friends went to other clubs. the people we didn't like went to the Hacienda."

But his home city has been transformed over the last 30 years. The place he returned to in the 1990s was an entirely different city.

"They hadn't built much, but there were people from all over the world there and it was very lively and it clearly had huge ambitions."

Manchester and London (since the revival of the mayoral office) have strong civic cultures interested in progressive ideas, he argues. They are cities where change is not a dirty word.

He's not sure Edinburgh, in particular, can say the same.

He has something of a love-hate relationship with the place, particularly in terms of its festival culture.

"Edinburgh is a city that wants you to look at it and it has really raised the bar in terms of culture and tourism and it does a lot of good things.

"But it's also created an experience where the expectations are very high. You get a weird monoculture in the summer and I've still not quite come to terms with it.

"I understand the economic reasons why it happens, the cultural reasons. I understand that a lot of people really like it. I find myself slightly alienated from it."

Williams's new book is a study of the global city but, as he points out, the population of London and other megacities is falling. Brexit has introduced a city-v-country binary into British political thinking. Is this book then an epitaph for the very idea of the global city, I ask him?

"I did wonder that. I don't know. We'll have to have another chat in five years' time. I don't actually think so really. Falling populations in global megacities is a thing, but how fast are they going to fall? How far? I don't know. It may be a small-term adjustment.

"At the moment it feels like an adjustment rather than a crisis. The idea of the city has come back so powerfully since the sixties. For decades people were thinking, 'it's over, it's finished.'"

That is no longer the case. The modern city has issues with gentrification, gated communities, the rise of house-owning for investment and the pricing out of working-class locals, but Williams argues that the city as an idea is still relevant and will remain so.

"For all kinds of reasons, some of them ecological. I can't imagine that really changing. My kids, their aspirations are to be in cities, whereas I had to come to that."

Why Cities Look the Way They Do, by Richard J Williams, is published by Polity Books, priced £15.99