He warned independence could result in 4000 jobs being lost at the city's shipyards.
Mr Matheson made the claims before the House of Lords Economic Affairs Select Committee, which was sitting in the City Chambers.
It is conducting an investigation into the economic consequences for the UK if Scotland gains independence.
The council leader told the committee: "There is a climate of fear among businesses in the city. Many companies have told me directly that the uncertainty of constitutional change is having a negative impact on their business.
"However, the current political climate has made it almost impossible for companies to voice their concerns about Scottish independence for fear they will be singled out.
"Uncertainty is a genuine source of worry among businesses in Glasgow. We have thriving financial services and energy sectors in the city – there are 22,000 people employed in financial services alone."
Mr Matheson told committee members independence would have a massive impact on employment in Glasgow.
He said: "Four thousand jobs at Glasgow's shipyards would be lost, given the inevitability of the departure of significant employers such as BAE Systems.
"And thousands of jobs reliant on hospitality and tourism would be put in danger. One of my greatest concerns is the potential collapse of our business tourism sector.
"This is worth hundreds of millions of pounds each year to Glasgow and the majority of this business comes from the rest of the UK. Let us be clear, UK organisations do not hold UK conferences outside the UK. "
He also said that independence would have a huge impact upon public services.
He said: "The loss of the UK's AAA credit rating would have a profound impact on the cost of borrowing, reducing public sector funding and putting an unbearable strain on already stretched local government resources.
"Any reduction in our credit rating would seriously threaten the value of the Strathclyde Pension Fund which has 205,000 members. The introduction of uncertainty and risk at a time when we are trying to position the city to emerge from recession is hugely unwelcome and could lead to Scotland's cities falling behind counterparts in the UK."
