The blaze broke out in the derelict Broomloan Road Primary School in the Govan area of Glasgow at around 2.30pm.
Five crews were sent to the scene and found the building "well alight". There are no reports of any injuries.
A service spokesman said: "Firefighters arrived to find the two-storey building well alight and the roof has subsequently collapsed.
"Crews are conducting external firefighting operations using an aerial rescue pump and two main jets. There are no reports of injuries and Police Scotland officers are in attendance to control traffic."
Sir Alex recently spoke about his time at the school in the Times Educational Supplement.
He revealed that he still keeps the belt he was beaten with by his favourite teacher at the school, Elizabeth Thomson, who inspired him to greatness.
''It's in my study," he said. "My grandchildren are terrified of it. Six from that belt and you were in absolute agony. I used to try to draw my hand away."
Despite the punishments he kept in touch with Ms Thomson throughout his career.
''When she first got to Broomloan Road Primary, she went round to the house of every student who wasn't in her class that day and said, 'If your kid isn't in school tomorrow, I'll be back at your door'.
''Maybe 2,000 teachers turned the job down, turned down that sort of challenge, but she thrived on it. She improved everyone she touched. She actively sought out challenge."