Glasgow City Council called a meeting at Mitchell Library to talk to the families affected by the closure of Berryknowes, Summerston and Hinshaw Street centres.
The council was represented by Councillor Matt Kerr and Social Work director David Williams, but campaigners had asked for Gordon Matheson to attend.
Mr Matheson replied in a letter: "Please be assured that I receive regular updates regarding the meetings which have taken place with carers and service users.I fully support the work Councillor Kerr has done so far... it is more appropriate for him to be the main political presence in meetings with carers."
On Wednesday night Mr Kerr and Mr Williams met with around 150 of the 530 families affected by the closures.
They held a question and answer session in the Mitchell Library, while protestors wore masks of Gordon Matheson's face.
Tommy Gorman, who has petitioned to save the centres, said: "Mr Matheson's refusal to meet with us shows that he places people with learning difficulties as low value citizens.
"Many left the meeting in tears as they were upset at how patronising the council representatives were to them and over the refusal to reflect on our proposal of a moratorium before closing the centre. But this has galvanised us. It has made us more determined than ever to keep fighting."
The council says the introduction of a new system of funding care called personalisation means the three centres have to close.
The hundreds of people affected will instead get personal budgets which will allow them to fund services directly suited to their needs.
A council spokesman: "It was always made clear that Councillor Kerr and Social Work director David Williams would attend on behalf of the council.
"The meeting provided an opportunity to discuss the reasons behind the reforms to our day services.
"The Scottish Government's plans for personalisation means social care changing fundamentally and it is up to the council to manage that change.
"Also, over the last ten years 90% of young people eligible to access our learning disabilities day services have opted out of that support and that means the current system is not sustainable.
"We would urge carers and service users to get involved in the new public social partnership to let them shape services to fit their own lives."