Labour councillors have backed a recommendation that Berryknowes, Summerston and Hinshaw Street, all in Glasgow, shut down over the next year.
At present about 520 people attend specialist centres run by the council but this decision means that places will now be available for only 200 people with severe learning difficulties, at Riddrie, Carlton, The Wedge and Southbrae day centres.
The council insists the move is a result of a Scottish Government decision to assess centre users and allocate them a budget to fund services tailored to their needs.
Social care executive spokesman Matt Kerr said that, over the past decade, only one person in 10 with a learning disability had left school and opted to use council day centres.
He said money saved by the closures would remain in the social work budget for people with learning disabilities.
"With the right support, they have more control over their lives."
Green Party convener Nina Baker said many carers had concerns about their future if no day centres were available.
But Mr Kerr said they will be offered the opportunity to be assessed to ensure their own needs are being met, and a partnership will be set up involving service users, carers and service providers in the design of individual care packages for service users.
He said: "The committee totally disregarded the views of the centre users and carers and the closure decision was taken before the consultation was even started.
"We are taking legal advice and will present a petition to the Scottish Parliament asking for an enquiry."
SNP social care spokesperson Susan Aitken said: "Service users in their hundreds have made it clear they want to continue attending their day centres.
"The strain this will put on carers and service users will lead to increased work for social care services."
Green councillor Martin Bartos said: "Change needs to happen through cooperation, not conflict, after the consideration of individual circumstances."