The city council is consulting on moves to shut centres at Berryknowes, Summerston, and Hinshaw Street at Queen's Cross, which each accommodate about 130 people on weekdays.
Social work bosses say reforming how the service operates will provide more opportunities for people in mainstream centres, allow them to get involved in other activities and save money.
But Carers Planit, which was set up to work with those providing services on learning disability issues, said: "Those who use the centres, carers and families affected are totally opposed to the proposals being rushed through.
"The cuts being made are fundamental and irreversible and will have a long-lasting and negative impact on the learning disability community.
"Not even the least sceptical could describe the current process as a consultation. In the interest of fairness and transparency, this consultation should be carried out by an organisation independent of Glasgow City Council.
"Many carers are convinced the council is more concerned about the welfare of the statues in George Square than the bleak future facing the learning disability community across the city."
The group is to contact every city councillor, MP and MSP to seek their support against the planned cuts.
The carers have also won the backing of Sam McCartney, of trade union Unison.
The Glasgow branch service and conditions officer said: "The whole process is moving too quickly.
"We are totally opposed to downgrading the service because that is what it is. The council plans to downgrade one of the best quality groups of staff in any service – staff who deal with some of the most vulnerable people.
"The council is shirking its responsibilities to carers, families and staff."
A council spokesman said: "A detailed plan has been devised to ensure the widest number of those involved are consulted on the proposals.
"We would encourage Carers Planit to bring forward its proposals by taking part in the consultation.
"Our consultation has already been welcomed by Enable Scotland, which represents a large number of those who use the centres and carers."