Two care homes supporting people with learning difficulties will be axed.
Brighton and Hove City Council put forward plans to close some of its smaller homes, claiming the move will save £600,000 a year.
The local authority, which provides one of the most expensive services in the country, claimed changes need to be made if it is to combat Government cuts and improve care.
But campaigners, who collected a petition of more than 1,600 signatures, claimed it would have a detrimental effect on the lives of some of the most vulnerable people in the city.
In a close vote, the proposal was voted through on the casting vote of the chairman when it was discussed by the local authority's adult care and health committee last night.
Sue Beatty, of Brighton and Hove Unison, said: "We as a union believe it's wrong, 1,650 members of the public who signed the petition believe it's wrong and family members believe it's wrong.
"These people are the most vulnerable in the city and have their no voice of their own.
"They have to rely on those people that know them to do their best on their behalf."
The local authority claims its current portfolio of care homes are "high quality and high cost".
Its original plan, which was discussed in June, was to shut three homes.
The latest option was to close the homes in Old Shoreham and New Church Road, both in Hove, and move clients to other care homes.
Four Greens and one Conservative voted for the plan, two Conservatives and three Labour councillors voted against.
This will put between eight and 13 jobs at risk although existing permanent staff are expected to be relocated.
Other options included maintaining the current system, handing the running over to the private sector or increasing the capacity at its current homes.
While there was information on potential annual savings, opposition councillors said they were concerned there was no information on how much it would cost to bring buildings up to scratch.
Labour councillor Anne Meadows said: "I'm concerned this is just a cost-cutting exercise. It seems the council is robbing Peter to pay Paul."
But Rob Jarrett, the chairman of the council’s adult care and health committee, said: "It is not something we would choose to do. We have had our hand forced by circumstances."
Fellow Green councillor Mike Jones said: "If we make these dedicated changes now with the dedicated staff we can keep the residents' together. I worry, if we don't act, what will happen in two, three or four years time."
Fellow Green councillor and committee member Steph Powell, who was away, sent a statement saying she did not agree with the plan.
However, her place was taken by Green councillor Sue Shanks who voted to close the homes.
Conservative councillor Ken Norman, who voted for the proposals, said: "I cannot imagine any of the officers doing anything cynical when it comes to vulnerable people.
"We have to do the right thing with the money we have."