Esther McVey, Conservative Department of Work and Pensions minister, had refused an invitation to appear before the Welfare Reform Committee but agreed to a private meeting with members.
Now Ms McVey has cancelled her planned visit next month on June 10.
Work and Pensions secretary Iain Duncan Smith previously refused to appear at the committee and instead had an informal off the record meeting with MSPs. And Lord Freud, another DWP minister, has also refused requests to attend at Holyrood.
Now Labour and SNP MSPs have accused the Tory ministers of being scared to account for the impact their policies have on vulnerable people.
Committee Convener Michael McMahon MSP said: "We wanted a public meeting, sooner rather than later.
"Instead, we get a private meeting that's eventually cancelled. How insulting to the people of Scotland.
"Ms McVey can run but she can't hide from our committee. She has to account for the impact of her department on the most vulnerable in Scotland's society."
Kevin Stewart, SNP MSP on the committee, accused one of the DWP officials who was sent to the committee in place of a minister of talking "claptrap" when he said people should take in a lodger or move house to beat the bedroom tax.
He said: "This is extremely disappointing but sadly not a surprising decision by Esther McVey, who is treating Holyrood with contempt."
A DWP spokesman said: "We are in touch with the committee about rearranging a further ministerial meeting.
"Ministers already appear in front of the UK Select Committees as social security is a UK matter."