The City regulator fined the bank £8.9m after it forced 22,000 customers into higher mortgage repayments to rectify a previous error by bank staff.
The Financial Conduct Authority (FCA) said Clydesdale had failed to clearly spell out customers' rights following the bank's miscalculation on the repayments on more than 42,500 mortgages.
The earlier blunder left a £21.2m shortfall in Clydesdale mortgages, with customers who had underpaid left with shortfalls on their accounts. These range from less than £20 to more than £18,000, with an average of £970.
Clydesdale, which is owned by the National Australia Bank, sent letters to customers in 2009 which suggested they had no alternative but to bring their repayments up to date.
But many could have rejected the demands to repay the shortfalls.
Clydesdale has now agreed to compensate customers who were adversely affected, resulting in a total cost to the bank of about £42m.
Customers left with shortfalls will be automatically compensated and the bank is writing to all customers affected by the blunder.
Clydesdale chief executive David Thorburn said he was "very sorry" that the situation had not been handled properly.
He said: "We should have made it clear that this was entirely our fault and that some customers may be entitled to compensation.
"Our priority is to fix this for customers as quickly as possible."
Some 14,000 customers will see an "immediate resolution", including a full apology and a refund within the next 48 hours, he said.
Other customers with more complicated cases should hear from the bank in the next couple of weeks.