The Financial Conduct Authority (FCA) said only two of the 164 sales it reviewed between June 2011 and March 2013 were considered to meet the standard required overall in a sales process.
It found RBS and its retail arm NatWest failed to consider the full extent of a customer's budget when making a recommendation, while staff did not advise customers what mortgage term was appropriate for them.
The regulator said there was no evidence that there was widespread detriment to customers, although RBS and NatWest will contact 30,000 consumers so they can raise any concerns they have about the advice they received.
RBS said that in response to the regulator's findings at the end of 2012, it overhauled its mortgage sales process and re-trained all mortgage advisers.
The Financial Services Authority, the FCA's predecessor, raised concerns in November 2011 about branch and telephone sales at RBS and NatWest but it was almost a year later before the firms started to take steps to put things right.
The firms even made assurances to the FSA in July 2012 that the necessary changes were under way to address the regulator's concerns.
Tracey McDermott, director of enforcement and financial crime at the FCA, said: "Where we raise concerns with firms we expect them to take effective action to resolve them without delay. This simply failed to happen in this case."
She added: "Taking out a mortgage is one of the most important financial decisions we can make. Poor advice could cost someone their home so it's vital that the advice process is fit for purpose. Both firms failed to ensure that their customers were getting the best advice for them."
Ross McEwan, who ran the RBS retail unit from August 2012 until his promotion to group chief executive in October last year, said the mortgage advice failings were "unacceptable and should never have happened".
He added: "We have worked hard to put things right. When I joined the bank we completely overhauled our processes, and took all our mortgage advisers off the front line for an extensive period of time to get the training required."
"Today's notice shows that we still have challenges to face, but we are determined to take the steps needed to earn back our customers' trust."
In 2012, RBS was the UK's sixth largest mortgage lender with gross lending of £13.9 billion, representing an estimated market share of 9.7%.
A mystery shopping exercise by RBS and NatWest in 2012 found that sales advisers provided customers with their own predictions on the future movement of the Bank of England base lending rate.
In one case when a customer asked if rates would rise, the adviser stated: "Yes. Absolutely," and suggested that rates could reach 5.5%.
The adviser recommended a five-year fixed rate mortgage to the customer and told them: "If we don't increase rates with this double dip recession the economy is in dire straits. Rates will rise. If you take a two-year deal then rates will be higher after this period."
The sales adviser was subsequently stopped from selling to the public by RBS but the FCA said in its report today that such conduct was highly inappropriate and gave rise to a very high likelihood of unsuitable advice.
The fine represents the latest in a string of financial hits the bank has faced including fines and compensation pay-outs in the wake of a series of scandals.
These include £3.25 billion to cover payment protection insurance mis-selling and £1.3 billion for interest rate swaps - complex financial products which were sold to small firms.
It has faced hundreds of millions of pounds in fines as part of the Libor rate-rigging scandal, and also paid out to settle sanctions-busting allegations with US authorities.
RBS made a loss of £8.2 billion last year which included making provisions for past scandals as well as the cost of setting up its internal "bad bank" to dispose of unwanted toxic assets.
But it swung to a pre-tax profit of £2.65 billion for the first half of this year on the back of the resurgent economy.