Monsignor Leo Cushley officially became Archbishop of St Andrews and Edinburgh at a ceremony in the capital.
It was announced in July that he is replacing Cardinal O'Brien who left the post because of inappropriate sexual conduct during his religious career.
First Minister Alex Salmond and the Moderator of the General Assembly of the Church of Scotland, Lorna Hood, were among those who attended the ordination ceremony at St Mary's RC Cathedral.
Archbishop Cushley told the clergy during the special service: "As you may know, a couple of weeks ago the Holy Father called me in to see him. He wanted to know about all of you and to hear what I could tell him about my plans and priorities, and he listened and commented at length, with the sympathy and understanding of a man who had been an archbishop in a big capital city for many years - Buenos Aires.
"One of the things he communicated then and in the coming days was the idea that I should be merciful in my ministry here."
He continued: "But he explained that being merciful doesn't mean being soft. It means being gentle but also firm at the same time. This is what the Pope asked me to be for all of you.
"It is also Pope Francis' proposal for the way we priests ought to be with each other: firmly resolved to be merciful, to forgive, to be humble, to rebuild, to dialogue."
The appointment marks a return to Scotland for the 52-year-old who was born in Airdrie, North Lanarkshire, and ordained a priest in Uddingston, South Lanarkshire.
He was until recently head of the English language section of the Vatican's secretariat of state and accompanied the Pope on visits to English-speaking countries.
Archbishop Cushley was consecrated by Archbishop of Glasgow Philip Tartaglia, administrator of the St Andrews and Edinburgh archdiocese, and Archbishop Antonio Mennini, the Pope's representative in the UK.
The homily was delivered by Cardinal James Harvey from the US state of Wisconsin.
Cardinal O'Brien stepped down after 27 years at the end of February when three priests and a former priest made allegations of inappropriate behaviour against him.
He later issued an apology, saying that ''there have been times that my sexual conduct has fallen below the standards expected of me''.
Archbishop Cushley said he was ''humbled'' by his appointment on July 24.
''I know it's a delicate moment and that there is a lot to be done but with God's grace and the kind support of the clergy and people of Edinburgh, I will work cheerfully and willingly with all the energy I can muster,'' he said.
Ms Hood offered the new archbishop a "warm welcome".
She said: "I have no doubt that he will be a caring, humble and loving shepherd of his new flock."
Mr Salmond said: "I warmly congratulate Archbishop Cushley on his ordination and wish him every success in his new role. The Catholic Church plays an important role in our society, in our history and in our future.
"I look forward to building on the strong and respectful relationship we have as a Government with the Catholic Church in Scotland, and I also look forward to working with Archbishop Cushley as he begins this new period of service both in the Catholic Church and among the people of Scotland."