It will take over the franchise from First Group next April.
Serco will be responsible for operating all aspects of the Caledonian Sleeper including marketing, sales, passenger services, station facilities and fleet maintenance.
It plans to invest more than £100 million in new rolling stock which is expected to be introduced by the summer of 2018, part-funded by a £60 million capital grant from Scottish ministers.
The company said the new fleet, to be built by rolling stock manuracturer CAF, will offer en-suite berths, pod flatbeds and a brasserie-style club car.
The sleeper connects London with Aberdeen, Edinburgh, Fort William, Glasgow, Inverness and many other of Scotland's main towns and cities.
Serco Group chief executive Rupert Soames said: "Serco has a strong track record of providing high quality and efficient rail services around the world, as well as experience in managing large-scale tourism-focused transport services.
"I am delighted that we can use this unique blend of expertise to transform the iconic Caledonian Sleeper into a modern, high quality hospitality service to make Scotland proud.
"Serco has a very constructive relationship with many parts of Scottish Government and already operates essential public transport and many other services.
"With 3,000 Serco employees living and working in Scotland, we are tremendously excited to be delivering these new developments and meeting the vision of Transport Scotland."
Total revenue to Serco over the 15-year franchise period is estimated at up to £800 million, of which approximately £180 million will be in the form of franchise payments.
International service company Serco said customers will be able to book through a new website with a much broader range of fares, earlier boarding of the trains, the ability to manage their journey virtually, and use of many other new service features.
FirstGroup currently runs the sleeper as part of its ScotRail franchise.
FirstGroup chief executive Tim O'Toole said: "I would like to thank all of our employees who have worked hard to deliver the best possible service for Caledonian Sleeper customers over the past 10 years. We submitted a strong bid which would have offered further high quality services for passengers and a good return for taxpayers.
"As a Scottish company, we remain committed to providing more and better rail services for Scotland.
"We have operated the current ScotRail franchise since 2004, providing more than 2,300 services per day, and over that time the franchise has achieved record highs in passenger numbers and also in customer satisfaction scores. We are shortlisted for the next generation of the ScotRail franchise due to be awarded by Transport Scotland in the coming months.
"We will continue to provide sleeper passengers with the best possible service until the new franchise commences, and will work closely with the new operator to ensure a seamless handover. We are tremendously proud to have provided this important service, providing a key link between Scotland and London, for the last decade and wish the team every success for the future."
WWF Scotland director Lang Banks said: "The sleeper service is a vital tool in reducing the number of flights within the UK and the climate emissions those flights generate.
"However, as a regular user of the existing service myself, it has long been clear for anyone to see that if more people were to be enticed to take the train, that major improvements to both the rolling stock and onboard facilities would be needed."
Rail, Maritime and Transport union acting general secretary Mick Cash said: "RMT is deeply concerned that this important and prestigious service has been awarded to Serco, a company with a truly shocking track record in the delivery of public services.
"Quite frankly, with their appalling list of failures in the UK and globally they should never have even been considered as contenders for the Scottish sleeper service.
"Serco is a company that has a reputation for promising the earth and delivering quite the opposite as they seek to maximise profits and sweat their assets for every single penny piece.
"RMT will be meeting with the other rail unions but I can make it crystal clear that if there is any threat to the jobs and conditions of the staff transferring across from Scotrail to Serco it will be met with fierce resistance including the use of industrial action if required."
Transport Minister Keith Brown MSP said: "The successful bidder has reflected the Scottish Government's objectives to transform this iconic Scottish rail service."
He added: "The contract secures the future of the Caledonian Sleeper, ensuring it delivers a service fit for the 21st century and that it provides - as it has done for over 100 years - a unique, valued and high-profile overnight service between Scotland and London."
Transport Scotland said that a partnership with luxury hospitality firm Inverlochy Castle Management International (ICMI) and Michelin-starred chef Albert Roux will introduce hotel-style service and an improved all-round travelling experience for passengers.
Mick Whelan, general secretary of Aslef, the train drivers' union, called on Serco to guarantee that the terms and conditions of train drivers will not be a target when it takes over the franchise.
He said: "Aslef has natural concerns that, up to now, no one has taken on board the concerns of the staff who are currently involved, and their futures. We shall be contacting all parties on behalf of our members nationally who have delivered this service for years."
Manuel Cortes, leader of the TSSA rail union, said: "This decision flies in the face of everything Alex Salmond has been saying about an independent, financially strong Scotland.
"He has taken the contract away from a Perth-based Scottish firm and handed it to an English-based firm which will be given a £60 million subsidy to build new trains in Spain. How does that create any new jobs in his dreamland of an independent Scotland?
"Once again, the Nats are guilty of hypocrisy."