The 17-year-old, from Motherwell, was beaten by 36-year-old mother of two Sam Bailey, who was the clear favourite for the title.
The pair went head to head in front of 10,000 fans at Wembley Arena, with Nicholas performing Superman by Five For Fighting.
Prison officer Sam sang Skyscraper by Demi Lovato and was almost speechless after being told she had received more than a million votes for her performance.
That has won her a recording contract and she will also support American star Beyonce on the UK leg of her Mrs Carter world tour in February.
When Bailey was shown her new CD single by host Dermot O'Leary, she was overwhelmed. Shey said: "I just can't believe this. I'm just so grateful. Thank you so much everybody. I really am appreciative."
Bookmaker Coral said Bailey's win netted punters more than £1million after she became the most backed contestant in the programme's history.
Nicholas, a pupil at St Aidan's High, Wishaw, accepted his defeat gracefully and said he would still make it in the music industry.
He said: "If I was getting beaten by anybody, it was going to be Sam. I just want to say thank you to every single person who voted for me.
"This is not the end of me - I know I'll have an album out."
Bailey, of Leicestershire, was quick to sing her competitor's praises. "I just want to say that I love that boy to pieces and he needs to get an album out as well," she said.
"He's got the most amazing voice. I love him to death."
Last week judge and mentor Louis Walsh had expressed fears that Nicholas, who has a rare heart condition, could become ill during the competition.
However, the teenager's mother, Eileen, said she wanted her son to reach the final on his own merits and not with a sympathy vote.
Speaking before last night's winner was announced she said: "Nicholas has already won, just to get this far.
"He has grown in confidence week by week, although the public have not quite seen his real personality come through. We call him Tricky Nicky but that cheekiness has not come through. We are incredibly proud of him, though."
Bookmaker Coral has made Bailey, the 10th X Factor winner, 1-3 favourite to top the charts this Christmas.
"Sam Bailey is the punters' champion tonight after winning this year's X Factor," said Coral spokeswoman Nicola McGeady.
"Faithful fans have backed her every week to land the crown and their loyalty has certainly paid off, taking bookmakers to the cleaners."