Ally McCoist's side looked to have snatched a point at the death in the SPFL Championship match at Ibrox yesterday when Nicky Law netted.
But the visitors netted within seconds of the restart with Osman Sow slotting a shot beyond Gers keeper Cammy Bell.
Neilson said: "For the first hour of the game they made it really difficult for us and we did ride our luck a little bit.
"But we also created a couple as well. But we knew that after an hour we would be able to start playing in the areas we wanted to play in.
"When they scored in the 91st minute I thought: 'It would be the easiest thing in the world for us to fold'.
"I've been here before and seen it and it ends up 2-1, 3-1 or 4-1. but we turned it around to show the character to come back and win it.
"We showed character to turn to go back out and grab the winner. But there is a long, long way to go this season.
The one downside on a good day for Hearts was the injury to goalkeeper Neil Alexander.
The former Gers keeper was carried off on a stretcher in the second half after clashing with his own defender Alim Ozturk, replaced by another former Ibrox keeper Scott Gallacher.
Neilson said: "It looks a bad one to be honest. He is away to the Southern General.
"He has a problem with his cheekbone. The doctor thinks it could be broken so it could be four to six weeks minimum.
"But I was pleased for Scott. He came in a few weeks ago. He wants to play football and he has the opportunity now."