Lewis Macleod returned from illness to claim an early opener for the league leaders, who looked set to hold on for the points and their 12th win in a row.
But the 46,406 crowd were stunned into silence when Alexander fumbled a Stewart Leslie header into the back of his own net to gift Elgin a share of the points.
Rangers made three changes following the midweek win at Annan.
Skipper Lee McCulloch and Emilson Cribari both recovered from knocks to take their place in the starting line-up, while Macleod was also added to the side after missing the last two games with a virus.
Darren Cole missed out with an ankle injury that could end his season, while Francisco Sandaza dropped to the bench and Robbie Crawford failed to make the squad.
A minute's silence was observed ahead of kick-off to mark the anniversary of the 1971 Ibrox Disaster, when 66 fans lost their lives.
Once the match was under way, Rangers immediately set about trying to add to the 14 goals scored against Elgin in the three previous meetings between the clubs this season.
McCulloch went to ground under pressure from Jamie Duff early on but appeals for a penalty from the home support were ignored by referee John Beaton.
The Light Blues did not have to wait too long for the opener and were ahead with just nine minutes on the clock.
McCulloch won the ball in the middle of the park before releasing Macleod into the box and he coolly dispatched a low shot past Joe Malin into the back of the net.
Elgin almost levelled a minute later when a powerful drive from Daniel Moore forced a decent save from goalkeeper Neil Alexander.
Duff was fortunate to escape with a caution when he picked up the first booking after 22 minutes for a poor challenge on David Templeton.
Templeton recovered from the foul to deliver a cheeky back-heel pass into the path of Lee Wallace but the defender's low cross was unable to find McCulloch in front of goal.
Andy Little then picked out Templeton in the box and the winger tried to curl past Malin but the Elgin goalkeeper was able to gather comfortably.
The visitors were reduced to 10 men four minutes before half-time when Paul Harkins was shown a straight red card for a foul on Ian Black.
McCulloch should have claimed Rangers' second of the day just before the break when he collected Wallace's cross from the left six yards out but a poor finish allowed Malin to block easily.
Rangers dominated possession after the restart but struggled to carve out much in the way of clear chances, although Templeton was not too far away with a volley from the edge of the area that fell just over the crossbar.
Barrie McKay was thrown into the action after an hour, replacing goal-scorer Macleod, and almost immediately he called Malin into action with a shot straight at the goalkeeper.
Malin came to the rescue again, this time producing a superb save to prevent Little's point-blank header from finding the back of the net.
Kyle Hutton made way for Dean Shiels with 20 minutes to go and the Northern Irishman quickly tried his luck with a volley that was just off-target.
Rangers made another change five minutes later, when Ross Perry was forced out of the action following a clash of heads, and was replaced by fellow defender Sebastien Faure.
The home side were still chasing an elusive second goal and the honours almost went to Templeton, who produced a ferocious shot that flashed across the face of goal and past the upright.
Instead, a horrific error from Alexander gifted Elgin the equaliser with three minutes to go.
Moore's free-kick was met by Leslie who directed a header at goal, which initially appeared to be blocked by the goalkeeper before he fumbled the ball over the line.