The Coatbridge club's boss saw his side lose Michael O'Byrne to a red card in the first half and then slip behind to a Paul Millar winner.
Martin, who praised his players' backs-to-the-wall efforts, said: "Elgin have got the advantage now with the one goal lead.
"But I thought my players were absolutely terrific under very difficult circumstances. It keeps the tie alive when the length of time we were down to 10 men we could easily have been out of it. It's Elgin's to lose now, but we've got to come out and have a go."
City piled on early pressure but missed a clutch of chances before Albion's Michael O'Byrne saw his evening end prematurely.
O'Byrne got caught on the wrong side of Millar and pulled the Elgin front man back on a run towards goal, resulting in a straight red card for the Albion defender on 28 minutes.
Rovers' resistance was finally broken on 53 minutes after a Daniel Moore effort spun into Millar's path and he netted from 10 yards.
Elgin boss Ross Jack blasted Rovers' negative tactics and said: "They came in, parked the bus and made it extremely difficult for us.
"We cut them open umpteen times but too many chances went abegging in the first half and in the second half we just didn't have the cutting edge."