Singer Carolynne Poole became the first act to go after the elimination show saw judge Louis Walsh unable to make up his mind and go to a deadlock.
Poole and flamboyant Rylan Clark were in the bottom two and were forced into a sing off.
Judge Nicole Scherzinger voted for Clark, who she is mentoring, while Barlow and Tulisa Contostavlos voted for Poole.
Walsh was unable to make up his mind, meaning the singer with the fewest public votes had to walk.
When host Dermot O'Leary announced that Poole would be leaving the competition Barlow stormed off stage.
Barlow said he was "embarrassed", and criticised the judges for keeping Clark, who sang Spandau Ballet's Gold, in the competition.
But Walsh stuck up for the singer, saying: "That was amazing. I love your vocals... You're a brilliant performer."
Contostavlos added: "This competition would be the same without you."