Kris Commons scored from the spot in the 18th minute after home midfielder Jamie Hamill was controversially penalised for handball before Jason Holt equalised in the 56th with a deflected drive from the edge of the box.
Anthony Stokes put Celtic back in front after 65 minutes with an unstoppable volley from 12 yards before substitute Teemu Pukki, making his debut, made it 3-1 five minutes from time to mark a perfect start to his Hoops career following his move from Schalke.
"We had so many good chances today," Lennon said on BT Sport.
"We made life a little bit hard for ourselves but in the end I thought it was a really good performance.
"In the second half we showed really good power going forward.
"With a little bit more care on the final pass we could have won the game more comfortably."
Stokes was particularly impressive for the visitors, scoring one and then providing a brilliant assist for Pukki's goal.
Stokes said: "We dominated the game. They had a high tempo, which we expected, but we saw it out in the end.
"It's a great atmosphere here. I love playing here, and I'm just glad we got the result.
"The most most important thing is we're winning games. It was a tough game today."
Hearts boss Gary Locke added: "Today's result is not going to define our season, games against Celtic. It's the teams around us, they're the games where we've got to pick up points.
"I was pleased with a lot of things today, disappointed with one or two other things, but we're learning all the time."