Spurs were in control of their Group J game against the Greek side and took the lead after 35 minutes when Michael Dawson glanced Tom Huddlestone's cross past Orestis Karnezis.
Jermain Defoe, Gareth Bale, Clint Dempsey and Huddlestone then all missed chances to put the game to bed and Panathinaikos took full advantage through striker Toche, who left Dawson for dead and turned in a Giorgos Seitaridis cross 13 minutes from time.
"I don't think we were complacent," Villas-Boas said. "We have to give credit to Panathinaikos, they pressed much more and started to gain belief from that and that led to the goal.
"I think the whole side felt a bit disappointed because the first half was so positive and we were in control.
"We were a bit sloppy in the second half. I understand the disappointment at not putting the game to bed, but in the end we congratulated the players because if it had been other way round, the whole perspective would have been different.
With Lazio beating Maribor, Spurs will find it hard to qualify if they fail to beat the Slovenians.
Meanwhile, strikes from Shola Ameobi, a Carlos Henrique own goal and Papisse Cisse were enough to hand Newcastle a 3-0 victory at home against Bordeux.
Following their 0-0 draw with Club Bruges, Alan Pardew's Magpies sit proudly at the top of Group D, with Maritimo up next.