The 48-year-old, on trial at Manchester Crown Court under his real name Michael Turner, denies 12 offences against a young girl, including five of rape.
The actor told the jury conversations he had about "something unforgivable" were concerned with extra-marital affairs.
He said he "totally" disagreed it was something "far more sinister".
The girl, who cannot be named, has alleged she was abused from the age of six until she was 14.
Mr Le Vell, from Hale in Greater Manchester, faces five counts of rape, three of indecent assault, two of causing a child to engage in sexual activity and two of sexual activity with a child.
Prosecutor Eleanor Laws QC told the jury about a conversation the actor, who plays Kevin Webster in the soap, had with an unnamed person concerning "secrets".
Mr Le Vell said he told them he had "had a few one night stands".
The case was adjourned until 10am on Monday when closing speeches will begin.
Meanwhile, DJ Dave Lee Travis has appeared in court to face 11 counts of indecent assault and one of sexual assault.
The 68-year-old spoke only to confirm his name and that he understood his bail conditions at the 20-minute preliminary hearing at the Old Bailey. He has indicated that he will plead not guilty to the charges, which are said to have taken place between 1977 and 2007 against nine alleged victims, aged 15 to 29.
Travis has denied any wrongdoing since he was first arrested on suspicion of sexual offences in November last year.
His lawyer, Stephen Vullo, told the court that he would prefer a trial to begin as soon as possible. After the hearing, Travis told reporters in the court building that he would not talk about the case. He said: "I've got nothing to say."
He was released on bail on condition he lives at his home in Mentmore, Bedfordshire, and does not contact the alleged victims.
Travis was charged last month as part of Operation Yewtree, the police investig-ation prompted by the Jimmy Savile abuse scandal, but the accusations against Travis have no connection to the disgraced television presenter.