Scott Bambrick, 25, from Stonehouse in Lanarkshire also hit the tot and her two-year-old brother.
Jailing him at the High Court in Edinburgh, judge Lord Uist told Bambrick: "You must appreciate that anyone who causes harm of this nature to children has to receive a long sentence of imprisonment."
Bambrick will also be subject to sex offenders' register rules for life.
An earlier trial heard how the children's mum went out for the evening, leaving Bambrick to look after them.
She returned to her Stirling home in the early hours of the morning and was woken next day by the little boy.
The mum - who like the children cannot be identified for legal reasons - was suspicious because the boy's nappy was fastened with masking tape.
She found that both he and his sister had badly-marked bottoms.
Doctors examined both youngsters and concluded that the injuries were not accidental. Some of the bruises appeared to be finger marks, the High Court in Edinburgh was told.
The little girl also had other, more suspicious bruising.
Bambrick claimed he did not know how the children were injured but admitted to police that he had given the little girl "a wee skelp" which had made her cry. He also said he had smacked the little boy's hand.
In court, Bambrick lodged papers blaming their mum for anything that had happened.
But a jury rejected his denials and also his suggestion that the little girl could have fallen against an up-turned table leg in the untidy house.
He was found guilty, unanimously, of striking both children, with his hand, on the night of May 19-20, last year.
Defence advocate Sarah Livingstone said Bambick continued to deny any wrong doing.
The lawyer also pointed out that he had no record for violence or sex offences.