Sir Harry Burns, now Professor of Global Health at Strathclyde University, said all areas of government and society need to be aware of the damage done to children in their formative years.
He told the Scottish Parliament health and sport committee the more a child's early years are difficult, the more likely they will suffer social problems as adults.
Mr Burns, who has studied early intervention techniques around the world and seen the health and care service close up in Scotland, said suppor-ting parents was crucial in determining a child's future.
He said: "What we need is an over-arching structure to change the lives of people who struggle. We need to support people who just don't know how to look after their children. Extreme poverty changes the way a baby's brain develops."
Mr Burns gave evidence to the committee with Professor Michael Marmot of University College London, a globally recognised expert in health inequalities.
He also said parenting was the key and that it was a political decision to allow children to live in poverty.
He said: "Some countries are intolerable of child poverty."
Talking about recent government decisions, he said: "A choice has been made that the worse off should suffer more. The lower you are the greater the drop in income.
"That will damage our children. I analyse the data, I don't make political statements."
He added: "I heard the Prime Minister say over the floods 'money was no object'.
"Why wouldn't he say that about child poverty? Floods are terrible, but so is child poverty."