While cities in England will enjoy reduced travel times to and from London, passengers from Glasgow and Edinburgh will see limited improvements, he said.
A high-speed rail line from London to Birmingham (HS2) is due to open in 2026, cutting the trip to 49 minutes, and to Manchester by 2032, reducing the trip to 68 minutes.
However, Mr Brown said Scots passengers using existing lines to link up in Manchester or Leeds would see the time cut by only 20% to three hours 38 minutes. Some services could be even slower.
He said: "In 2026 new services on existing tracks will run slower than at present and reductions will be modest.
"There will be little scope to meet rising passenger demand."
Mr Brown said it could result in economic disadvantage for Scotland. He was speaking at the Scottish Parliament Transport Infrastructure and Capital Expenditure Committee, where he outlined the Scottish Government's plan for a high speed line between Glasgow and Edinburgh.
He said: "We are planning high-speed between Edinburgh and Glasgow to link to the south. A new line would not only benefit the two cities but relieve pressure on services.
"I don't doubt the UK Government's commitment to high-speed rail, but we need it to come to Scotland."
Mr Brown said he was aware of scepticism after previous governments pledged to include Scotland in high-speed links to the Channel Tunnel, which did not materialise.