Muriwai Beach near Auckland was closed after the fatal attack, one of only about a dozen in New Zealand in the past 180 years.
Pio Mose, who was fishing at the beach, said he saw the swimmer struggle against the "huge" shark. He told the man to swim to the rocks, but it was too late.
"All of a sudden there was blood everywhere," Mr Mose said. "... I was shaking, scared, panicked."
Police inspector Shawn Rutene said the swimmer, in his 40s, was about 200 meters (650ft) offshore when the shark attacked. He said police went out in inflatable surf-lifesaving boats and shot at the shark, which they estimated was 12 to 14ft long.
"It rolled over and disappeared," Mr Rutene said.
About 200 people had been enjoying the beach during the Southern Hemisphere summer at the time of the attack. Police said Muriwai and other beaches nearby have been closed until further notice.
Police did not say what species of shark was involved in the attack. Clinton Duffy, a shark expert with the Department of Conservation, said New Zealand was a hotspot for great white sharks, and other potentially lethal species also inhabit the waters.
Attacks are rare. Mr Duffy estimated that only 12 to 14 people have been killed by sharks in New Zealand since record-keeping began in the 1830s.
"There are much lower levels of shark attacks here than in Australia," he said. "It's possibly a function of how many people are in the water" in New Zealand's cooler climate.
He said that during the Southern Hemisphere summer, sharks often come in closer to shore to feed and to give birth, although that doesn't necessarily equate to a greater risk of attack.
"Ninety-nine per cent of the time they ignore people," he said. "Sometimes, people get bitten."
Around the world, sharks attacked humans 80 times last year, and seven people were killed, according to the University of Florida's International Shark Attack File.
The death toll was lower than it was in 2011 but higher than the average of 4.4 from 2001 to 2010.