The 26-year-old man, from Berkshire, died this morning as the first official day of music got under way at Worthy Farm in Somerset.
He was taken to Bristol Royal Infirmary in a "life-threatening condition" on Wednesday.
Avon and Somerset Police tweeted: "Unfortunately the man taken ill after a suspected reaction to ketamine has died this morning.
"His family are aware and being supported."
Police said it is not thought that the batch of the Class B drug the man took was contaminated, but that he suffered an adverse reaction to it.
A 67-year-old woman died at the site on Wednesday night from natural causes. Her death is not being treated as suspicious.
Kicking off the official music today, a 17-piece Japanese orchestra made an enthusiastic effort to banish the bad weather.
Turtle Island got proceedings off to a rousing start on the Pyramid Stage, drawing a small crowd to their pulsating punk-rock beats.
As steady rain fell on the site, many appeared reluctant to leave the relative comfort of their tents, while others tentatively ventured out in wellies and anoraks to see just how much mud they might have to contend with.
Although the rain - expected to continue throughout the day - was not welcome to many, stall holders charging £15 a time for wellies and waterproofs were enjoying a roaring trade.
Also performing on the Pyramid Stage later today will be Elbow, Lily Allen and Rudimental, while acts on other stages include Haim, MIA and Jurassic 5.
Police said 85 crimes were reported within the first 48 hours of the festival opening its gates on Wednesday morning, a 30% decrease on last year.
About half of these were property thefts, with 30 people arrested so far.
Police said they used tracking software to locate one stolen phone and arrested a woman overnight.
Inspector Liz Hughes, of Avon and Somerset Police, said: "Thanks to a tracking app, we've been able to recover a stolen phone, which shows that they really do work.
"They are really useful and I would urge anyone who hasn't already got one installed to do so today.
"We've also arrested three men on suspicion of theft and this shows that there are a very small number of people who come here intending to steal and spoil the fun for everyone else.
"While the arrests are a positive step, it's a reminder to us all to look after your possessions very carefully."
Police said that the man who died was from Reading and was taken ill overnight on Wednesday into Thursday.
A spokesman said: "The family of the man had been informed of his condition yesterday. His death is not considered suspicious.
"At this stage we believe this was an individual reaction to ketamine and was not from a bad batch of the drug."