A 76th-minute goal from James Maddison looked to have secured the three points for Brendan Rodgers’ Leicester until a dramatic finale with the Dutch substitute turning the game around.
Bergwijn bagged an equaliser in the fifth minute of stoppage time after Hojberg floated a ball to Matt Doherty in the box and the Dutchman pounced to fire home the loose ball.
He then astonishingly grabbed the winner after Youri Tielemans gave the ball away and Harry Kane played a superb through ball to the Dutchman who rounded Kasper Schmeichel and slotted into the unguarded goal.
Kane went close twice in the early stages, including a header against the bar in the 19th minute, as Antonio Conte’s side dominated.
The visitors got their eventual reward when they drew level in the 38th minute with a clinical finish from Kane, who beat the offside trap, skipped inside Caglar Soyuncu and drilled a low shot in off the far post.
Kane then wasted a glorious chance, firing high over the bar after being put through on goal by Lucas Moura.
It was a tighter affair after the break but Leicester got in front in the 76th minute when substitute Harvey Barnes found Maddison whose low shot struck the outstretched leg of Japhet Tanganga and flew in via the post.
That looked to be enough until substitute Bergwijn levelled before sparking wild celebrations with the winner in the seventh minute of stoppage time.
“What a rollercoaster of emotions that was,” said Kane.
“Overall we deserved the victory. We were the better side. Disappointed to go 2-1 down but we were creating the chances. We never gave up and Stevey came on and made the difference. To get two goals like that — these are games you will never forget in a career.”