Jurgen Klopp made two changes to the side which edged out RB Salzburg 4-3 in the Champions League, with Dejan Lovren and Milner replacing Jordan Henderson and Joe Gomez.
The Foxes were able to recall fit-again midfielder James Maddison, with Ayoze Perez dropping to the bench.
A bright opening to the game saw Mo Salah test visiting keeper Kasper Schmeichel, with the Reds eager to try and dominate from the off.
Trent Alexander-Arnold then set up Milner with a ball in from the right, but the veteran was unable to keep his volley down from around ten yards out.
The Foxes were finding it tough to make much progress in the final third, with Sadio Mane warming the gloves of Schmeichel at the other end, before Roberto Firmino somehow failed to hit the target from a superb delivery from Alexander-Arnold.
Mane made no mistake soon after as he latched on to an error from Jonny Evans to drive into the area and dispatch a cool finish past Schmeichel to give the hosts a 1-0 half-time lead.
City brought on Marc Albrighton at the start of the second half, while Schmeichel was soon called into action to deny Salah once again, with Mane shooting narrowly wide as the hosts looked for a second goal.
Firmino also saw an effort drift wide, before Jamie Vardy ran through to be denied by the alert Adrian in the home goal, although the offside flag had been raised.
Dennis Praet fired just wide of an upright from just outside the area as City went in search of an equaliser and they got their reward ten minutes from time as Adrian failed to keep out a Maddison shot which he could have dealt with better.
The home side were not to be denied, however, as they won a penalty in stoppage time as Mane fell under the challenge of Albrighton and Milner rammed home the resulting spot-kick to send the home fans wild in celebration.