The Clarets made one alteration to the side beaten 4-2 at Liverpool last time out, with Johann Berg Gudmundsson taking the place of Jeff Hendrick.
Meanwhile, Foxes boss Brendan Rodgers named an unchanged XI from the 3-1 victory at home to Huddersfield Town last weekend.
Just four minutes in, City were reduced to 10 men after England defender Harry Maguire fouled Gudmundsson as he drove through on goal and was shown a straight red card.
From the resulting free-kick, Kasper Schmeichel made a fine fingertip save to keep out a swerving effort from Gudmundsson.
Rodgers reacted by sending on Morgan for Demarai Gray and the visitors stood firm at the back to remain on level terms as the midway point of the first half ticked by.
And they even broke the deadlock just past the half-hour mark as James Maddison curled home a superb right-footed free-kick, with Tom Heaton getting a hand to it but being unable to keep it out.
Heaton was forced into action soon after as Jamie Vardy broke clear before the hosts broke forward to find an equaliser, with Charlie Taylor pulling the ball back for Dwight McNeil to arrow a left-foot effort into the far corner of the net.
Early in the second half, Jack Cork curled an effort narrowly over after being set up by Ashley Barne, although the hosts were still struggling to make the most of the man advantage.
Chris Wood then failed to make the most of a major chance when he failed to convert from close range from a McNeil pass, while Taylor saw penalty claims waved away with a little under 20 minutes to play.
Morgan appeared to get away with a handball in the area soon after before McNeil whacked a shot narrowly over the crossbar from 12 yards out, with the same player seeing a long-range effort fly inches over as time ticked away.
Morgan then had the final say in the game when he headed home from a deflected cross from Youri Tielemans as the visitors earned a superb 2-1 win - despite playing for 86 minutes with 10 men.