In what was dramatic match in the Mother City, the home side took the lead in the 34th minute with a superb effort by Kermit Erasmus but Samir Nurkovic equalised six minutes from full-time to send the game into the extra time and then penalties.
Chiefs started the game on the front foot but did not really test City goalkeeper Peter Leeuwenburgh, with a glancing header that went over the bar by Daniel Cardoso the best effort in the first ten minutes.
The game was played at a frenetic pace and the hosts came got more into encounter as Mpho Makola and Mark Mayambela had efforts on goal but could not break the deadlock.
Amakhosi had a glorious chance to open the scoring when they were awarded a penalty just after the half-hour mark after a shot by Khama Billiat struck the arm of City defender Taariq Fielies.
The home sides players protested furiously at the referee's decision with Makola lucky to escape a red card after giving the official in the middle a shove in the chest.
Cardoso though failed to give the visitors the advantage on the scoreboard as he blazed his spot-kick high over the cross-bar.
Chiefs would go onto rue that miss as just two minutes later they fell behind as Erasmus gave the Citizens the lead with a rasping long-range effort that hit the top corner.
The Soweto giants' pain was further compounded as they were forced into two injury-enforced substitutions late in the first half as Kearyn Baccus and Bernard Parker were stretchered off.
The second half was just as riveting as the first as both teams refused to give an inch and the tackles kept flying in as the ref dished out 12 yellow cards on the day.
George Maluleka should have leveled the game five minutes after the interval but he shot over from close-range after the City defence failed to deal with a cross into the box.
Chiefs shot-stopper Daniel Akpeyi kept his side in the hunt as he made a brilliant double save to deny Erasmus while Makola hit the bar with a long-range shot as City looked for the second goal.
The visitors threw everything into their attack but it also left them vulnerable on the counter which made for a fascinating match but City failed to grab their opportunities.
The hosts would eventually pay for not killing the match off as substitute Nurkovic equalised in the 84th minute, steering the ball past Leeuwenburgh after Reeve Frosler delivered an inviting cross into the penalty area.
The pace of the game understandably dropped in extra time as both teams' players started to show signs of fatigue with the 30 minutes failing to produce any clear-cut chances for any team to win it.
The match went into the lottery of penalties and Chiefs took the early advantage as Akepyi saved from Ebrahim Seedat while Amakhosi converted their first three spot-kicks.
Even though Lebogang Manyama missed, Fielies sent his penalty wide to allow Mathoho to score and send Chiefs into the Last 8 of the Telkom Knockout.