After suffering a shock 2-1 defeat at home to Lyon a fortnight ago, the Premier League champions badly needed the three points to move off the bottom of Group F.
Silva poached the victor on 87 minutes when he reacted quickest to nick the ball off Hoffenheim defender Stefan Posch on the penalty spot and fire home.
Pep Guardiola's men began the campaign with a 2-1 loss to Ligue 1 outfit Olympique at the Etihad on match day one.
Hoffenheim continued their attacking momentum in the moments after Belfodil's goal, Joelinton firing in a cross which Ederson gathered. Silva passes to Sane, who squares it for Aguero to knock in as a matter of utmost simplicity. They are certainly a team in form and are playing at home - it will certainly be a hard game for us.
Hoffenheim broke menacingly in the opening seconds of the second half, a low cross from Belfodil flying across the area before Florian Grillitsch lashed high from distance.
City got themselves level soon after, with Sergio Aguero scoring after a cut-back from Leroy Sané.
And the turnaround was completed late on courtesy of David Silva's opportunistic strike just three minutes from time. Quick feet from Baumann to get back on his line and gather Aguero's header from Bernardo Silva's cross.
Sane wanted a 73rd-minute penalty when Baumann came out to meet him as he tried to round him - but the referee gave a goal-kick despite the Hoffenheim keeper appearing to bring him down.