It had looked like the Gunners were set for a second away defeat in Champions League Group D after the Belgians went ahead through Andy Najar on 71 minutes.
However, just as time was running out, England full-back Kieran Gibbs cracked in a superb volleyed equaliser, before German forward Podolski swept home a dramatic winner.
Arsenal had started brightly, but Santi Cazorla blasted an early chance horribly wide after good work from Alexis Sanchez.
Anderlecht had a point to prove having been undone 3-0 at home by the Group D leaders Borussia Dortmund last time out and were dangerous going forwards, led by Cyriac, the former Charlton forward, and the industry of Dennis Praet, who once had a trail at Arsenal as a teenager.
The Gunners had plenty of possession in the Anderlecht half, but rarely looked like breaking down the home defence during a tepid first half.
Sanchez saw a close-range flick saved after a floated free-kick into the Anderlecht penalty area as Arsenal showed some signs of life, with Aaron Ramsey then shooting just wide.
Arsenal's clearest opening of the night came on 63 minutes when Cazorla was played clear down the left, but his angled drive was saved by Silvio Proto.
It proved a costly miss as Anderlecht took the lead on 71 minutes when a floated cross from Praet was headed in by Najar.
Former Portsmouth defender Anthony Vanden Borre had a miskicked effort loop onto the top of the crossbar.
Martinez saved from Steven Defour, but Arsenal rallied after Gibbs' smashed home a deep cross by right-back Calum Chambers before Podolski won it in the most dramatic of finishes.
Source : PA