Gunners boss Arsene Wenger had warned his side they already faced a must-win tie following their opening Group F loss at Dinamo Zagreb.
The Barclays Premier League side were behind against the run of play on 32 minutes when a low shot from Felipe Pardo deflected off Alex Oxlade-Chamberlain.
Although Theo Walcott netted a quick equaliser, Arsenal found themselves trailing again when Ospina, chosen ahead of Petr Cech, dropped a corner over the line.
Alexis Sanchez levelled on 65 minutes, but Arsenal hit the self-destruct button once more as Olympiacos immediately scored through substitute Alfred Finnbogason to give the Greeks a first win in England at the 13th attempt.
Arsenal are now bottom of the group, and face a daunting double header against leaders Bayern Munich.
It had been a positive start as Arsenal looked dangerous on the break through Oxlade-Chamberlain, who fired into the side netting, and Walcott, leading the attack as Olivier Giroud was suspended
The visitors, though, took a shock lead on 33 minutes
A corner from the right was cut back to the edge of the penalty area, where Pardo drilled in a low shot, which took a massive deflection off the heel of Oxlade-Chamberlain and flew past Ospina.
Arsenal were level within three minutes.
Sanchez got away down the left, and fed Walcott, who beat the offside trap before cutting a low shot goalwards, which squirmed under goalkeeper Jimenez Roberto into the far corner.
No sooner had Arsenal recovered, they hit the self-destruct button again.
A corner from the right was floated in by Kostas Fortounis, which Ospina jumped to collect, but inexplicably patted the ball down behind the line and the goal was immediately awarded by the Dutch fifth official Danny Makkelie despite replays proving inconclusive.
Arsenal aimed to regroup for the second half, but lacked a decisive pass in the final third.
There was an enforced change on 56 minutes when centre-back Laurent Koscielny had to be replaced by Per Mertesacker after picking up what looked like a hamstring injury following a sliding tackle.
Arsenal came close to an equaliser on 60 minutes when Sanchez was played in behind on the right, and his cross into the six-yard box was deflected on to the keeper
The rebound fell to Mertesacker, whose low shot was saved, with Santi Cazorla smashing the ball goalwards, only for Omar Elabdellaoui to head off the line.
Arsenal remained on the front foot, and equalised on 65 minutes.
Walcott floated the ball in from the right, and Sanchez guided a cushioned header into the far corner.
However, the Emirates Stadium was silenced within a minute as Olympiacos regained the lead.
Arsenal failed to clear from the edge of their own penalty area, and the ball was put back in by Pardo, with substitute Finnbogason clipping it past Ospina at the near post.
Slowly, the home side regained some momentum, with a 25-yard free-kick from Mesut Ozil saved at full stretch by Roberto.
Arsenal, though, could not find a way back into the tie, and now face an uphill battle to qualify for the knockout stages.
TWEET OF THE MATCH
"So bored of Wenger's ineptitude
How can you leave your No1 'keeper out of a game you have to win??? Utter madness
#AFCvOLY" - Piers Morgan
David Ospina: 3
Hector Bellerin: 5
Laurent Koscielny: 5
Kieran Gibbs: 5
Francis Coquelin: 5
Santi Cazorla: 6
Alex Oxlade-Chamberlain: 5
Alexis Sanchez: 7
Mesut Ozil: 6
Theo Walcott: 7
Per Mertesacker: 6
Aaron Ramsey: 5
Joel Campbell: 5
Esteban Cambiasso: Leicester fans were devastated when the experienced Argentinian opted against a second season at the King Power Stadium but the 35-year-old made a sound decision to sign for Olympiacos
He is back in the Champions League and anchored the midfield with all the class and determination he has shown throughout the years.
MOMENT OF THE MATCH
David Ospina: Plenty of eyebrows were raised as Wenger rested first-choice goalkeeper Cech, having already lost their opening Group F game
But the Frenchman's decision will come under even more scrutiny after Ospina's first-half blunder gifted Olympiacos the lead as he failed to catch Fortounis' simple corner and fumbled it over the line.
VIEW FROM THE BENCH
Wenger knew a win here was hugely important to qualification hopes with back-to-back games against Bayern Munich on the horizon
His decision to rest Cech will need answering but his defence as a whole let him down as they could not stay level long enough to exert pressure on the visitors.
MOAN OF THE MATCH
More English despair in the Champions League
Throughout the rest of Europe this will be seen as a triumphant night for Olympiacos, who had lost all of their other 12 visits to England over the years
Arsenal just did not turn up and could face dropping out of the competition at the group stages with that tricky double-header with Bayern to come.
WHO'S UP NEXT
Arsenal v Manchester United (Premier League, Sunday October 4)
PAOK Salonika v Olympiacos (Superleague, Sunday October 4)
Source : PA