In a battle at the bottom of the table, Alex Brinkhurst gave Oak a third minute lead.
Thomas Johnson levelled on 21 minutes but Durciel Nathan Mampono restored Oakwood’s advantage 11 minutes later.
Storrington threw the kitchen sink at the Oaks in the second half in search for an equaliser but they were stung on the counter.
Two goals in second half stoppage time from Mervin Bhujun and substitute Arnold Saurombe saw Oakwood claim a vital three points.
Defeat for the third-from-bottom Swans sees place-below Oakwood cut the gap between the two sides to four points.
Storrington remain eight points clear of bottom-of-the-table Billingshurst, although Hurst have two games in hand.
Bardouleau said: “We started poorly and Oakwood punished us. Credit to the boys as they responded well and dominated the rest of the half, equalising through Thomas Johnson.
“Just before half time Gary Elliott made a poor mistake, again Oakwood capitalised.
“Gary has won and saved us games in the past so that mistake is easily forgotten.
“We dominated possession and the share of the chances in the second half but could not find the break through.
“We hit the post a couple of times and had narrow misses.
“As we were in the closing stages of injury time we were camped in their half, committing everyone forward. Oakwood caught us on the break twice.
“We did not deserve a loss, certainly not 4-1. We should’ve won the game, but it was a lesson learnt in taking our chances and having quality on the final ball.
“Positive signs in Martin Harris got more minutes under his belt in recovering from injury.
“New signing Hayden Hunter (acquired from Premier Division side Pagham) looks a class signing.”
Storrington’s trip to 14th-placed Hailsham Town on Tuesday evening was postponed due to a waterlogged pitch.
A new date for the game will be arranged in due course.
Storrington host Worthing United, in tenth, on Saturday and Bardouleau said: “Worthing United went on a run of games and climbed the table, so it’s going to be a test.”