Premier League ref steps down from County FA role
The Football Association already banned John Cummings for four months for telling Lucy May - who is Oliver's fiancee - "a woman's place is in the kitchen and not on a football field".
The NFA board decided the FA's ruling, including a £250 fine, was sufficient and decided against further punishment.
Four other NFA members also stood down in protest at the decision, which according to the BBC was made by secret ballot.
One source, who wished to remain anonymous, told the BBC "the NFA board is like a body of dinosaurs. Many of them hold views that are not those of the people they represent."
Cummings, 77, made the comments to referee development officer May at a referee workshop event in March.
May, 24, asked Cummings about the possibility of officiating in the North East Sunday League.
Cummings told her she "wouldn't be able to handle it".
He added: "It's nothing against you personally but all the time I'm alive, a woman will never referee in my league."
Cummings, who had his appeal against the FA's four-month ban dismissed, has been told he must attend an education programme before he returns to his position.
As well as Oliver, it is understood NFA finance director Steve Drew, youth representative Ian Coates, referee coordinator Dave Martin and referee coach Scott Duncan have stepped down.