The 1912 Olympics in Stockholm had 2407 athletes, including 48 women. Team USA did not send any female members - surprisingly, for similar reasons as Saudi Arabia until this year, due largely to modesty. -
(link) "because U.S. team rules required women to wear long skirts in all their events, no American women competed."