Well...when Rolex launched Air-King model in 50's, it was A BIG mens watch (=34mm), but nowdays fashion trend keeps that as small ladies watch. But I have a feeling that in your case.
A) 99.9% of people do not even notice
B) 99.9% of people do not care
And if you can get some vintage Rolex and Tudor, its quite classy still, and you can find those pre-owned quite decent prices. (Tudor specially) Wear it pride and joy.
By the way: I sometimes borrow wife's watch. Get nice kicks knowing it hers and smell her parfume on it.
But she wears men's pieces so its not a same I quess..