style - Is it frustrating not to know the narrator's gender?

I have written a 1st person piece and reading it I realize it's very difficult to tell if the narrator is male or female. There are one or two clues, and they come pretty late in the piece.

Would you as readers busy yourself with trying to figure out gender? Or do you think you would automatically assume one gender? and then would it be frustrating to find out you were wrong?

Would you as a writer try to clarify this early on?

--

edit: There are good answers. Obviously part of every answer is that it depends on what I want to achieve.

To clarify, it isn't important information for the understanding of the piece, as far as I am concerned. I don't really mind if it was understood one way or another; I just want it out of the reader's way.