As you will find with the language used in diagrams, etc. "Sleeve" or "Shield" = "Ground." Not speaking to the correctness of the usage, simply the colloquial usage.
Yes, in general, the "male" end of the cable is the end you would not use Pin 1. I think a better way of looking at it is in relation to signal flow, like Speedskater explains.
However, since you stipulated a balanced connection in your title, you would use Pin 1 on both ends. An unbalanced connection binds pins 1 & 2.