I think it is hard for people to think outside the box on this issue because our leaders have taught that every able and worthy young man should serve. I think a situation like yours is rare and won't be something that people think about readily. I think they look at you and see your strengths and wonder "why not".
In the end, I don't think it is ignorance. I mean how could people possibly know your circumstances. And we all make general assumptions. It is human nature. I think it all comes from a place of loving the idea of an RM and they want that for you too. That ideal image may be a stumbling block for us as LDS, in this area and others too. And perhaps we need to think about things out of the box sometimes.
I used to get comments like this all the time about not being married before I was 25. I don't know how many people patted me on the knee and unsolicited said, "he will come, dear" or "what is wrong with you" or " I have found your perfect husband!" It was annoying and some people were really unsensitive, but in the end, I think it was all because they wanted something good for me.
So, perhaps the trick is to stand strong in your convictions and choices and not to become defensive or self conscious in any way. You can greet such comments or attitudes with patience and try to see any good intention that might be behind it. You don't have to justify yourself or explain. At the end of the day, it is not their business and you have no obligation to enlighten them. Perhaps you could say, "what an interesting perspective. Have you ever thought about it differently?" Or maybe just find some private humor and forgive the person their fumbling. No sense wrecking your own peace over such things.
We are always safe when following the guidance of the Holy Spirit -- even if it doesn't follow convention. Correction: ESPECIALLY when it doesn't follow convention.
J Smith knew all about that one!