If I was you, I'd consider these things holding you back as things to place on the altar of sacrifice, and ask the Lord to help you feel willing to give them up.
And remember this -- there's a scripture in D&C that describes how 'your family will live' if you serve a mission. Your going on a mission may well bring huge blessings to your brother that wouldn't otherwise materialize if you just stay home.
And think about how sweet it will be to come home and see him again after two years!!!!
I gave up a lot when I left - a girlfriend particularly, and my 8 year old brother. My father's business took off like you wouldn't imagine and then I ended up with a job in it the second I got home. Also, someone put $4000 away for me when I returned home, which was a lot of dough back in the late 1980's.
Go do it -- you'll have a ton of experiences that will strengthen your testimony for the rest of your life. I'm only active now because of those memories -- you'll fall on hard times at some point in your life going foward (spiritually) -- wouldn't it be great to have a reservoir of powerful spiritual experiences to draw on during those times?