First off... Hi and Welcome to the site.
Secondly, you have kinda answered your own question...
But to clarify a bit more and get the official reason for not wearing the cross as a symbol of our faith, below you'll find a link to a talk by the late Pres. Gordon B. Hinckley, which I think explains it really well.
If this still doesn't settle your issue on this particular subject I would encourage you to pray about it as in my opinion you won't get much of a better answer to your question than from the Lord and a late Prophet of the Lord.
LDS.org - Ensign Article - The Symbol of Our Faith
I hope this helps