I think that there are two issues here on the table. One, worthiness to hold and use the priesthood, and Two, dislike of a persons attitudes.
Only a priesthood leader could determine someones worthiness. Priesthood holders are not perfect. And they aren't required to be. I think priesthood power and sensitivity to the spirit IS compromised with any less than righteous behavior or thinking. But, I guess I see that as an effectiveness issue rather than a worthiness one.
I think that we are all in process. Rough stones rolling, as it were. We all have that dang natural man to deal with. Sometimes our spouses can see our flaws and failings. And sometimes may be in a position to help us with them. I know with my own weakness, I do better when someone comes to me with kind suggestions or perhaps deeper searching questions about why I may feel or see the world the way I do. It is also helpful when they talk about the good they see in me as they nudge me to higher ground.
In this case, maybe you speaking your mind IS helping him even if he outwardly continues the argument. Perhaps over time, he will slowly make the changes you wish to see.
I know in my situation, it has been helpful for me to pray and ask Heavenly Father to lend me His mind on my H's weaknesses with regards to how I should be viewing them, and then with what role I might take in effecting change, if any. Sometimes, I learn that it is me that needs to learn something too. Which chaps a little on occasion.
If your H is imperfect but worthy, that is one thing. If your H is legitamately unworthy, that is something else. Having said that, I know what it is like to look at two priesthood holders and trust ones righteousness over another based upon the feel of their spirits and I have felt the difference in blessings given.