Am I wrong for not giving my husband sex?

Absolutely not. If there is no interaction between the two of you, you've tried to fix your relationship on the emotional side of things but not getting anywhere there's absolutely no need for you to continute a physical part of your relationship either. He is absolutely using you. He wants all the benefits of being married but none of the responsibility. I'd suggest counseling or if that doesn't work then a separation would be in order.