Hi Mnn
I think most of us who have lived with the addiction know the feeling that home does not feel like home anymore and it seems that that only person who doesn’t ‘feel’ the irritation is the CG.  
I like Berber’s suggestion that maybe you could find somewhere he could go, where his addiction could not do any damage and give you a well-deserved break.   You mentioned shelters before – do they offer respite facilities?  
I saw he was alone one night a week but is he eating with you most evenings?   Are you going out because he is there?   
It sounds as though the addiction is pulling the strings somewhat and your home life is suffering.   CGs do not want to listen to things they do not want to hear but it is really important, in my opinion, for you to prioritise, in your mind again, that he is a CG and as a CG he will have behaviour traits that are difficult/impossible to live with and that he ***** the right support to help take control of his life.   At the moment he is allowing you to do his worrying, smell his tobacco and listen to his idle chat.
Perhaps you could suggest evenings where he doesn’t come in with you and your wife at all.   Your brother is with you because he has messed up his life and if it was me I would be inclined towards suggesting the invitation is open only as long as he seeks help and gets a part time job that takes him out of your home.   His addiction is selfish.    If life is too easy for your brother he is not going to lift a finger to help himself.
Did you have any agreement about what would happen if one or the other parties in your arrangement was not happy or is the invitation open to him ad infinitum?  
I have likened active CGs to the Prodigal Son many ***** myself, however your brother’s addiction will suck the bones of the fatted cow dry if you don’t put your foot down.  
You said before that you did not want to be your brother’s jailor but if you are not careful you will feel imprisoned by an addiction that you do no own.   I think it would help if your brother was aware that you ‘know’ he is a CG even if he will not accept it and that seeking help is important to the success of his ‘visit’ with you.  In a GA group he would find others that he could talk to – CGs who want to change their lives forge terrific bonds.  Perhaps you could find the addresses where groups meet and give them to you brother.  
I hope this makes sense.   I believe that feelings of irritation should not be repressed but I know that hollering at the addiction is a waste of energy.   It is important to let off steam or you could implode and that will not do you or your wife any good.  I found that writing down all my confused thoughts was therapeutic but I’m afraid that for me the top of the pressure cooker just blew off in the end.    I want to support you so that this does not happen to you.
Keep posting and think of ways to improve ‘your’ life first before making your brother too comfortable.