You're referring to leaving 4 brick walls. The Church is not confined to your particular 4 brick walls. If they're going to not attend your specific fellowship, the important thing is that they are fulfilling their function in the body within the 2 Greatest Commandments. Where 2 or 3 are gathered, not "your church".