Matthew 12:30 He that is not with me is against me; and he that gathereth not with me scattered abroad.
Jesus was very clear, there is no neutrality. Whether you like it or not, you must chose sides. If you are not on the side of Jesus then you are against him.
Also unless you are gathering with him, you are actually scattering abroad. Yes, if you are not helping to gather souls into his kingdom, then you are actually driving people away from God.
If you are on the side of Jesus then Please come out here into the streets and the byways and help us to gather the lost souls for Jesus.