How do I avoid being friends with this mom?

You need to be strong and tell the woman you don't want your child going over there or you going over there. Tell her she should be watching her own children, not you. Just tell her to leave you alone, but nicer.