VIC Visitation Arrangement - please help

I have currently stopped my daughter from staying overnight at her dads house due to concerns for her welfare (1 year old). Her dad has minimal involvement in her life (just moved 1 hour away) and due to the condition of my daughter upon pick ups (hungry, tired (so overtired she cries all the way home until she throws up on herself and falls asleep) and dirty, I made the decision to stop overnight access. My ex partner has a 4 year old from a previous relationship and has admitted to me that he struggles looking after both children and as a result my daughter does not sleep, eat or be dressed properly (no socks outside in the middle of winter) in his care. I emailed my ex partner and explained my concerns and that I wanted to sit down with our parents to discuss visitation, there are no current court orders in place but at this stage I am not comfortable arranging anything without third party presence and advice due to my concerns. It has now been 2 weeks and my ex partner is not agreeing to meet and make any arrangements, he just keeps sending me emails saying "24 hours notice and i will pick her up, when can she see her dad?". I am becoming deeply concerned that if this proceeds to court that I will be painted as withholding our child from him even though I am trying to arrange a meeting to discuss his visitation. Please note: He is very difficult to communicate with, I am not comfortable meeting him alone or discussing visitation by phone or email as we need an arrangement written out and witnessed for him to stick to it. There are no parenting orders in place or mediation... he is currently going through mediation with the mother of his first child to change court orders.