Yet neither you nor the kids are subject to an AVO sought on your behalf by the police...
See, a lawyer, a police officer, a psychologist and a DHS official will all advise a parent to withhold their kids' time with the other parent if they say the magic words, 'I am concerned for the kids' safety when in the other parent's care'. It's a basic, better-to-be-safe-than-sorry approach because in the event the danger is genuine, they don't want to be held responsible for making the wrong call.
However, the Family Court is far less cautious. It demands evidence, and I don't mean the he-said-she-said evidence that usually appears in self-written affidavits. I mean evidence from police officers of an actual risk, a report from DHS of an actioned investigation. Even the psychologist's evidence is probably going to be wishy-washy. If you took the kids to see him without consulting the father, then the father's frustration can quite easily be taken in the context of being actively excluded from major long-term decisions affecting his kids, and in perspective, if he was an uncaring father, he wouldn't be attending such appointments to begin with, would he?
Reading between the lines, as you've said, you're insinuating that during your relationship, he was violent toward you, yet you have not once said he was ever violent toward the kids. To the contrary, for a period there, you were insisting he see the kids more often. This sounds like an acrimonious co-parenting relationship, and not even really an insurmountable one at that, but it doesn't sound like he poses a genuine risk to the kids.
Forgive my cynicism, but I think every parent should know what they face if they force their parenting matter to Court. It's unlikely that the Court will make an order for no contact between parent and child. It's unlikely it will allow just phone calls for such young kids to try and maintain a meaningful relationship with the father. It's unlikely that your unilateral decision to withhold the kids will do your case any favours.
Again, I think you should attend a post-separation parenting course, and mediation with the father.