Yes... You're much better of at least having a parenting plan that provides some access for both parents. Relationships Australia can help with that as well as helping sort out concerns about child safety.
If I can use my case as an example. I used to look after the kids at least 2 nights a week while my then wife went out. No worries, I worked she didn't - I enjoyed the one on one daddy time, etc...
Things go bad and I'm forced out of the house... All of a sudden she is claiming I can't handle the kids on my own and she does everything she can for me not to see them. She wants court orders before she'll let me see the kids. Now that could take 6 months or more.
So I don't get to see the kids for 6 months all because she is scared that if I were to pick them up from pre-school I might not return them? Isn't that exactly what she is imposing on me?
Now I didn't get to see my kids for about 3 months. Technically there was nothing stopping me from picking them up from pre-school - except that my solicitor advised me not to...
So what is wrong with the other parent having one night a week? I'd suggest unless Doc's has been involved then maybe your concerns are a bit of hyper-vigilance. Let the other parent have the kids - if they come back battered and bruised then go to the cops, that way you'll have good grounds to refuse access.
If they come back smiling and happy then maybe, just maybe the other parent ain't as bad as you think. You gotta remember sometimes our judgements are clouded by the raw emotions of relationship break-ups, etc.