Up until a week ago today, he was living out at a well known criminal-activity house/ drug
dealer
So as he is no longer living there, that's not a problem... BTW, make an anonymous report to police regrading the house if you are sure there is criminal activity going on.
So a couple of points to be made first.... Centrelink doesn't grant custody, full or otherwise... In the eyes of the law, unless you have orders made that say different, both parents have equal & shared parental responsibility, & the CHILD has the right to a meaningful relationship with BOTH parents, whether they are together or separated...Your ex may be your ex but he is still your child's dad..
What to do... First of all it's not about 'power'.... Him keeping the child & making ultimatums is not okay.... Likewise you only allowing the child access at your say so is not okay.... Some of his actions may be in part a result of frustration with the later along with adjusting to the separation. So what you need is an agreement about days & times that dad & child can spend some amount of time together... That is best done as a properly set out parenting plan. There are organizations such as relationships Australia that can help you put together a PP.
Between the ages of 2 - 5 is when most children form close relationships. It's important that a child of that age be given enough time to form that kind of relationship... A week can be a very long time for a child of this age, so if you could agree to something along the lines of What Sammy has suggested, ie, at least 1 overnight a week & with some additional time midweek if possible...
Hopefully a set routine in writing may help remove some of the frustration around access times