Without any court orders in place, both you and the father have shared parental responsibility for the children, which means you are expected to try and reach agreement about all major long-term decisions affecting the children. This includes relocation that significantly impacts the children's capacity to spend time with the other parent.
If you relocate without the father's consent, he would be able to pursue a recovery order to have the children returned to the town of origin, and this would likely be accompanied by an initiating application for parenting orders.
If you wish to minimise the risk of proceedings, however, you should attend a family dispute resolution conference with the father to try and establish a care regime that works for both of you.