You don't have to agree to anything at mediation, but let's say it fails and dad does decide to take it to court. We'll assume there's no other circumstances of concern in this case, either, just to keep it all relatively straight forward.
At court, it's pretty standard for each parent to get half of all school holidays with the child, which amounts to about six weeks each in total for the year (one week for each of the short holiday periods, three weeks in the long summer holiday period). Dad only wants four weeks, right? Maybe that's a win worth compromising for at mediation, rather than possibly ending up with six weeks at Court.
It's also courteous for a parent to communicate with the other about where they intend to travel with the kids, but the court doesn't require the other parent to approve travel provided it's within Australian borders. Dad's parenting time with the child is when he makes those kinds of decisions, just as your parenting time with the child is when you get to make the decisions. Provided his travel falls within his time with the child and he's back in time to present the child at changeover, the Court isn't going to be phased by where he's actually travelling to with the child, or how far away that might be from you (again, provided it's within Australian borders).
Yep, it goes against the grain to be away from our kids for longer than a day or two, but the court doesn't see it as the child being away from his parents. He's just going to be with his dad sometimes, and with his mum sometimes, and the absence of one in favour of the other isn't one of those things that the court considers a risk to the child.
Really, kiddo is going to really enjoy the trip to see his extended family. My three-year-old loves flying and spending time with her cousins and grandparents, even if I'm not there. Heck, she's probably more resilient than I am when it comes to being away.