First up, no, without orders, then, inside Australia,
you basically can't refuse, or stop, him.
(overseas, very different story)
Second up, if it's that far in advance, then I do not suspect a ploy to abduct get the child out of your reach.
Anybody who was going to steal the child and hide it behind Corona-closed borders
would not be giving you several months advance notice.
Third up, I am slow to accept as reasonable your claim of "...I'm concerned that he won't handle it".
Rather, I wonder what is it about the child as an individual, even given his tender years
that he is so lacking in relisilence that he "can't handle" (whatever that means)
two weeks in the company of a parent he knows well, and surrounded by his family.
The child is entitled to know and have a relationship with both sides of his family.
Absent orders, you don't get to obstruct that.
Bottom line, if you really want to resist, apply for orders.