Passports do not belong to the people they are issued to; they remain the property of the Commonwealth (s54 of the Australian Passports Act 2005). It's a criminal offence to have possession or control of someone else's passport without reasonable excuse (s32). The securing of a debt would not likely be considered a reasonable excuse by a court, in my opinion.
Given that the Commonwealth has the heads of power relating to border control (immigration and emigration), and a passport is an Australian travel document, it's also unlikely that any State Court has the power to make an order with respect to the surrender of a passport (unless operating under Commonwealth jurisdiction).