You can do either, but as always, it's a good idea to get legal advice from a lawyer before acting, even if you intend on self-representing.
You can file an initiating application with either the Federal Circuit Court or the Family Court of Australia, but the Family Court will ordinarily only deal with applications that are complex or require special attention (such as Magellan Listed cases), so it's best to file in the Federal Circuit Court first and let the judge transfer it if s/he sees fit.
Is this for a parenting matter or property?