NSW Beneficiary to Mother's House - What are My Rights?

If your sister is the executor, then it is appropriate for the agent to deal with her.
The agent is not required to deal with anybody else.

 

An executor's duty is to obtain the best possible price for the house
(this is true of any asset in which the executor is dealing).
Apart from that, there's nothing special about it being a deceased estate.

If the executor set a reserve price at auction,*
but the bids did not exceed the reserve,
then executor is not required to accept the highest bid on the day,
just because it's the highest bid on the day.**

On the other hand, if (as the real estate agents say)
the house "goes on without reserve",
then it sells on the fall of the hammer, and that's it, deal done.

Perhaps you could have a discussion with the executor about

• getting a proper valuation
  (not a mere speculation of market price by an agent), and;
  
  
• maybe changing agents.

-------------------------------------
* Find out - by asking the executor
** Auctioneers call this "a Lot being passed in"