Two issues
1) Your partner ... Your ex can't demand you don't live with, or stop living with anybody ... IF your partner represents a danger to the child in some way, then there are things your ex can do to have those concerns investigated and if warranted, seek an order to prevent the partner being around the child.
2) Distance .... Unless you have a current court order restricting how far you can move, then you can ... 50 minutes is not at all significant.
Options ... Engage in meaningful genuine dialogue in an attempt to resolve the concerns or ignore