Unfortunately, you do still have a duty to actively (not just passively) follow the orders, which means positively encouraging the child to spend time with the other parent. Speaking negatively of the other parent or implying the child has a choice about the matter is not considered positive encouragement to spend time with the other parent.
If the court felt the other parent posed an unacceptable risk of harm to the child, it would have ordered supervised time or no time at all, and I daresay a domestic violence restraining order would have been granted. If the parents can't agree on care arrangements and then ask the court to decide their parenting decisions for them, that decision is taken out of both your hands and the child's.
What you can do is invite the other parent to mediation (or family dispute resolution) to discuss reaching a new agreement by consent. You can do a child-inclusive conference whereby a child expert will speak to the child and determine their feelings about the matter so that information can be relayed to the parents, enabling them to consider the child's wants before making a decisions.
If you still can't reach agreement, you can apply to the court, but the court will not amend final parenting orders unless the principles of Rice & Asplund are met, which are that circumstances have changed significantly in that the orders no longer meet the best interests of the child. It may be a case of the court ordering that the child spend time with the other parent in accordance with his wishes.