It's not ideal that the birthday party was organised at a time when your daughter would be spending time with you. In a perfect world, the mother would have either agreed to the change of time or RSVP'd the child's non-attendance on her commitment to supporting the child's relationship and time spent with you.
However, since the mother failed in her duty to co-parent properly, that duty was then left to you, and in my view, the better choice would have been to take the child to the party.
The reason I say this is because the average kid not subject to a court hearing wouldn't miss out on a birthday party with friends so that their parents could spend more time with them.
I understand the dynamic is different. It's easy to want to protect what little time you do have with her so that she has plenty of time to build a stronger bond with you. However, if you treat your time with your daughter like a special occasion instead of just regular parenting time, so too will the court. If you treat it like parenting time, where you're invested in even the inconvenient parts of parenting time, such as kids parties and soccer games and gym classes, the court will treat you accordingly.
I also don't agree with your comments to the child about "mummy being angry" because that's a discussion, however brief, with the child about adult conflict. I also think it would be better not to treat your parenting role as a decision for the child to make by saying you'll be there until she says she doesn't want you around. That suggests she has a choice, when she really doesn't. You're a parent, she loves you, you're a given in her life, no matter what she says, aren't you?
That's my view, anyhow. In the same token, I can't imagine the court putting a lot of weight on a missed birthday party. At the end of the day, what your daughter does while in your care is your decision, not the mother's. That's not to say the court can't order attendance at extra-curricular activities, but a mark against the mother, I would argue, is not agreeing to the change of time. To me, that says she doesn't place a lot of importance on your daughter spending time with you, if she insists that your time with the child should be spent watching her play with other kids at a birthday party.
Anyway, that's my two cents.