What do the terms of the restraining order say - you have to read them carefully? Why are you visiting that person if you have a restraining order against them? Have a look at the Legal Aid WA "Violence restraining orders - information" page that has some useful information.
Being an initiator of contact when you're supposed to be the aggrieved tends to create difficulties for police if anything happens, to the effect of reluctance to investigate potential breaches. After all, how can you say you fear for your safety when you can't stay away from them?
You do need to read the orders and try to consider them as though they work both ways. If he can't approach you within 100m, for example, you probably shouldn't be approaching him within 100m either.