Retaining wall vs raised garden bed

[happyneighbours]

I have raised garden beds sitting against my back boundary fence with a height of about 1.3mtrs and a depth of approx 1.5 meters from the fence. This was put in instead of a retaining wall by previous owners and the fence sits on top of it on the boundary line. I want to remove the garden beds and reclaim some of my yard for a few reasons, one being personal safety (im to old to be climbing up onto meter high garden beds). When I remove the garden beds there will be no retaining wall and the fence structure will be compromised. Note my property sits lower than my neighbours. Who is responsible for installing and paying for the retaining wall? I live in Ipswich QLD