While children have a legal right to know, spend time and communicate with their grandparents on a regular basis, grandparents don't have the benefit of parental responsibility, which means they are not legally responsible for making major long-term decisions about the child's care welfare and development, which includes care and living arrangements.
However, they can apply to the court for parental responsibility and will need to argue that the orders they are seeking are in the best interests of the child.
If the grandparents do take the child from your care without your consent, you can file for a recovery order to have the child returned to your care.