Was there an agreement as to how much you would be paying off with each bit of work that you did for him? I.e. at an hourly rate? Or in small lump sums? Did you keep track of how much you were working and how much you were paying off the debt?
Usually nothing would prevent you from enforcing this type of agreement simply because its not in writing, however it will be his word against yours. If there was not even any agreement about how much you would be paying off per hour or unit of work you did for him, then there is no way of knowing whether it is paid off or not. However, there are things which a court will presume from your prior course of dealings. Such as the hourly rate that you worked for him prior to doing work to pay off the debt.
I would put it to your employer to show you financial records that prove you have not yet cleared the debt, then compare it wiht your own records of when you worked.