Can he do this? Sure - but you can then go to the Police and report it stolen.
A few things to consider:
1. The car is registered to you, so if he takes it without your consent, the Police will treat that as theft and he will most likely be charged if you report it.
2. If you were living together for 2 years or more, then you were in a de-facto relationship in which case, both the car and any oustanding debt could be considered joint property. But that still doesn't mean that he can just come and take it.
3. More important than 1 and 2, is the fact that the finance is in his name. If that's the case, then he can simply stop the payments at any time; inform the financier that he is defaulting; and the financier can come and repossess the car. This leaves you without a car and there's absolutely nothing you can do about it because you have no rights to the car at all if he defaults.
So the most important things you should consider at the moment, are that the financier essentially owns the car and your ex, solely out of spite if he wants to, can easily have the car taken from you.
So you don't have many options if you want your ownership to be "secure".
If you want to keep the car, then you will probably have to come to some agreement that involves refinancing in your name.
If you leave things as they are, he can simply default on the payments at any time to screw you over.
Claiming something was a gift is very difficult to prove and given that there is a debt involved, it really won't help you here anyway. That debt still has to be repaid and if you want to keep the car, it will most likely be you that ends up having to pay it out. The fact that he's threatening to take the car back is a pretty clear indicator that he's not going to let you keep it any other way.