Who decided he was mentally unstable? A psychologist? Or an ex who is going through a rough break-up? Or possibly the friend of the ex going through a rough break-up?
There have been dozens and dozens of cases where a mother has tried to claim to the court that the father was "mentally unstable", only to lose primary carership all together because the allegations were found to be false and instead indicative of the mother's inability to support a relationship between the father and his child - a relationship which is, as a rule of thumb, usually considered to be in the best interests of the child.
The reasons you have thus far provided (e.g. "mental instability" and being "financially inept at the moment") won't fly in court for forcing a child to grow up without a dad. Your friend's adult problems have no bearing on the child's problems, one of which will more than likely be the lack of father figure in his/her life and the role you and your friend played in facilitating that issue, especially when the father clearly wants to be involved.
I would strongly urge your friend to carefully consider the potential consequences of pursuing this to court, instead of just getting over whatever emotional bruises she might be suffering and letting the child have a father.