Yes. Will you actually go to prison is a different issue, but you can go to prison for trespass.
The very fact they've put a fence up is notice to the world that they want to restrict who enters the site - and given it is a construction site, the safety aspect at least makes that a reasonable thing. Owners of private property generally have the right to determine who is allowed to enter that property, and who is excluded. You've no doubt heard of the notion of "break and enter" as a criminal act? It's actually two separate actions:
- "breaking" the seal, which includes things like cutting a padlock, forcing a lock, breaking a window and so on; and
- "entering" the property.
You can break without entering, and you can enter without breaking. Hopping a fence is the entering part.