I tend to agree with Harald. I do not see, or understand how man handling him is going to build trust with him, or any other animal.
A slow gentle hand would show many more rewards with him. Sounds like he is having a fight or flight responce. Man handling him will only make him worse.
JMHO.