Is he a baby? If so, most of them start out a little untrusting of that gigantic hand coming at them. If he's still small, you may want to try hand feeding him his favorite foods and see if that helps. If he's big, you could try it, but with gloves on!
I would continue taking him out and handling him for as long as he'll let you. If he starts acting out, don't let him know that he scares you, or he'll take full advantage. Just be patient and consistent, and he will get accustomed to you eventually!