It's very hard to hand tame one that is not tame. It takes time and patience. The bird is afraid of you and more than likely really, really afraid of your hands.
If you've had him for three weeks, he has to be eating something or he would have died a couple of weeks ago.
Offer him spray millet. Lovebirds (well most birds) love spray millet. Also offer him fresh, leafy green veggies and baby carrots. If you don't already know this, birds on an all seed diet are not very healthy birds.
You can sit by his cage and get him acclimated to you by talking to him, etc. Move very slowly around him, especially when you have to put your hand in his cage. It will take some time before he stops freaking out and trying to get away from you. At that point I would then try to hold him.
You probably won't be able to tame him, but it is well worth a try. Lovebirds make excellent pets. I raise them. Here are a few of my pets.
Here's a picture of Wilma.
Here's a picture of Margalo.
Here's a picture of Spitfire.
Here's a picture of Helga who isn't tame and would like nothing better than to get ahold of my finger. LOL