Your horse doesn’t learn anything if he is being fearful. First of all he shouldn’t be scared of you or any of the tools you are using.
If your horse does anything out of fear he is most likely just reacting but not thinking. Your horse won’t trust you if he doesn’t respect you and vice versa.
You need to prove to your horse that you are a great leader that he would want to follow. You can earn your horse’s respect and trust by moving their feet which is exactly what they do to each other in the herd.