I think we give deer too much credit. They can not "reason" like a human. He was just reacting to the situation. He probably heard, saw or smelled you and he was getting the heck out of there.

Actually, it sounds like he just heard something he couldn't identify. If he would have seen and/or smelled you he would have bolted and left the area.

I say he will return to that area at some point. I don't think deer remember every time they are startled and avoid those areas forever.