drop some trees to build a cover wall on your side of the line or plant a plot screen if at all possible.

I would also find out who they are talk to them to let them know you will be hunting the other side. Be polite but let them know you will stick to your property and expect them to stick on theirs to hunt. Th exception might be tracking a deer but better to have this worked out ahead of time for both of you.

Just knowing you will be hunting it might keep them "honest" and on their side of the fence line. Good luck, these situations can turn bad but don't have to....