In sum: The Dept. appreciates local governments’ concerns for public safety. The Department likewise puts a high priority on safety. However, no evidence exists that hunting with rifles is more dangerous than hunting with shotguns.
Conventional wisdom holds that shotguns are inherently less risky than rifles when hunting deer. This is evidenced by the fact that the PGC as well as other states have established shotgun only hunting areas. This study, however, has concluded that this is not always the case.
This safety thing seems to have been studied pretty well and a little googling should help you to form a reasoned opinion on it. So far, I haven't found any data supporting the safety side of prohibiting rifles.