Some interesting comments on the safety: http://rule-303.blogspot.com/2009/10/shotgun-only-rationale-for-deer-hunting.html

Wisconsin's rule change allowing rifles (see p. 4 for the safety discussion): http://www.wisctowns.com/uploads/ckfiles/files/Statewide%20Rifle%20Memo%20v1%202.pdf

One quote:

Quote
In sum: The Dept. appreciates local governments’ concerns for public safety. The Department likewise puts a high priority on safety. However, no evidence exists that hunting with rifles is more dangerous than hunting with shotguns.
Pennsylvania study: http://lbfc.legis.state.pa.us/factsheets/2007/rifle_shotgun_webpost.pdf

One Quote:

Quote
Conventional wisdom holds that shotguns are inherently less risky than rifles when
hunting deer. This is evidenced by the fact that the PGC as well as other states have
established shotgun only hunting areas. This study, however, has concluded that
this is not always the case.
This safety thing seems to have been studied pretty well and a little googling should help you to form a reasoned opinion on it. So far, I haven't found any data supporting the safety side of prohibiting rifles.