On Mac OS X, the way I enabled Jenkins to pull from my (private) Github repo is:
First, ensure that your user owns the Jenkins directory
sudo chown -R me:me /Users/Shared/Jenkins
Then edit the LaunchDaemon plist for Jenkins (at /Library/LaunchDaemons/org.jenkins-ci.plist) so that your user is the GroupName and the UserName:
<key>GroupName</key>
<string>me</string>
...
<key>UserName</key>
<string>me</string>
Then reload Jenkins:
sudo launchctl unload -w /Library/LaunchDaemons/org.jenkins-ci.plist
sudo launchctl load -w /Library/LaunchDaemons/org.jenkins-ci.plist
Then Jenkins, since it's running as you, has access to your ~/.ssh directory which has your keys.