[python] "OSError: [Errno 1] Operation not permitted" when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection)

I'm trying to install Scrapy Python framework in OSX 10.11 (El Capitan) via pip. The installation script downloads the required modules and at some point returns the following error:

OSError: [Errno 1] Operation not permitted: '/tmp/pip-nIfswi-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

I've tried to deactivate the rootless feature in OSX 10.11 with the command:

sudo nvram boot-args="rootless=0";sudo reboot

but I still get the same error when the machine reboots.

Any clue or idea from my fellow StackExchangers?

If it helps, the full script output is the following:

sudo -s pip install scrapy
Collecting scrapy
  Downloading Scrapy-1.0.2-py2-none-any.whl (290kB)
    100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 290kB 345kB/s 
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from scrapy)
Collecting w3lib>=1.8.0 (from scrapy)
  Downloading w3lib-1.12.0-py2.py3-none-any.whl
Collecting lxml (from scrapy)
  Downloading lxml-3.4.4.tar.gz (3.5MB)
    100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 3.5MB 112kB/s 
Collecting Twisted>=10.0.0 (from scrapy)
  Downloading Twisted-15.3.0.tar.bz2 (4.4MB)
    100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 4.4MB 94kB/s 
Collecting six>=1.5.2 (from scrapy)
  Downloading six-1.9.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Twisted>=10.0.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from zope.interface>=3.6.0->Twisted>=10.0.0->scrapy)
Installing collected packages: six, w3lib, lxml, Twisted, scrapy
  Found existing installation: six 1.4.1
    DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling six-1.4.1:
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/basecommand.py", line 223, in main
status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/commands/install.py", line 299, in run
root=options.root_path,
  File "/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/req/req_set.py", line 640, in install
requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/req/req_install.py", line 726, in uninstall
paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/req/req_uninstall.py", line 125, in remove
renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/utils/__init__.py", line 314, in renames
shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
copy2(src, real_dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
os.chflags(dst, st.st_flags)
OSError: [Errno 1] Operation not permitted: '/tmp/pip-nIfswi-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

This question is related to python macos python-2.7 scrapy

The answer is


I was getting the same error on on my MacOS Sierra. I followed these steps and successfully able to install scarpy package.

1. sudo pip install --ignore-installed six
2. sudo pip install --ignore-installed scrapy

MacBook-Air:~ shree$ scrapy version
Scrapy 1.4.0

This command would work perfectly fine :D

sudo -H pip install --upgrade package_name --ignore-installed six


This did the trick for me.

sudo pip install --ignore-installed scrapy


This did the trick for me:

   sudo pip install scrapy --ignore-installed six

Tried a combination of some answers and this eventually worked:

sudo -H pip install --upgrade --ignore-installed awsebcli

Cheers


You should disable "System Integrity Protection" which is a new feature in El Capitan.

First, you should run the command for rootless config on your terminal

# nvram boot-args="rootless=0"
# reboot

Then, you should run the command below on recovery partition's terminal (Recovery OS)

# csrutil disable
# reboot

I've just solved my problem like that. I'm not sure that the first part is necessary. Try as you like.

--WARNING

You should enable SIP again after everything works;

Simply reboot again into Recovery Mode and run in terminal

# csrutil enable

csrutil: Configuring System Integrity Protection


Warnings

I would suggest very strongly against modifying the system Python on Mac; there are numerous issues that can occur.

Your particular error shows that the installer has issues resolving the dependencies for Scrapy without impacting the current Python installation. The system uses Python for a number of essential tasks, so it's important to keep the system installation stable and as originally installed by Apple.

I would also exhaust all other possibilities before bypassing built in security.

Package Manager Solutions:

Please look into a Python virtualization tool such as virtualenv first; this will allow you to experiment safely.

Another useful tool to use languages and software without conflicting with your Mac OS is Homebrew. Like MacPorts or Fink, Homebrew is a package manager for Mac, and is useful for safely trying lots of other languages and tools.

"Roll your own" Software Installs:

If you don't like the package manager approach, you could use the /usr/local path or create an /opt/local directory for installing an alternate Python installation and fix up your paths in your .bashrc. Note that you'll have to enable root for these solutions.

How to do it anyway:

If you absolutely must disable the security check (and I sincerely hope it's for something other than messing with the system languages and resources), you can disable it temporarily and re-enable it using some of the techniques in this post on how to Disable System Integrity-Protection.


  1. -- close SIP(system Integrity Protection) -- then reboot, use command +R to enter debug mode, then select terminal: csrutil disable reboot

2.

sudo C_INCLUDE_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/libxml2 :/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/libxml2/libxml :/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include pip install scrapy --ignore-installed six

3. -- then remove old six, install it again sudo rm -rf /Library/Python/2.7/site-packages/six* sudo rm -rf /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six* sudo pip install six

4. -- then set it back csrutil enable reboot

-- crappy works now


The high voted answers didn't work for me, it seems to work for El Capitan users. But for MacOS Sierra users try the following steps

  1. brew install python
  2. sudo pip install --user <package name>

pip install --ignore-installed six

Would do the trick.

Source: github.com/pypa/pip/issues/3165


Restart Mac -> hold down "Command + R" after the startup chime -> Opens OS X Utilities -> Open Terminal and type "csrutil disable" -> Reboot OS X -> Open Terminal and check "csrutil status"


Sometimes such behavior may be achieved if you try to install python3 lib in python2 folder using pip instead of pip3.


I tried to install AWS via pip in El Capitan but this error appear

OSError: [Errno 1] Operation not permitted: '/var/folders/wm/jhnj0g_s16gb36y8kwvrgm7h0000gp/T/pip-wTnb_D-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

I found the answer here

sudo -H pip install awscli --upgrade --ignore-installed six

It works for me :)


install python again:

brew install python

try it again:

sudo pip install scrapy

works for me, hope it can help


I was missing a dependency somewhere else along the line, so I installed the other requirements for the project like this:

pip install --user -r requirements.txt

As the other answers said, it's because of the new System Integrity Protection, but I believe the other answers are overcomplicated.

If you're only gonna use that package in the current user, you should be able to install it just fine, without the need to disable the SIP, by using the --user flag. Like this:

sudo pip install --user packagename

it work for me:

pip install scrapy --user -U

Examples related to python

programming a servo thru a barometer Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? python variable NameError Why my regexp for hyphenated words doesn't work? Comparing a variable with a string python not working when redirecting from bash script is it possible to add colors to python output? Get Public URL for File - Google Cloud Storage - App Engine (Python) Real time face detection OpenCV, Python xlrd.biffh.XLRDError: Excel xlsx file; not supported Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to macos

Problems with installation of Google App Engine SDK for php in OS X dyld: Library not loaded: /usr/local/opt/openssl/lib/libssl.1.0.0.dylib dyld: Library not loaded: /usr/local/opt/icu4c/lib/libicui18n.62.dylib error running php after installing node with brew on Mac Could not install packages due to an EnvironmentError: [Errno 13] How do I install Java on Mac OSX allowing version switching? Git is not working after macOS Update (xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools) Can't compile C program on a Mac after upgrade to Mojave You don't have write permissions for the /Library/Ruby/Gems/2.3.0 directory. (mac user) How can I install a previous version of Python 3 in macOS using homebrew? Could not install packages due to a "Environment error :[error 13]: permission denied : 'usr/local/bin/f2py'"

Examples related to python-2.7

Numpy, multiply array with scalar Not able to install Python packages [SSL: TLSV1_ALERT_PROTOCOL_VERSION] How to create a new text file using Python Could not find a version that satisfies the requirement tensorflow Python: Pandas pd.read_excel giving ImportError: Install xlrd >= 0.9.0 for Excel support Display/Print one column from a DataFrame of Series in Pandas How to calculate 1st and 3rd quartiles? How can I read pdf in python? How to completely uninstall python 2.7.13 on Ubuntu 16.04 Check key exist in python dict

Examples related to scrapy

Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org TypeError: Object of type 'bytes' is not JSON serializable "OSError: [Errno 1] Operation not permitted" when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection) pip is not able to install packages correctly: Permission denied error Can scrapy be used to scrape dynamic content from websites that are using AJAX?