Py Initialize fails - unable to load the file system codec

Question

I am attempting to put together a simple c   test project that uses an embedded python 3 2 interpreter  The project builds fine but Py Initialize raises a fatal error   Fatal Python error  Py Initialize  unable to load the file system codec LookupError  no codec search functions registered  can t find encoding   Minimal code    include  lt Python h gt   int main  int  char        Py Initialize       Py Finalize       return 0      The OS is 32bit Vista   The python version used is a python 3 2 debug build  built from sources using VC   10   The python d exe file from the same build runs without any problems   Could someone explain the problem and how to fix it  My own google-fu fails me   EDIT 1  After going through the python source code I ve found that  as the error says  no codec search functions have been registered  Both codec register and PyCodec Register are as they should be  It s just that nowhere in the code are any of these functions called   I don t really know what this means as I still have no idea when and from where these functions should have been called  The code that raises the error is entirely missing from the source of my other python build  3 1 3    EDIT 2  Answered my own question below

User · Answer

I had the same issue and found this question. However from the answers here I was not able to solve my problem. I started debugging the cpython code and thought that I might be discovered a bug. Therefore I opened a issue on the python issue tracker.

My mistake was that I did not understand that Py_SetPath clears all inferred paths. So one needs to set all paths when calling this function.

Link to the issue conversation

For completion I also copied the most important part of the conversation below.

My original issue text

I compiled the source of CPython 3.7.3 myself on Windows with Visual Studio 2017 together with some packages like e.g numpy. When I start the Python Interpreter I am able to import and use numpy. However when I am running the same script via the C-API I get an ModuleNotFoundError.

So the first thing I did, was to check if numpy is in my site-packages directory and indeed there is a folder named numpy-1.16.2-py3.7-win-amd64.egg. (Makes sense because the python interpreter can find numpy)

The next thing I did was to get some information about the sys.path variable created when running the script via the C-API.

#### sys.path content ####
C:\Work\build\product\python37.zip
C:\Work\build\product\DLLs
C:\Work\build\product\lib
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\2017\PROFESSIONAL\COMMON7\IDE\EXTENSIONS\TESTPLATFORM
C:\Users\rvq\AppData\Roaming\Python\Python37\site-packages

Examining the content of sys.path I noticed two things.

C:\Work\build\product\python37.zip has the correct path 'C:\Work\build\product\'. There was just no zip file. All my files and directory were unpacked. So I zipped the files to an archive named python37.zip and this resolved the import error.
C:\Users\rvq\AppData\Roaming\Python\Python37\site-packages is wrong it should be C:\Work\build\product\Lib\site-packages but I dont know how this wrong path is created.

The next thing I tried was to use Py_SetPath(L"C:/Work/build/product/Lib/site-packages") before calling Py_Initialize(). This led to

Fatal Python Error 'unable to load the file system encoding' ModuleNotFoundError: No module named 'encodings'

I created a minimal c++ project with exact these two calls and started to debug Cpython.

int main()
{
  Py_SetPath(L"C:/Work/build/product/Lib/site-packages");
  Py_Initialize();
}

I tracked the call of Py_Initialize() down to the call of

static int
zipimport_zipimporter___init___impl(ZipImporter *self, PyObject *path)

inside of zipimport.c

The comment above this function states the following:

Create a new zipimporter instance. 'archivepath' must be a path-like object to a zipfile, or to a specific path inside a zipfile. For example, it can be '/tmp/myimport.zip', or '/tmp/myimport.zip/mydirectory', if mydirectory is a valid directory inside the archive. 'ZipImportError' is raised if 'archivepath' doesn't point to a valid Zip archive. The 'archive' attribute of the zipimporter object contains the name of the zipfile targeted.

So for me it seems that the C-API expects the path set with Py_SetPath to be a path to a zipfile. Is this expected behaviour or is it a bug? If it is not a bug is there a way to changes this so that it can also detect directories?

PS: The ModuleNotFoundError did not occur for me when using Python 3.5.2+, which was the version I used in my project before. I also checked if I had set any PYTHONHOME or PYTHONPATH environment variables but I did not see one of them on my system.

Answer

This is probably a documentation failure more than anything else. We're in the middle of redesigning initialization though, so it's good timing to contribute this feedback.

The short answer is that you need to make sure Python can find the Lib/encodings directory, typically by putting the standard library in sys.path. Py_SetPath clears all inferred paths, so you need to specify all the places Python should look. (The rules for where Python looks automatically are complicated and vary by platform, which is something I'm keen to fix.)

Paths that don't exist are okay, and that's the zip file. You can choose to put the stdlib into a zip, and it will be found automatically if you name it the default path, but you can also leave it unzipped and reference the directory.

A full walk through on embedding is more than I'm prepared to type on my phone. Hopefully that's enough to get you going for now.

User · Answer

For those working in Visual Studio simply add the include  Lib and libs directories to the Include Directories and Library Directories under  Projects Properties - gt  Configuration Properties  gt  VC   Directories    For example I have Anaconda3 on my system and working with Visual Studio 2015 This is how the settings looks like  note the Include and Library directories       Edit   As also pointed out by bossi setting PYTHONPATH in your user Environment Variables section seems necessary   a sample input can be like this  in my case     C  Users Master Anaconda3 Lib C  Users Master Anaconda3 libs C  Users Master Anaconda3 Lib site-packages C  Users Master Anaconda3 DLLs   is necessary it seems     Also  you need to restart Visual Studio after you set up the PYTHONPATH in your user Environment Variables for the changes to take effect     Also note  that          Make sure the PYTHONHOME environment variable is set to the Python   interpreter you want to use  The C   projects in Visual Studio rely on   this variable to locate files such as python h  which are used when   creating a Python extension

User · Answer

Check the PYTHONPATH and PYTHONHOME environment variables and make sure they don t point to Python 2 x   http   bugs python org issue11288

User · Answer

So  for some reason the python dll fails to locate the encodings module  The python exe executable apparently finds it because it has the expected relative path  Modifying the search path works   The reason for all of this  Don t know but at least it works  I highly suspect a typo on my part somewhere  that s usually the reason for odd bugs it seems

User · Answer

I had this issue with python 3 5  anaconda 3  windows 7 32 bit  I solved it by moving my pythonX lib and pythonX dll files into my working directory and calling   Py SetPythonHome L C   Path  To  My  Python  Installation      before initialize so that it could find the headers that it needed  where my path was to      Anaconda3    The extra step of calling Py SetPythonHome was required for me or else I d end up getting other strange errors where python import files

User · Answer

I just ran into the exact same problem  same Python version  OS  code  etc    You just have to copy Python s Lib  directory in your program s working directory   on VC it s the directory where the  vcproj is

User · Answer

From python3k  the startup need the encodings module  which can be found in PYTHONHOME Lib directory   In fact  the API Py Initialize    do the init and import the encodings module  Make sure PYTHONHOME Lib is in sys path and check the encodings module is there

User · Answer

The core reason is quite simple  Python does not find its modules directory  so it can of course not load encodings  too  Python doc on embedding says  Py Initialize   calculates the module search path based upon its best guess       In particular  it looks for a directory named lib pythonX Y   Yet  if the modules are installed in  just  lib - relative to the python binary - above guess is wrong   Although docs says that PYTHONHOME and PYTHONPATH are regarded  we observed that this was not the case  their actual presence or content was completely irrelevant   The only thing that had an effect was a call to Py SetPath   with e g   path-to  lib as argument before Py Initialize      Sure this is only an option for an embedding scenario where one has direct access and control over the code  with a ready-made solution  special steps may be necessary to solve the issue

User · Answer

I had the problem and was tinkering with different solutions mentioned here  Since I was running my project from Visual Studio  apparently  I needed to set the environment path inside Visual Studio and not the system path   Adding a simple PYTHONHOME PATH TO PYTHON DIR in the project solution properties environment solved the problem

User · Answer

Ran into the same thing trying to install brew s python3 under Mac OS  The issue here is that in Mac OS  homebrew puts the  real  python a whole layer deeper than you think  You would think from the homebrew output that    echo  PYTHONHOME  usr local Cellar python3 3 6 2    echo  PYTHONPATH  usr local Cellar python3 3 6 2 bin   would be correct  but invoking  PYTHONPATH python3 immediately crashes with the abort 6  can t find encodings   This is because although that  PYTHONHOME looks like a complete installation  having a bin  lib etc  it is NOT the actual Python  which is in a Mac OS  Framework   Do this   PYTHONHOME  usr local Cellar python3 3 x y Frameworks Python framework Versions 3 x PYTHONPATH  PYTHONHOME bin    substituting version numbers as appropriate  and it will work fine

User · Answer

In my cases  for windows  if you have multiple python versions installed  if PYTHONPATH is pointing to one version the other ones didn t work  I found that if you just remove PYTHONPATH  they all work fine

User · Answer

There appears to be something going wrong with the release build either failing to include the appropriate codecs or else misidentifying the codec to use for system APIs  Since the python d executable is working  what does that return for os getfsencoding     Use the C API to call that between your Initialize Finalize calls

User · Answer

For me this happened when I updated Python 64 bit from 3 6 4 to 3 6 5  It threw some error like  unable to extract python dll  Do you have permissions     Pycharm also failed to load interpreter  even though I reloaded it in settings  Running python command gave same error  with and without administrator mode    Reason  There was error in installation of Python  include folder in python installation directory C  Users USERNAME AppData Local Programs Python Python36 was missing   Reinstalling Python also dint solve the issue  Not removal and install   Solution  Uninstall Python and Install of Python again    Because running installer was just extracting same files excluding include folder

User · Answer

Parts of this have been mentioned before  but in a nutshell this is what worked for my environment where I have multiple Python installs and my global OS environment set-up to point to a different install than the one I attempt to work with when encountering the problem   Make sure your  local or global  environment is fully set-up to point to the install you aim to work with  e g  you have two  or more  installs of  let s say a python27 and python33  sorry these are windows paths but the following should be valid for equivalent UNIX-style paths just as well  please let me know about anything I m missing here  probably the DLLs path might differ     C  python27 x86  C  python33 x64  Now  if you intend to work with your python33 install but your global environment is pointing to python27  make sure you update your environment as such  while PATH and PYTHONHOME may be optional  e g  if you temporarily work in a local shell     PATH  C  python33 x64  PATH    PYTHONPATH  C  python33 x64 DLLs C  python33 x64 Lib C  python33 x64 Lib site-packages   PYTHONHOME C  python33 x64  Note  that you might need want to append any other library paths to your PYTHONPATH if required by your development environment  but having your DLLs  Lib and site-packages properly set-up is of prime importance   Hope this helps

[c++] Py_Initialize fails - unable to load the file system codec

Examples related to c++

Examples related to python