[python] Reading file using relative path in python project

Say I have a python project that is structured as follows:

project
    /data
        test.csv
    /package
        __init__.py
        module.py
    main.py

__init__.py:

from .module import test

module.py:

import csv

with open("..data/test.csv") as f:
    test = [line for line in csv.reader(f)]

main.py:

import package

print(package.test)

When I run main.py I get the following error:

 C:\Users\Patrick\Desktop\project>python main.py
Traceback (most recent call last):
  File "main.py", line 1, in <module>
    import package
  File "C:\Users\Patrick\Desktop\project\package\__init__.py", line 1, in <module>
    from .module import test
  File "C:\Users\Patrick\Desktop\project\package\module.py", line 3, in <module>
    with open("../data/test.csv") as f:
FileNotFoundError: [Errno 2] No such file or directory: '../data/test.csv'

However, if I run module.py from the package directory I get no errors. So it seems that the relative path used in open(...) is only relative to where the originating file is being run from (i.e __name__ == "__main__")? I don't want to use absolute paths. What are some ways to deal with this?

I was thundered when the following code worked.

import os

for file in os.listdir("../FutureBookList"):
    if file.endswith(".adoc"):
        filename, file_extension = os.path.splitext(file)
        print(filename)
        print(file_extension)
        continue
    else:
        continue

So, I checked the documentation and it says:

Changed in version 3.6: Accepts a path-like object.

path-like object:

An object representing a file system path. A path-like object is either a str or...

I did a little more digging and the following also works:

with open("../FutureBookList/file.txt") as file:
   data = file.read()

try

with open(f"{os.path.dirname(sys.argv[0])}/data/test.csv", newline='') as f:

For Python 3.4+:

import csv
from pathlib import Path

base_path = Path(__file__).parent
file_path = (base_path / "../data/test.csv").resolve()

with open(file_path) as f:
    test = [line for line in csv.reader(f)]

This worked for me.

with open('data/test.csv') as f:

 

My Python version is Python 3.5.2 and the solution proposed in the accepted answer didn't work for me. I've still were given an error

FileNotFoundError: [Errno 2] No such file or directory

when I was running my_script.py from the terminal. Although it worked fine when I run it through Run/Debug Configurations from PyCharm IDE (PyCharm 2018.3.2 (Community Edition)).

Solution:

instead of using:

my_path = os.path.abspath(os.path.dirname(__file__)) + some_rel_dir_path 

as suggested in the accepted answer, I used:

my_path = os.path.abspath(os.path.dirname(os.path.abspath(__file__))) + some_rel_dir_path

Explanation: Changing os.path.dirname(__file__) to os.path.dirname(os.path.abspath(__file__)) solves the following problem:

When we run our script like that: python3 my_script.py the __file__ variable has a just a string value of "my_script.py" without path leading to that particular script. That is why method dirname(__file__) returns an empty string "". That is also the reson why my_path = os.path.abspath(os.path.dirname(__file__)) + some_rel_dir_path is actually the same thing as my_path = some_rel_dir_path. Consequently FileNotFoundError: [Errno 2] No such file or directory is given when trying to use open method because there is no directory like "some_rel_dir_path".

Running script from PyCharm IDE Running/Debug Configurations worked because it runs a command python3 /full/path/to/my_script.py (where "/full/path/to" is specified by us in "Working directory" variable in Run/Debug Configurations) instead of justpython3 my_script.py like it is done when we run it from the terminal.

Hope that will be useful.