[python] Find all CSV files in a directory using Python

How can I find all files in directory with the extension .csv in python?

This question is related to python csv file directory find

The answer is


While solution given by thclpr works it scans only immediate files in the directory and not files in the sub directories if any. Although this is not the requirement but just in case someone wishes to scan sub directories too below is the code that uses os.walk

import os
from glob import glob
PATH = "/home/someuser/projects/someproject"
EXT = "*.csv"
all_csv_files = [file
                 for path, subdir, files in os.walk(PATH)
                 for file in glob(os.path.join(path, EXT))]
print(all_csv_files)

Copied from this blog.


Use the python glob module to easily list out the files we need.

import glob
path_csv=glob.glob("../data/subfolrder/*.csv")

By using the combination of filters and lambda, you can easily filter out csv files in given folder.

import os

files = os.listdir("/path-to-dir")    
files = list(filter(lambda f: f.endswith('.csv'), files))

# lambda returns True if filename name ends with .csv or else False
# and filter function uses the returned boolean value to filter .csv files from list files.

use Python OS module to find csv file in a directory.

the simple example is here :

import os

# This is the path where you want to search
path = r'd:'

# this is the extension you want to detect
extension = '.csv'

for root, dirs_list, files_list in os.walk(path):
    for file_name in files_list:
        if os.path.splitext(file_name)[-1] == extension:
            file_name_path = os.path.join(root, file_name)
            print file_name
            print file_name_path   # This is the full path of the filter file

from os import listdir

def find_csv_filenames( path_to_dir, suffix=".csv" ):
    filenames = listdir(path_to_dir)
    return [ filename for filename in filenames if filename.endswith( suffix ) ]

The function find_csv_filenames() returns a list of filenames as strings, that reside in the directory path_to_dir with the given suffix (by default, ".csv").

Addendum

How to print the filenames:

filenames = find_csv_filenames("my/directory")
for name in filenames:
  print name

You could just use glob with recursive = true, the pattern ** will match any files and zero or more directories, subdirectories and symbolic links to directories.

import glob, os

os.chdir("C:\\Users\\username\\Desktop\\MAIN_DIRECTORY")

for file in glob.glob("*/.csv", recursive = true):
    print(file)

Many (linked) answers change working directory with os.chdir(). But you don't have to.

Recursively print all CSV files in /home/project/ directory:

pathname = "/home/project/**/*.csv"

for file in glob.iglob(pathname, recursive=True):
    print(file)

Requires python 3.5+. From docs [1]:

  • pathname can be either absolute (like /usr/src/Python-1.5/Makefile) or relative (like ../../Tools/*/*.gif)
  • pathname can contain shell-style wildcards.
  • Whether or not the results are sorted depends on the file system.
  • If recursive is true, the pattern ** will match any files and zero or more directories, subdirectories and symbolic links to directories

[1] https://docs.python.org/3/library/glob.html#glob.glob


import os

path = 'C:/Users/Shashank/Desktop/'
os.chdir(path)

for p,n,f in os.walk(os.getcwd()):
    for a in f:
        a = str(a)
        if a.endswith('.csv'):
            print(a)
            print(p)

This will help to identify path also of these csv files


Please use this tested working code. This function will return a list of all the CSV files with absolute CSV file paths in your specified path.

import os
from glob import glob

def get_csv_files(dir_path, ext):
    os.chdir(dir_path)
    return list(map(lambda x: os.path.join(dir_path, x), glob(f'*.{ext}')))

print(get_csv_files("E:\\input\\dir\\path", "csv"))

I had to get csv files that were in subdirectories, therefore, using the response from tchlpr I modified it to work best for my use case:

import os
import glob

os.chdir( '/path/to/main/dir' )
result = glob.glob( '*/**.csv' )
print( result )

import os
import glob

path = 'c:\\'
extension = 'csv'
os.chdir(path)
result = glob.glob('*.{}'.format(extension))
print(result)

This solution uses the python function filter. This function creates a list of elements for which a function returns true. In this case, the anonymous function used is partial matching '.csv' on every element of the directory files list obtained with os.listdir('the path i want to look in')

import os

filepath= 'filepath_to_my_CSVs'  # for example: './my_data/'

list(filter(lambda x: '.csv' in x, os.listdir('filepath_to_my_CSVs')))

Examples related to python

programming a servo thru a barometer Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? python variable NameError Why my regexp for hyphenated words doesn't work? Comparing a variable with a string python not working when redirecting from bash script is it possible to add colors to python output? Get Public URL for File - Google Cloud Storage - App Engine (Python) Real time face detection OpenCV, Python xlrd.biffh.XLRDError: Excel xlsx file; not supported Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to csv

Pandas: ValueError: cannot convert float NaN to integer Export result set on Dbeaver to CSV Convert txt to csv python script How to import an Excel file into SQL Server? "CSV file does not exist" for a filename with embedded quotes Save Dataframe to csv directly to s3 Python Data-frame Object has no Attribute (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape How to write to a CSV line by line? How to check encoding of a CSV file

Examples related to file

Gradle - Move a folder from ABC to XYZ Difference between opening a file in binary vs text Angular: How to download a file from HttpClient? Python error message io.UnsupportedOperation: not readable java.io.FileNotFoundException: class path resource cannot be opened because it does not exist Writing JSON object to a JSON file with fs.writeFileSync How to read/write files in .Net Core? How to write to a CSV line by line? Writing a dictionary to a text file? What are the pros and cons of parquet format compared to other formats?

Examples related to directory

Moving all files from one directory to another using Python What is the reason for the error message "System cannot find the path specified"? Get folder name of the file in Python How to rename a directory/folder on GitHub website? Change directory in Node.js command prompt Get the directory from a file path in java (android) python: get directory two levels up How to add 'libs' folder in Android Studio? How to create a directory using Ansible Troubleshooting misplaced .git directory (nothing to commit)

Examples related to find

Find a file by name in Visual Studio Code Explaining the 'find -mtime' command find files by extension, *.html under a folder in nodejs MongoDB Show all contents from all collections How can I find a file/directory that could be anywhere on linux command line? Get all files modified in last 30 days in a directory FileNotFoundError: [Errno 2] No such file or directory Linux find and grep command together find . -type f -exec chmod 644 {} ; Find all stored procedures that reference a specific column in some table