Python recursive folder read

Question

I have a C   Obj-C background and I am just discovering Python  been writing it for about an hour   I am writing a script to recursively read the contents of text files in a folder structure   The problem I have is the code I have written will only work for one folder deep  I can see why in the code  see  hardcoded path   I just don t know how I can move forward with Python since my experience with it is only brand new   Python Code   import os import sys  rootdir   sys argv 1   for root  subFolders  files in os walk rootdir        for folder in subFolders          outfileName   rootdir         folder     py-outfile txt    hardcoded path         folderOut   open  outfileName   w            print  outfileName is     outfileName          for file in files              filePath   rootdir         file             f   open  filePath   r                toWrite   f read               print  Writing      toWrite      to    filePath             folderOut write  toWrite               f close            folderOut close

User · Answer

Agree with Dave Webb  os walk will yield an item for each directory in the tree  Fact is  you just don t have to care about subFolders   Code like this should work   import os import sys  rootdir   sys argv 1   for folder  subs  files in os walk rootdir       with open os path join folder   python-outfile txt     w   as dest          for filename in files              with open os path join folder  filename    r   as src                  dest write src read

User · Answer

Try this   import os import sys  for root  subdirs  files in os walk path        for file in os listdir root            filePath   os path join root  file           if os path isdir filePath               pass          else              f   open  filePath   r                 Do Stuff

User · Answer

If you prefer an  almost  Oneliner  from pathlib import Path  lookuppath        use your path filelist    str item  for item in Path lookuppath  glob  quot      quot   if Path item  is file     In this case you will get a list with just the paths of all files located recursively under lookuppath  Without str   you will get PosixPath   added to each path

User · Answer

Make sure you understand the three return values of os walk   for root  subdirs  files in os walk rootdir     has the following meaning    root  Current path which is  walked through  subdirs  Files in root of type directory files  Files in root  not in subdirs  of type other than directory   And please use os path join instead of concatenating with a slash  Your problem is filePath   rootdir         file - you must concatenate the currently  walked  folder instead of the topmost folder  So that must be filePath   os path join root  file   BTW  file  is a builtin  so you don t normally use it as variable name   Another problem are your loops  which should be like this  for example   import os import sys  walk dir   sys argv 1   print  walk dir       walk dir     If your current working directory may change during script execution  it s recommended to   immediately convert program arguments to an absolute path  Then the variable root below will   be an absolute path as well  Example    walk dir   os path abspath walk dir  print  walk dir  absolute        os path abspath walk dir    for root  subdirs  files in os walk walk dir       print  -- nroot       root      list file path   os path join root   my-directory-list txt       print  list file path       list file path       with open list file path   wb   as list file          for subdir in subdirs              print   t- subdirectory     subdir           for filename in files              file path   os path join root  filename               print   t- file  s  full path   s      filename  file path                with open file path   rb   as f                  f content   f read                   list file write   The file  s contains  n    filename  encode  utf-8                    list file write f content                  list file write b  n     If you didn t know  the with statement for files is a shorthand   with open  filename    rb   as f      dosomething      is effectively the same as  f   open  filename    rb   try      dosomething   finally      f close

User · Answer

If you want a flat list of all paths under a given dir  like find   in the shell       files             os path join parent  name         for  parent  subdirs  files  in os walk YOUR DIRECTORY         for name in files   subdirs        To only include full paths to files under the base dir  leave out   subdirs

User · Answer

This worked for me  import glob  root dir    quot C   Users  Scott   quot    Don t forget trailing  last  slashes     for filename in glob iglob root dir         jpg   recursive True        print filename         do stuff

User · Answer

I ve found the following to be the easiest  from glob import glob import os  files    f for f in glob  rootdir      recursive True  if os path isfile f     Using glob  some path      recursive True  gets all files  but also includes directory names  Adding the if os path isfile f  condition filters this list to existing files only

User · Answer

I think the problem is that you re not processing the output of os walk correctly   Firstly  change   filePath   rootdir         file   to   filePath   root         file   rootdir is your fixed starting directory  root is a directory returned by os walk   Secondly  you don t need to indent your file processing loop  as it makes no sense to run this for each subdirectory   You ll get root set to each subdirectory   You don t need to process the subdirectories by hand unless you want to do something with the directories themselves

User · Answer

use os path join   to construct your paths - It s neater   import os import sys rootdir   sys argv 1  for root  subFolders  files in os walk rootdir       for folder in subFolders          outfileName   os path join root folder  py-outfile txt           folderOut   open  outfileName   w            print  outfileName is     outfileName         for file in files              filePath   os path join root file              toWrite   open  filePath  read               print  Writing      toWrite      to    filePath             folderOut write  toWrite           folderOut close

User · Answer

import glob import os  root dir    lt root dir here gt   for filename in glob iglob root dir            recursive True       if os path isfile filename           with open filename  r   as file              print file read            is used to get all files recursively including directory   if os path isfile filename  is used to check if filename variable is file or directory  if it is file then we can read that file  Here I am printing file

User · Answer

If you are using Python 3 5 or above  you can get this done in 1 line  import glob    root dir needs a trailing slash  i e   root dir   for filename in glob iglob root dir         txt   recursive True        print filename   As mentioned in the documentation  If recursive is true  the pattern      will match any files and zero or more directories and subdirectories   If you want every file  you can use import glob  for filename in glob iglob root dir            recursive True        print filename

User · Answer

TL DR  This is the equivalent to find -type f to go over all files in all folders below and including the current one   for currentpath  folders  files in os walk           for file in files          print os path join currentpath  file       As already mentioned in other answers  os walk   is the answer  but it could be explained better  It s quite simple  Let s walk through this tree   docs   -- doc1 odt pics  todo txt   With this code   for currentpath  folders  files in os walk           print currentpath    The currentpath is the current folder it is looking at  This will output       docs   pics   So it loops three times  because there are three folders  the current one  docs  and pics  In every loop  it fills the variables folders and files with all folders and files  Let s show them   for currentpath  folders  files in os walk           print currentpath  folders  files    This shows us     currentpath  folders           files                  pics    docs      todo txt     pics                                docs                             doc1 odt     So in the first line  we see that we are in folder    that it contains two folders namely pics and docs  and that there is one file  namely todo txt  You don t have to do anything to recurse into those folders  because as you see  it recurses automatically and just gives you the files in any subfolders  And any subfolders of that  though we don t have those in the example    If you just want to loop through all files  the equivalent of find -type f  you can do this   for currentpath  folders  files in os walk           for file in files          print os path join currentpath  file     This outputs     todo txt   docs doc1 odt

User · Answer

The pathlib library is really great for working with files   You can do a recursive glob on a Path object like so   from pathlib import Path  for elem in Path   path to my files   rglob             print elem

User · Answer

os walk does recursive walk by default  For each dir  starting from root it yields a 3-tuple  dirpath  dirnames  filenames   from os import walk from os path import splitext  join  def select files root  files               simple logic here to filter out interesting files      py files in this example              selected files           for file in files           do concatenation here to get full path          full path   join root  file          ext   splitext file  1           if ext      py               selected files append full path       return selected files  def build recursive dir tree path               path    -    where to begin folder scan             selected files           for root  dirs  files in walk path           selected files    select files root  files       return selected files

[python] Python recursive folder read

Examples related to python

Examples related to scripting

Examples related to file-io