How to use glob to find files recursively

Question

This is what I have   glob os path join  src     c      but I want to search the subfolders of src  Something like this would work   glob os path join  src     c    glob os path join  src         c    glob os path join  src             c    glob os path join  src                 c      But this is obviously limited and clunky

User · Answer

I needed a solution for python 2 x that works fast on large directories  I endet up with this   import subprocess foundfiles  subprocess check output  ls src   c src      c   shell True  for foundfile in foundfiles splitlines        print foundfile   Note that you might need some exception handling in case ls doesn t find any matching file

User · Answer

This is a working code on Python 2 7  As part of my devops work  I was required to write a script which would move the config files marked with live-appName properties to appName properties  There could be other extension files as well like live-appName xml   Below is a working code for this  which finds the files in the given directories  nested level  and then renames  moves  it to the required filename  def flipProperties searchDir      print  Flipping properties to point to live DB     for root  dirnames  filenames in os walk searchDir         for filename in fnmatch filter filenames   live-               targetFileName   os path join root  filename split  live-   1           print  File    os path join root  filename     will be moved to     targetFileName         shutil move os path join root  filename   targetFileName    This function is called from a main script  flipProperties searchDir    Hope this helps someone struggling with similar issues

User · Answer

pathlib Path rglob Use pathlib Path rglob from the the pathlib module  which was introduced in Python 3 5  from pathlib import Path  for path in Path  src   rglob    c        print path name   If you don t want to use pathlib  use can use glob glob       c    but don t forget to pass in the recursive keyword parameter and it will use inordinate amount of time on large directories  For cases where matching files beginning with a dot      like files in the current directory or hidden files on Unix based system  use the os walk solution below  os walk For older Python versions  use os walk to recursively walk a directory and fnmatch filter to match against a simple expression  import fnmatch import os  matches      for root  dirnames  filenames in os walk  src        for filename in fnmatch filter filenames     c            matches append os path join root  filename

User · Answer

based on other answers this is my current working implementation  which retrieves nested xml files in a root directory   files      for root  dirnames  filenames in os walk myDir       files extend glob glob root       xml      I m really having fun with python

User · Answer

For python  gt   3 5 you can use     recursive True   import glob for x in glob glob   path      c   recursive True       print x   Demo   If recursive is True  the pattern    will match any files and zero or more directories and subdirectories  If the pattern is followed by an os sep  only directories and subdirectories match    Note  Python3 6 seems to default to recursive True when using    so it can be omitted  Demo Python 3 6

User · Answer

import os import fnmatch   def recursive glob treeroot  pattern       results          for base  dirs  files in os walk treeroot           goodfiles   fnmatch filter files  pattern          results extend os path join base  f  for f in goodfiles      return results   fnmatch gives you exactly the same patterns as glob  so this is really an excellent replacement for glob glob with very close semantics   An iterative version  e g  a generator   IOW a replacement for glob iglob  is a trivial adaptation  just yield the intermediate results as you go  instead of extending a single results list to return at the end

User · Answer

import sys  os  glob  dir list     c   books  heap    while len dir list   gt  0      cur dir   dir list 0      del dir list 0      list of files   glob glob cur dir            for book in list of files          if os path isfile book               print book          else              dir list append book

User · Answer

Just made this   it will print files and directory in hierarchical way  But I didn t used fnmatch or walk     usr bin python  import os glob sys  def dirlist path  c   1            for i in glob glob os path join path                         if os path isfile i                           filepath  filename   os path split i                          print  ----   c   filename                  elif os path isdir i                           dirname   os path basename i                          print  ----   c   dirname                         c  1                         dirlist i c                          c- 1   path   os path normpath sys argv 1   print os path basename path   dirlist path

User · Answer

If the files are on a remote file system or inside an archive  you can use an implementation of the fsspec AbstractFileSystem class   For example  to list all the files in a zipfile  from fsspec implementations zip import ZipFileSystem fs   ZipFileSystem  quot  tmp test zip quot   fs glob  quot     quot      equivalent  fs find  quot   quot    or to list all the files in a publicly available S3 bucket  from s3fs import S3FileSystem fs s3   S3FileSystem anon True  fs s3 glob  quot noaa-goes16 ABI-L1b-RadF 2020 045    quot      or use fs s3 find  you can also use it for a local filesystem  which may be interesting if your implementation should be filesystem-agnostic  from fsspec implementations local import LocalFileSystem fs   LocalFileSystem   fs glob  quot  tmp test    quot    Other implementations include Google Cloud  Github  SFTP SSH  Dropbox  and Azure   For details  see the fsspec API documentation

User · Answer

Here is a solution that will match the pattern against the full path and not just the base filename   It uses fnmatch translate to convert a glob-style pattern into a regular expression  which is then matched against the full path of each file found while walking the directory   re IGNORECASE is optional  but desirable on Windows since the file system itself is not case-sensitive   I didn t bother compiling the regex because docs indicate it should be cached internally    import fnmatch import os import re  def findfiles dir  pattern       patternregex   fnmatch translate pattern      for root  dirs  files in os walk dir           for basename in files              filename   os path join root  basename              if re search patternregex  filename  re IGNORECASE                   yield filename

User · Answer

import os  glob  for each in glob glob  path      c   recursive True       print f Name with path   each   nName without path   os path basename each       glob glob    c                       matches all files ending in  c in current directory glob glob      c                     same as 1 glob glob       c                    matches all files ending in  c in the immediate subdirectories only  but not in the current directory glob glob    c  recursive True          same as 1 glob glob      c  recursive True      same as 3 glob glob       c  recursive True    matches all files ending in  c in the current directory and in all subdirectories

User · Answer

In addition to the suggested answers  you can do this with some lazy generation and list comprehension magic   import os  glob  itertools  results   itertools chain from iterable glob iglob os path join root    c                                                   for root  dirs  files in os walk  src     for f in results  print f    Besides fitting in one line and avoiding unnecessary lists in memory  this also has the nice side effect  that you can use it in a way similar to the    operator  e g   you could use os path join root   some path   c   in order to get all  c files in all sub directories of src that have this structure

User · Answer

Recently I had to recover my pictures with the extension  jpg  I ran photorec and recovered 4579 directories 2 2 million files within  having tremendous variety of extensions With the script below I was able to select 50133 files havin  jpg extension within minutes      usr binenv python2 7  import glob import shutil import os  src dir     home mustafa Masa  st   yedek  dst dir     home mustafa Genel media  for mediafile in glob iglob os path join src dir          jpg          is for subdirectory     shutil copy mediafile  dst dir

User · Answer

That one uses fnmatch or regular expression   import fnmatch  os  def filepaths directory  pattern       for root  dirs  files in os walk directory           for basename in files              try                  matched   pattern match basename              except AttributeError                  matched   fnmatch fnmatch basename  pattern              if matched                  yield os path join root  basename     usage if   name         main         from pprint import pprint as pp     import re     path   r  Users hipertracker app myapp      pp  x for x in filepaths path  re compile r     py           pp  x for x in filepaths path     py

User · Answer

Consider pathlib rglob        This is like calling Path glob   with       added in front of the given relative pattern    import pathlib   for p in pathlib Path  src   rglob    c        print p    See also  taleinat s related post here and a similar post elsewhere

User · Answer

Starting with Python 3 4  one can use the glob   method of one of the Path classes in the new pathlib module  which supports    wildcards  For example   from pathlib import Path  for file path in Path  src   glob       c        print file path    do whatever you need with these files   Update  Starting with Python 3 5  the same syntax is also supported by glob glob

User · Answer

I ve modified the glob module to support    for recursive globbing  e g    gt  gt  gt  import glob2  gt  gt  gt  all header files   glob2 glob  src      c     https   github com miracle2k python-glob2   Useful when you want to provide your users with the ability to use the    syntax  and thus  os walk   alone is not good enough

User · Answer

For python 3 5 and later  import glob   file names array   glob glob  path   c   recursive True   above works for files directly at path  as guided by NeStack   updated version file names array   glob glob  path      c   recursive True    further you might need  for full path in src in  file names array      print  full path in src     be like  abc xyz c       Full system path of this would be like   gt   path till src abc xyz c

User · Answer

Or with a list comprehension     gt  gt  gt  base   r c  User xtofl    gt  gt  gt  binfiles     os path join base f               for base     files in os walk root               for f in files if f endswith   jpg

User · Answer

Here is my solution using list comprehension to search for multiple file extensions recursively in a directory and all subdirectories   import os  glob  def  globrec path   exts       Glob recursively a directory and all subdirectories for multiple file extensions      Note  Glob is case-insensitive  i  e  for     jpg  you will get files ending     with  jpg and  JPG      Parameters     ----------     path   str         A directory name     exts   tuple         File extensions to glob for      Returns     -------     files   list         list of files matching extensions in exts in path and subfolders              dirs    a 0  for a in os walk path       f filter    d e for d in dirs for e in exts          return  f for files in  glob iglob files  for files in f filter  for f in files   my pictures    globrec r C  Temp       jpg      bmp      png      gif   for f in my pictures      print f

User · Answer

Simplified version of Johan Dahlin s answer  without fnmatch   import os  matches      for root  dirnames  filenames in os walk  src      matches     os path join root  f  for f in filenames if f -2        c

User · Answer

Another way to do it using just the glob module  Just seed the rglob method with a starting base directory and a pattern to match and it will return a list of matching file names   import glob import os  def  getDirs base       return  x for x in glob iglob os path join  base        if os path isdir x     def rglob base  pattern       list          list extend glob glob os path join base pattern        dirs    getDirs base      if len dirs           for d in dirs              list extend rglob os path join base d   pattern       return list

User · Answer

Johan and Bruno provide excellent solutions on the minimal requirement as stated  I have just released Formic which implements Ant FileSet and Globs which can handle this and more complicated scenarios  An implementation of your requirement is   import formic fileset   formic FileSet include   src      c   for file name in fileset qualified files        print file name

User · Answer

In case this may interest anyone  I ve profiled the top three proposed methods  I have about  500K files in the globbed folder  in total   and 2K files that match the desired pattern  here s the  very basic  code import glob import json import fnmatch import os from pathlib import Path from time import time   def find files iglob        return glob iglob  quot   data    data json quot   recursive True    def find files oswalk        for root  dirnames  filenames in os walk  data            for filename in fnmatch filter filenames   data json                yield os path join root  filename   def find files rglob        return Path  data   rglob  data json    t0   time   for f in find files oswalk    pass     t1   time   for f in find files rglob    pass t2   time   for f in find files iglob    pass  t3   time   print t1-t0  t2-t1  t3-t2   And the results I got were  os walk   3 6sec rglob  14 5sec iglob   16 9sec The platform  Ubuntu 16 04  x86 64  core i7

User · Answer

Similar to other solutions  but using fnmatch fnmatch instead of glob  since os walk already listed the filenames   import os  fnmatch   def find files directory  pattern       for root  dirs  files in os walk directory           for basename in files              if fnmatch fnmatch basename  pattern                   filename   os path join root  basename                  yield filename   for filename in find files  src      c        print  Found C source    filename   Also  using a generator alows you to process each file as it is found  instead of finding all the files and then processing them

User · Answer

You ll want to use os walk to collect filenames that match your criteria  For example   import os cfiles      for root  dirs  files in os walk  src      for file in files      if file endswith   c          cfiles append os path join root  file

User · Answer

Here s a solution with nested list comprehensions  os walk and simple suffix matching instead of glob   import os cfiles    os path join root  filename            for root  dirnames  filenames in os walk  src             for filename in filenames if filename endswith   c      It can be compressed to a one-liner   import os cfiles  os path join r f  for r d fs in os walk  src   for f in fs if f endswith   c      or generalized as a function   import os  def recursive glob rootdir      suffix          return  os path join looproot  filename              for looproot     filenames in os walk rootdir              for filename in filenames if filename endswith suffix    cfiles   recursive glob  src     c     If you do need full glob style patterns  you can follow Alex s and Bruno s example and use fnmatch   import fnmatch import os  def recursive glob rootdir      pattern           return  os path join looproot  filename              for looproot     filenames in os walk rootdir              for filename in filenames             if fnmatch fnmatch filename  pattern    cfiles   recursive glob  src      c

User · Answer

I modified the top answer in this posting   and recently created this script which will loop through all files in a given directory  searchdir  and the sub-directories under it    and prints filename  rootdir  modified creation date  and size   Hope this helps someone    and they can walk the directory and get fileinfo   import time import fnmatch import os  def fileinfo file       filename   os path basename file      rootdir   os path dirname file      lastmod   time ctime os path getmtime file       creation   time ctime os path getctime file       filesize   os path getsize file       print   s   t s t s t s t s     rootdir  filename  lastmod  creation  filesize   searchdir   r D  Your Directory Root  matches       for root  dirnames  filenames in os walk searchdir           for filename in fnmatch filter filenames     c        for filename in filenames                  matches append os path join root  filename             print matches         fileinfo os path join root  filename

[python] How to use glob() to find files recursively?

Examples related to python

Examples related to path

Examples related to filesystems

Examples related to glob

Examples related to fnmatch