Python glob multiple filetypes

Question

Is there a better way to use glob glob in python to get a list of multiple file types such as  txt   mdown  and  markdown   Right now I have something like this   projectFiles1   glob glob  os path join projectDir     txt     projectFiles2   glob glob  os path join projectDir     mdown     projectFiles3   glob glob  os path join projectDir     markdown

User · Answer

After coming here for help  I made my own solution and wanted to share it  It s based on user2363986 s answer  but I think this is more scalable  Meaning  that if you have 1000 extensions  the code will still look somewhat elegant   from glob import glob  directoryPath     C   temp       fileExtensions      jpg    jpeg    png    bmp    gif    listOfFiles          for extension in fileExtensions      listOfFiles extend  glob  directoryPath   extension     for file in listOfFiles      print file      Or do other stuff

User · Answer

with glob it is not possible  you can use only    matches everything   matches any single character  seq  matches any character in seq   seq  matches any character not in seq    use os listdir and a regexp to check patterns   for x in os listdir         if re match      txt     sql   x       print x

User · Answer

Python 3 We can use pathlib   glob still doesn t support globbing multiple arguments or within braces  as in POSIX shells  but we can easily filter the result  For example  where you might ideally like to do    NOT VALID Path config dir  glob  quot    ini toml  quot     NOR IS Path config dir  glob  quot   ini quot    quot   toml quot    you can do  filter lambda p  p suffix in   quot  ini quot    quot  toml quot    Path config dir  glob  quot   quot     which isn t too much worse

User · Answer

This worked for me  split      -1   Will separate the filename suffix    xxx  so using if help to check it     for filename in glob glob folder                   print folder filename          if  filename split      -1      tif  and               filename split      -1      tiff  and               filename split      -1      bmp  and               filename split      -1      jpg  and               filename split      -1      jpeg  and               filename split      -1      png                   continue           Your code

User · Answer

Not glob  but here s another way using a list comprehension   extensions    txt mdown markdown  split   projectFiles    f for f in os listdir projectDir                     if os path splitext f  1  1   in extensions

User · Answer

Yet another solution  use glob to get paths using multiple match patterns and combine all paths into a single list using reduce and add    import functools  glob  operator paths   functools reduce operator add   glob glob pattern  for pattern in        path1   ext1        path2   ext2

User · Answer

Use a list of extension and iterate through from os path import join from glob import glob  files      extensions       gif      png      jpg   for ext in extensions     files extend glob join  quot path to dir quot   ext     print files

User · Answer

For example  for   mp3 and   flac on multiple folders  you can do  mask   r music      mf  pl  3a    glob glob mask   The idea can be extended to more file extensions  but you have to check that the combinations won t match any other unwanted file extension you may have on those folders  So  be careful with this  To automatically combine an arbitrary list of extensions into a single glob pattern  you can do the following  def multi extension glob mask mask base   extensions       mask ext           format    join set c    for c in zip  extensions       if not mask ext or len set len e  for e in extensions    gt  1          mask ext append          return mask base      join mask ext   mask   multi extension glob mask  music         mp3    flac    wma   print mask     music      mfw  pml  a3

User · Answer

The following function  glob globs for multiple file extensions   import glob import os def  glob path   exts          Glob for multiple file extensions      Parameters     ----------     path   str         A file name without extension  or directory name     exts   tuple         File extensions to glob for      Returns     -------     files   list         list of files matching extensions in exts in path              path   os path join path       if os path isdir path  else path           return  f for files in  glob glob path   ext  for ext in exts  for f in files   files    glob projectDir    txt     mdown     markdown

User · Answer

A one-liner  Just for the hell of it    folder    C   multi pattern glob one liner  files    item for sublist in  glob glob folder   ext  for ext in      txt       bat    for item in sublist    output     C   multi pattern glob one liner  dummy txt txt    C   multi pattern glob one liner  dummy bat bat

User · Answer

From previous answer glob    jpg     glob    png    Here is a shorter one  from glob import glob extensions     jpg    png     to find these filename extensions    Method 1  loop one by one and extend to the output list output       output extend glob f    name     for name in extensions  print output     Method 2  even shorter   loop filename extension to glob   it and flatten it to a list output    p for p2 in  glob f    name    for name in extensions  for p in p2  print output

User · Answer

glob returns a list  why not just run it multiple times and concatenate the results  from glob import glob project files   glob    txt     glob    mdown     glob    markdown

User · Answer

Maybe there is a better way  but how about  import glob types       pdf      cpp     the tuple of file types files grabbed      for files in types      files grabbed extend glob glob files      files grabbed is the list of pdf and cpp files  Perhaps there is another way  so wait in case someone else comes up with a better answer

User · Answer

You can try to make a manual list comparing the extension of existing with those you require   ext list     gif   jpg   jpeg   png    file list      for file in glob glob           if file rsplit     1  1  in ext list       file list append file

User · Answer

files   glob glob    txt   files extend glob glob    dat

User · Answer

For example   import glob lst img      base dir     home xy img      get all the jpg file in base dir  lst img    glob glob base dir      jpg   print lst img      home xy img 2 jpg     home xy img 1 jpg      append all the png file in base dir to lst img lst img    glob glob base dir      png   print lst img      home xy img 2 jpg     home xy img 1 jpg     home xy img 3 png     A function   import glob def get files base dir   home xy img    lst extension     jpg      png                  param base dir base directory      param lst extension lst extension  list like     jpg      png             return file lists like    home xy img 2 jpg    home xy img 3 png               lst files          for ext in lst extension          lst files    glob glob base dir ext      return lst files

User · Answer

By the results I ve obtained from empirical tests  it turned out that glob glob isn t the better way to filter out files by their extensions  Some of the reason are    The globbing  language  does not allows perfect specification of multiple extension  The former point results in obtaining incorrect results depending on file extensions  The globbing method is empirically proven to be slower than most other methods  Even if it s strange even other filesystems objects can have  extensions   folders too    I ve tested  for correcteness and efficiency in time  the following 4 different methods to filter out files by extensions and puts them in a list   from glob import glob  iglob from re import compile  findall from os import walk   def glob with storage args        elements      join  f   i    for i in args extensions       globs   f  args target       elements       results   glob globs  recursive True       return results   def glob with iteration args        elements      join  f   i    for i in args extensions       globs   f  args target       elements       results    i for i in iglob globs  recursive True        return results   def walk with suffixes args        results          for r  d  f in walk args target           for ff in f              for e in args extensions                  if ff endswith e                       results append path join r ff                       break     return results   def walk with regs args        reg   compile     join  f  i    for i in args extensions         results          for r  d  f in walk args target           for ff in f              if len findall reg ff                    results append path join r  ff        return results   By running the code above on my laptop I obtained the following auto-explicative results   Elapsed time for  7 times glob with storage      0 365023 seconds  mean     0 05214614 median   0 051861 stdev    0 001492152 min      0 050864 max      0 054853  Elapsed time for  7 times glob with iteration      0 360037 seconds  mean     0 05143386 median   0 050864 stdev    0 0007847381 min      0 050864 max      0 052859  Elapsed time for  7 times walk with suffixes      0 26529 seconds  mean     0 03789857 median   0 037899 stdev    0 0005759071 min      0 036901 max      0 038896  Elapsed time for  7 times walk with regs      0 290223 seconds  mean     0 04146043 median   0 040891 stdev    0 0007846776 min      0 04089 max      0 042885  Results sizes  0 2451 1 2451 2 2446 3 2446  Differences between glob   and walk    0 E  x y z venv lib python3 7 site-packages Cython Includes numpy 1 E  x y z venv lib python3 7 site-packages Cython Utility CppSupport cpp 2 E  x y z venv lib python3 7 site-packages future moves xmlrpc 3 E  x y z venv lib python3 7 site-packages Cython Includes libcpp 4 E  x y z venv lib python3 7 site-packages future backports xmlrpc  Elapsed time for  main    1 317424 seconds    The fastest way to filter out files by extensions  happens even to be the ugliest one  Which is  nested for loops and string comparison using the endswith   method    Moreover  as you can see  the globbing algorithms  with the pattern E  x y z      py  pyc   even with only 2 extension given  py and pyc  returns also incorrect results

User · Answer

To glob multiple file types  you need to call glob   function several times in a loop  Since this function returns a list  you need to concatenate the lists   For instance  this function do the job   import glob import os   def glob filetypes root dir   patterns       return  path             for pattern in patterns             for path in glob glob os path join root dir  pattern      Simple usage   project dir    path to project dir  for path in sorted glob filetypes project dir     txt      mdown      markdown         print path    You can also use glob iglob   to have an iterator      Return an iterator which yields the same values as glob   without actually storing them all simultaneously    def iglob filetypes root dir   patterns       return  path             for pattern in patterns             for path in glob iglob os path join root dir  pattern

User · Answer

You could use filter   import os import glob  projectFiles   filter      lambda x  os path splitext x  1  in    txt     mdown     markdown       glob glob os path join projectDir

User · Answer

This is a Python 3 4  pathlib solution   exts     pdf     doc     xls     csv     ppt  filelist    str i  for i in map pathlib Path  os listdir src   if i suffix lower   in exts and not i stem startswith         Also it ignores all file names starting with

User · Answer

Chain the results   import itertools as it  glob  def multiple file types  patterns       return it chain from iterable glob iglob pattern  for pattern in patterns    Then   for filename in multiple file types    txt      sql      log          do stuff

User · Answer

You could also use reduce   like so   import glob file types       txt      mdown      markdown   project files   reduce lambda list1  list2  list1   list2   glob glob t  for t in file types     this creates a list from glob glob   for each pattern and reduces them to a single list

User · Answer

One glob  many extensions    but imperfect solution  might match other files    filetypes     tif    jpg    filetypes   zip   list ft  for ft in filetypes   filetypes       join ch  for ch in filetypes  filetypes       s     ch for ch in filetypes  filetypes      join filetypes        print filetypes      gt   tj  ip  fg    glob glob   path to    s    filetypes

User · Answer

So many answers that suggest globbing as many times as number of extensions  I d prefer globbing just once instead   from pathlib import Path  files    p resolve   for p in Path path  glob         if p suffix in    c     cc     cpp     hxx     h

User · Answer

Here is one-line list-comprehension variant of Pat s answer  which also includes that you wanted to glob in a specific project directory    import os  glob exts       txt      mdown      markdown   files    f for ext in exts for f in glob glob os path join project dir  ext      You loop over the extensions  for ext in exts   and then for each extension you take each file matching the glob pattern  for f in glob glob os path join project dir  ext     This solution is short  and without any unnecessary for-loops  nested list-comprehensions  or functions to clutter the code  Just pure  expressive  pythonic Zen    This solution allows you to have a custom list of exts that can be changed without having to update your code   This is always a good practice    The list-comprehension is the same used in Laurent s solution  which I ve voted for   But I would argue that it is usually unnecessary to factor out a single line to a separate function  which is why I m providing this as an alternative solution   Bonus    If you need to search not just a single directory  but also all sub-directories  you can pass recursive True and use the multi-directory glob symbol    1   files    f for ext in exts           for f in glob glob os path join project dir        ext   recursive True     This will invoke glob glob   lt project dir gt       txt   recursive True  and so on for each extension   1 Technically  the    glob symbol simply matches one or more characters including forward-slash    unlike the singular   glob symbol   In practice  you just need to remember that as long as you surround    with forward slashes  path separators   it matches zero or more directories

User · Answer

from glob import glob  files   glob    gif   files extend glob    png    files extend glob    jpg     print files    If you need to specify a path  loop over match patterns and keep the join inside the loop for simplicity   from os path import join from glob import glob  files      for ext in     gif      png      jpg       files extend glob join  path to dir   ext     print files

User · Answer

this worked for me   import glob images   glob glob    JPG  or    jpg  or    png

User · Answer

While Python s default glob doesn t really follow after Bash s glob  you can do this with other libraries  We can enable braces in wcmatch s glob    gt  gt  gt  from wcmatch import glob  gt  gt  gt  glob glob     md ini    flags glob BRACE    LICENSE md    README md    tox ini     You can even use extended glob patterns if that is your preference   from wcmatch import glob  gt  gt  gt  glob glob      md ini    flags glob EXTGLOB    LICENSE md    README md    tox ini

User · Answer

This Should Work   import glob extensions       txt      mdown      markdown   for i in extensions      for files in glob glob i           print  files

User · Answer

If you use pathlib try this   import pathlib  extensions      py     txt   root dir      test    files   filter lambda p  p suffix in extensions  pathlib Path root dir  glob           print list files

User · Answer

import os     import glob import operator from functools import reduce  types       jpg      png      jpeg   lazy paths    glob glob os path join  my path   t   for t in types  paths   reduce operator add  lazy paths        https   docs python org 3 5 library functools html functools reduce https   docs python org 3 5 library operator html operator add

User · Answer

I have released Formic which implements multiple includes in a similar way to Apache Ant s FileSet and Globs   The search can be implemented   import formic patterns       txt      markdown      mdown   fileset   formic FileSet directory projectDir  include patterns  for file name in fileset qualified files          Do something with file name   Because the full Ant glob is implemented  you can include different directories with each pattern  so you could choose only those  txt files in one subdirectory  and the  markdown in another  for example   patterns       unformatted      txt     formatted      mdown      I hope this helps

User · Answer

import glob import pandas as pd  df1   pd DataFrame columns   A    for i in glob glob  C  dir path   txt        df1   df1 append   A   i   ignore index True  for i in glob glob  C  dir path   mdown        df1   df1 append   A   i   ignore index True  for i in glob glob  C  dir path   markdown       df1   df1 append   A   i   ignore index True

User · Answer

I had the same issue and this is what I came up with   import os  sys  re   without glob  src dir     mnt mypics   src pics      ext   re compile               format     join   png    jpeg    jpg    encode  utf-8     for root  dirnames  filenames in os walk src dir     for filename in filter lambda name ext search name  filenames       src pics append os path join root  filename

[python] Python glob multiple filetypes

Examples related to python

Examples related to glob