Shell command to tar directory excluding certain files folders

Question

Is there a simple shell command script that supports excluding certain files folders from being archived   I have a directory that need to be archived with a sub directory that has a number of very large files I do not need to backup   Not quite solutions   The tar --exclude PATTERN command matches the given pattern and excludes those files  but I need specific files  amp  folders to be ignored  full file path   otherwise valid files might be excluded   I could also use the find command to create a list of files and exclude the ones I don t want to archive and pass the list to tar  but that only works with for a small amount of files  I have tens of thousands   I m beginning to think the only solution is to create a file with a list of files folders to be excluded  then use rsync with --exclude-from file to copy all the files to a tmp directory  and then use tar to archive that directory   Can anybody think of a better more efficient solution   EDIT  Charles Ma s solution works well  The big gotcha is that the --exclude    folder  MUST be at the beginning of the tar command  Full command  cd first  so backup is relative to that directory    cd  folder to backup tar --exclude    folder  --exclude    upload folder2  -zcvf  backup filename tgz

User · Answer

I had no luck getting tar to exclude a 5 Gigabyte subdirectory a few levels deep. In the end, I just used the unix Zip command. It worked a lot easier for me.

So for this particular example from the original post
(tar --exclude='./folder' --exclude='./upload/folder2' -zcvf /backup/filename.tgz . )

The equivalent would be:

zip -r /backup/filename.zip . -x upload/folder/**\* upload/folder2/**\*

(NOTE: Here is the post I originally used that helped me https://superuser.com/questions/312301/unix-zip-directory-but-excluded-specific-subdirectories-and-everything-within-t)

User · Answer

After reading this thread  I did a little testing on RHEL 5 and here are my results for tarring up the abc directory   This will exclude the directories error and logs and all files under the directories   tar cvpzf abc tgz abc  --exclude  abc error  --exclude  abc logs    Adding a wildcard after the excluded directory will exclude the files but preserve the directories   tar cvpzf abc tgz abc  --exclude  abc error    --exclude  abc logs

User · Answer

You can use standard  ant notation  to exclude directories relative  This works for me and excludes any  git or node module directories   tar -cvf myFile tar --exclude     git   --exclude    node modules    -T  data txt myInputFile txt 2 gt   data txt myTarLogFile txt   myInputFile txt contains   dev2 java  dev2 javascript

User · Answer

Your best bet is to use find with tar  via xargs  to handle the large number of arguments    For example   find   -print0   xargs -0 tar cjf tarfile tar bz2

User · Answer

To avoid possible  xargs  Argument list too long  errors due to the use of find       xargs     when processing tens of thousands of files  you can pipe the output of find directly to tar using find     -print0   tar --null         archive a given directory  but exclude various files  amp  directories    specified by their full file paths find    pwd -P   -type d    -path   path to dir1  -or -path   path to dir2     -prune      -or -not    -path   path to file1  -or -path   path to file2     -print0       gnutar --null --no-recursion -czf archive tar gz --files-from -     bsdtar --null -n -czf archive tar gz -T -

User · Answer

You can exclude directories with --exclude for tar   If you want to archive everything except  usr you can use   tar -zcvf  all tgz   --exclude  usr   In your case perhaps something like  tar -zcvf archive tgz arc dir --exclude dir ignore this dir

User · Answer

If you are trying to exclude Version Control System  VCS  files  tar already supports two interesting options about it       Option   --exclude-vcs   This option excludes files and directories used by following version control systems  CVS  RCS  SCCS  SVN  Arch  Bazaar  Mercurial  and Darcs   As of version 1 32  the following files are excluded    CVS   and everything under it RCS   and everything under it SCCS   and everything under it  git   and everything under it  gitignore  gitmodules  gitattributes  cvsignore  svn   and everything under it  arch-ids   and everything under it  arch    and everything under it  RELEASE-ID  meta-update  update  bzr  bzrignore  bzrtags  hg  hgignore  hgrags  darcs    Option   --exclude-vcs-ignores    When archiving directories that are under some version control system  VCS   it is often convenient to read exclusion patterns from this VCS  ignore files  e g   cvsignore   gitignore  etc   This option provide such possibility   Before archiving a directory  see if it contains any of the following files  cvsignore   gitignore   bzrignore  or  hgignore  If so  read ignore patterns from these files   The patterns are treated much as the corresponding VCS would treat them  i e     cvsignore  Contains shell-style globbing patterns that apply only to the directory where this file resides  No comments are allowed in the file  Empty lines are ignored    gitignore  Contains shell-style globbing patterns  Applies to the directory where  gitfile is located and all its subdirectories   Any line beginning with a   is a comment  Backslash escapes the comment character    bzrignore  Contains shell globbing-patterns and regular expressions  if prefixed with RE  16   Patterns affect the directory and all its subdirectories   Any line beginning with a   is a comment    hgignore  Contains posix regular expressions 17   The line syntax  glob switches to shell globbing patterns  The line syntax  regexp switches back  Comments begin with a    Patterns affect the directory and all its subdirectories     Example   tar -czv --exclude-vcs --exclude-vcs-ignores -f path to my-tar-file tar gz path to my project

User · Answer

I d like to show another option I used to get the same result as the answers before provide  I had a similar case where I wanted to backup android studio projects all together in a tar file to upload to media fire  using the du command to find the large files  I found that I didn t need some directories like  build  linux e  dart tools Using the first answer of Charles ma I modified it a little bit to be able to run the command from the parent directory of the my Android directory  tar --exclude    build  --exclude    linux  --exclude     dart tool  -zcvf androidProjects tar Android   It worked like a charm  Ps  Sorry if this kind of answer is not allowed  if this is the case I will remove

User · Answer

You can also use one of the  --exclude-tag  options depending on your needs    --exclude-tag FILE --exclude-tag-all FILE --exclude-tag-under FILE   The folder hosting the specified FILE will be excluded

User · Answer

This exclude pattern handles filename suffix like png or mp3 as well as directory names like  git and node modules  tar --exclude    png   mp3   wav  git node modules  -Jcf   target tarball     source dirname

User · Answer

After reading all this good answers for different versions and having solved the problem for myself  I think there are very small details that are very important  and rare to GNU Linux general use  that aren t stressed enough and deserves more than comments   So I m not going to try to answer the question for every case  but instead  try to register where to look when things doesn t work   IT IS VERY IMPORTANT TO NOTICE    THE ORDER OF THE OPTIONS MATTER  it is not the same put the --exclude before than after the file option and directories to backup  This is unexpected at least to me  because in my experience  in GNU Linux commands  usually the order of the options doesn t matter  Different tar versions expects this options in different order  for instance   Andrew s answer indicates that in GNU tar v 1 26 and 1 28 the excludes comes last  whereas in my case  with GNU tar 1 29  it s the other way  THE TRAILING SLASHES MATTER  at least in GNU tar 1 29  it shouldn t be any    In my case  for GNU tar 1 29 on Debian stretch  the command that worked was  tar --exclude   home user  config chromium  --exclude   home user  cache  -cf file tar   dir1   home   dir3    The quotes didn t matter  it worked with or without them   I hope this will be useful to someone

User · Answer

gnu tar v 1 26 the --exclude needs to come after archive file and backup directory arguments  should have no leading or trailing slashes  and prefers no quotes  single or double    So relative to the PARENT directory to be backed up  it s   tar cvfz  path to mytar tgz   dir to backup --exclude some path to exclude

User · Answer

You can have multiple exclude options for tar so    tar --exclude    folder  --exclude    upload folder2  -zcvf  backup filename tgz     etc will work  Make sure to put --exclude before the source and destination items

User · Answer

The following bash script should do the trick  It uses the answer given here by Marcus Sundman      bin bash  echo -n  Please enter the name of the tar file you wish to create with out extension   read nam  echo -n  Please enter the path to the directories to tar   read pathin  echo tar -czvf  nam tar gz excludes  find  pathin -iname    CC  -exec echo  --exclude            xargs  echo  pathin  echo tar -czvf  nam tar gz  excludes  pathin   This will print out the command you need and you can just copy and paste it back in  There is probably a more elegant way to provide it directly to the command line   Just change   CC for any other common extension  file name or regex you want to exclude and this should still work   EDIT  Just to add a little explanation  find generates a list of files matching the chosen regex  in this case   CC   This list is passed via xargs to the echo command  This prints --exclude  one entry from the list   The slashes    are escape characters for the   marks

User · Answer

Success Case  1  if giving full path to take backup  in exclude also should be used full path   tar -zcvf  opt ABC BKP 27032020 backup 27032020 tar gz --exclude   opt ABC csv   --exclude   opt ABC log    opt ABC  2  if giving current path to take backup  in exclude also should be used current path only   tar -zcvf backup 27032020 tar gz --exclude  ABC csv   --exclude  ABC log   ABC  Failure Case    if giving currentpath directory to take backup and full path to ignore then wont work  tar -zcvf  opt ABC BKP 27032020 backup 27032020 tar gz --exclude   opt ABC csv   --exclude   opt ABC log   ABC   Note  mentioning exclude before after backup directory is fine

User · Answer

You can use cpio 1  to create tar files  cpio takes the files to archive on stdin  so if you ve already figured out the find command you want to use to select the files the archive  pipe it into cpio to create the tar file   find       cpio -o -H ustar   gzip -c  gt  archive tar gz

User · Answer

old question with many answers  but I found that none were quite clear enough for me  so I would like to add my try   if you have the following structure   home ftp mysite    with following file folders   home ftp mysite file1  home ftp mysite file2  home ftp mysite file3  home ftp mysite folder1  home ftp mysite folder2  home ftp mysite folder3   so  you want to make a tar file that contain everyting inside  home ftp mysite  to move the site to a new server   but file3 is just junk  and everything in folder3 is also not needed  so we will skip those two   we use the format  tar -czvf  lt name of tar file gt   lt what to tar gt   lt any excludes gt    where the c   create  z   zip  and v   verbose  you can see the files as they are entered  usefull to make sure none of the files you exclude are being added   and f  file   so  my command would look like this  cd  home ftp  tar -czvf mysite tar gz mysite --exclude  file3  --exclude  folder3    note the files folders excluded are relatively to the root of your tar  I have tried full path here relative to   but I can not make that work     hope this will help someone  and me next time I google it

User · Answer

I want to have fresh front-end version  angular folder  on localhost  Also  git folder is huge in my case  and I want to exclude it  I need to download it from server  and unpack it in order to run application   Compress angular folder from  var lib tomcat7 webapps  move it to  tmp folder with name angular 23 12 19 tar gz  Command     tar --exclude   git  -zcvf  tmp angular 23 12 19 tar gz  var lib tomcat7 webapps angular

User · Answer

I agree the --exclude flag is the right approach     tar --exclude    folder or file  --exclude  file pattern  --exclude  fileA    A word of warning for a side effect that I did not find immediately obvious  The exclusion of  fileA  in this example will search for  fileA  RECURSIVELY   Example A directory with a single subdirectory containing a file of the same name  data txt   data txt config txt -- dirA      data txt      config docx    If using --exclude  data txt  the archive will not contain EITHER data txt file  This can cause unexpected results if archiving third party libraries  such as a node modules directory  To avoid this issue make sure to give the entire path  like --exclude    dirA data txt

User · Answer

Check it out  tar cvpzf zip folder tgz   --exclude   public --exclude   tmp --exclude   log --exclude fileName

User · Answer

For those who have issues with it  some versions of tar would only work properly without the      in the exclude value   Tar --version      tar  GNU tar  1 27 1   Command syntax that work   tar -czvf    allfiles-butsome tar gz   --exclude acme foo   These will not work     tar -czvf    allfiles-butsome tar gz   --exclude   acme foo   tar -czvf    allfiles-butsome tar gz   --exclude    acme foo    tar --exclude   acme foo -czvf    allfiles-butsome tar gz     tar --exclude    acme foo  -czvf    allfiles-butsome tar gz     tar -czvf    allfiles-butsome tar gz   --exclude  full path acme foo   tar -czvf    allfiles-butsome tar gz   --exclude   full path acme foo    tar --exclude  full path acme foo -czvf    allfiles-butsome tar gz     tar --exclude   full path acme foo  -czvf    allfiles-butsome tar gz

User · Answer

Possible options to exclude files directories from backup using tar   Exclude files using multiple patterns  tar -czf backup tar gz --exclude PATTERN1 --exclude PATTERN2      path to backup   Exclude files using an exclude file filled with a list of patterns  tar -czf backup tar gz -X  path to exclude txt  path to backup   Exclude files using tags by placing a tag file in any directory that should be skipped  tar -czf backup tar gz --exclude-tag-all exclude tag  path to backup

User · Answer

I ve experienced that  at least with the Cygwin version of tar I m using   CYGWIN NT-5 1 1 7 17 0 262 5 3  2012-10-19 14 39 i686 Cygwin  on a Windows XP Home Edition SP3 machine   the order of options is important   While this construction worked for me   tar cfvz target tgz --exclude   lt dir1 gt   --exclude   lt dir2 gt   target dir   that one didn t work   tar cfvz --exclude   lt dir1 gt   --exclude   lt dir2 gt   target tgz target dir   This  while tar --help reveals the following   tar  OPTION      FILE    So  the second command should also work  but apparently it doesn t seem to be the case     Best rgds

User · Answer

Possible redundant answer but since I found it useful  here it is   While a FreeBSD root  i e  using csh  I wanted to copy my whole root filesystem to  mnt but without  usr and  obviously   mnt  This is what worked  I am at      tar --exclude   usr --exclude   mnt --create --file -    cd  mnt  amp  amp  tar xvd -    My whole point is that it was necessary  by putting the     to specify to tar that the excluded directories where part of the greater directory being copied   My    0 02

User · Answer

For Mac OSX I had to do   tar -zcv --exclude  folder  -f theOutputTarFile tar folderToTar    Note the -f after the --exclude

User · Answer

I found this somewhere else so I won t take credit  but it worked better than any of the solutions above for my mac specific issues  even though this is closed    tar zc --exclude   MACOSX --exclude  DS Store -f  lt archive gt   lt source s  gt

User · Answer

tar -cvzf destination folder source folder -X  home folder excludes txt   -X indicates a file which contains a list of filenames which must be excluded from the backup  For Instance  you can specify    in this file to not include any filenames ending with   in the backup

User · Answer

Use the find command in conjunction with the tar append  -r  option    This way you can add files to an existing tar in a single step  instead of a two pass solution  create list of files  create tar    find  dir dir -prune     -o etc etc     -exec tar rvf   tarfile tar

[linux] Shell command to tar directory excluding certain files/folders

Examples related to linux

Examples related to shell

Examples related to archive

Examples related to tar