How to grep a continuous stream

Question

Is that possible to use grep on a continuous stream   What I mean is sort of a tail -f  lt file gt  command  but with grep on the output in order to keep only the lines that interest me   I ve tried tail -f  lt file gt    grep pattern but it seems that grep can only be executed once tail finishes  that is to say never

User · Answer

I use the tail -f  lt file gt    grep  lt pattern gt  all the time   It will wait till grep flushes  not till it finishes  I m using Ubuntu

User · Answer

you certainly won t succeed with  tail -f  var log foo log  grep --line-buffered string2search   when you use  colortail  as an alias for tail  eg  in bash  alias tail  colortail -n 30    you can check by     type alias if this outputs something like     tail isan alias of colortail -n 30  then you have your culprit     Solution   remove the alias with  unalias tail   ensure that you re using the  real  tail binary by this command  type tail   which should output something like   tail is  usr bin tail   and then you can run your command  tail -f foo log  grep --line-buffered something   Good luck

User · Answer

If you want to find matches in the entire file  not just the tail   and you want it to sit and wait for any new matches  this works nicely   tail -c  0 -f  lt file gt    grep --line-buffered  lt pattern gt    The -c  0 flag says that the output should start 0 bytes  -c  from the beginning     of the file

User · Answer

sed would be a better choice  stream editor   tail -n0 -f  lt file gt    sed -n   search string p   and then if you wanted the tail command to exit once you found a particular string   tail --pid     BASHPID 1   -n0 -f  lt file gt    sed -n   search string  p  q    Obviously a bashism   BASHPID will be the process id of the tail command   The sed command is next after tail in the pipe  so the sed process id will be  BASHPID 1

User · Answer

Didn t see anyone offer my usual go-to for this   less  F  lt file gt  ctrl   c   lt search term gt   lt enter gt  shift   f   I prefer this  because you can use ctrl   c to stop and navigate through the file whenever  and then just hit shift   f to return to the live  streaming search

User · Answer

Use awk another great bash utility  instead of grep where you dont have the line buffered option  It will continuously stream your data from tail   this is how you use grep   tail -f  lt file gt    grep pattern   This is how you would use awk  tail -f  lt file gt    awk   pattern  print  0

User · Answer

Yes  this will actually work just fine  Grep and most Unix commands operate on streams one line at a time  Each line that comes out of tail will be analyzed and passed on if it matches

User · Answer

In most cases  you can tail -f  var log some log  grep foo and it will work just fine   If you need to use multiple greps on a running log file and you find that you get no output  you may need to stick the --line-buffered switch into your middle grep s   like so   tail -f  var log some log   grep --line-buffered foo   grep bar

User · Answer

you may consider this answer as enhancement    usually I am using   tail -F  lt fileName gt    grep --line-buffered   lt pattern gt  -A 3 -B 5   -F is better in case of file rotate  -f will not work properly  if file rotated   -A and -B is useful to get lines just before and after the pattern occurrence    these blocks will appeared between dashed line separators  But For me I prefer doing the following  tail -F  lt file gt    less   this is very useful if you want to search inside streamed logs  I mean go back and forward and look deeply

User · Answer

Turn on grep s line buffering mode when using BSD grep  FreeBSD  Mac OS X etc   tail -f file   grep --line-buffered my pattern  It looks like a while ago --line-buffered didn t matter for GNU grep  used on pretty much any Linux  as it flushed by default  YMMV for other Unix-likes such as SmartOS  AIX or QNX   However  as of November 2020  --line-buffered is needed  at least with GNU grep 3 5 in openSUSE  but it seems generally needed based on comments below

User · Answer

This one command workes for me  Suse     mail-srv  var log   tail -f  var log mail info  grep --line-buffered LOGIN   gt  gt  logins to mail   collecting logins to mail service

User · Answer

I think that your problem is that grep uses some output buffering  Try   tail -f file   stdbuf -o0 grep my pattern   it will set output buffering mode of grep to unbuffered

[linux] How to 'grep' a continuous stream?

Examples related to linux

Examples related to bash

Examples related to shell

Examples related to grep

Examples related to tail