SOLVED: Output of Multiple Greps Are Delayed or Missing (No Output) When Tailing a File

I recently ran into an issue when running the output of tail -F file.log through a series of multiple grep instances.

One level of grep worked fine and I saw the output in real time. However, once I added a second level, I would get nothing for minutes after, then sudden output (at first I thought my regex syntax was off).

What I discovered was happening is that grep (at least on FreeBSD) buffers its output by default. Multiple pipes/calls means multiple buffers and that causes the delay. Adding the –line-buffered option to grep forces grep to output each line immediately (and thus passing it on to the next grep process in the pipe):

# tail -F haystack.log | grep –line-buffered “needle” | grep -v –line-buffered ” junk “

Now you can filter your log files in real time!

As a side note I have used grep for years and only recently ran into this issue, specifically with Apache web server log files. There might be some interaction with how Apache buffers the writing of log files, or how tail -F works.