Text Processing: grep, sed, awk & Pipelines

The Unix toolbox for logs and data files: filter, cut, count, transform and summarize.

Unix tools each do one job and read and write lines of text, so they can be chained with | into pipelines. With grep, sort, uniq, cut, sed and awk you can answer most questions about a log file in one line.

Questions about the log and the results

grep ERROR prints matching lines.

awk '{print $3}' prints the third field (the level); sort | uniq -c counts each distinct value; sort -rn orders by count.

cut -d' ' -f4 takes the fourth space-separated field; tr -d ':' deletes the colons; sort -u keeps unique values.

sed 's/old/new/' substitutes text on every line.

awk -F, splits at commas; NR > 1 skips the header; END runs after the last line, here printing totals and averages.

The toolbox

cat app.log | grep ERROR | wc -l grep -c ERROR app.log Not wrong, but grep can read files and count itself; fewer processes, clearer intent.

How do you list the 5 most frequent subsystems in app.log?

awk '{print $4}' app.log | sort | uniq -c | sort -rn | head -5