Bash

Using Pipes '|' from command line

In complex data projects, one bash command is often not enough to do the job. You'll need to apply several commands after each other. In theory, you could do that by creating temporary files as the outputs to reuse them in the next commands as inputs... But that'd be repetitive and way too complicated.


To get things done more simply and cleanly, you can use pipes instead.

A pipe takes the output of your command and redirects it directly into your next command.


Examples:

head -50 some_data.csv | tail -10
Takes the first 50 rows of the some_data.csv file. The pipe redirects this output into tail, which prints the last 10 rows of those first 50 rows. As a result, you will have the rows from 41 to 50.


sort some_data.csv | uniq -c
Sorts the some_data.csv file and then it counts all occurrences of every unique row.


cut -d';' -f1 some_data.csv | grep "something" | wc -l
Takes the first column of the some_data.csv file. Then it filters for the rows that contain the string "something". Eventually it counts the lines. As a result, you will have the number of lines that contain the string "something" in the first column of the file.