Versions Compared

Old Version 12

changes.mady.by.user Simon Alexander

Saved on

compared with

New Version Current

changes.mady.by.user Simon Alexander

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Output Redirection and Pipes

Info

  • Redirection of output: > vs >>

    • A redirect sends a channel of output to a file.

    • A channel refers to standard input/output as well as standard error (not covered here).

    • You can redirect a file as input to a command using < and << (not

...

    • covered here).

  • Using pipe “|’

    • A pipe passes standard output as the standard input to another command

  • Examples of the form: 

    • View a text file and pipe to the grep command to filter lines by looking for text.

    • Cat a list and sort by line.

    • Sort and then find unique items.

    • View folder contents and look for a specifically named

...

    • filename.

...

Redirection of output: > vs >>

...

Code Block
[intro_to_linux]$ grep -i bayes software.csv
Info

Pipe Redirect this output to a file:

Code Block
[intro_to_linux]$ grep -i bayes software.csv > apps.txt
[intro_to_linux]$ ls 
apps.txt  clusters  data  software.csv
[intro_to_linux]$ cat apps.txt

...

Info

Over the next set of examples we will pipe (using the “|” character) together a series of commands that will “generate a file that contains a sorted list of unique terms that include the sub string berry”.

Code Block
[intro_to_linux]$ cat fruits.txt
Gooseberry
Apple
Apricot
Avocado
Strawberry
...
[intro_to_linux]$ cat fruits.txt | wc -l
97
97
Info

We could use the wc command as it is normally intended, and confirm the file contains 97 lines.

Code Block
[intro_to_linux]$ wc -l fruits.txt
97 fruits.txt

...

Example continued

Info

First step, find all the terms that contain the sub string berry.

...

Code Block
# Duplicates have been removed leaving only the unique names.
[intro_to_linux]$ cat fruits.txt | grep berry | sort | uniq
Bilberry
Blackberry
Blueberry
Boysenberry
Cloudberry
Cranberry
Elderberry
Goji berry
Gooseberry
Honeyberry
Huckleberry
Juniper berry
Marionberry
Mulberry
Raspberry
Salal berry
Salmonberry
Strawberry

[intro_to_linux]$ cat fruits.txt | grep berry | sort | uniq | wc –l
18

...

Code Block
[intro_to_linux]$ cat fruits.txt | grep berry | sort | uniq > berries.txt
[intro_to_linux]$ cat berries.txt
Bilberry
Blackberry
Blueberry
Boysenberry
Cloudberry
Cranberry
Elderberry
Goji berry
Gooseberry
Honeyberry
Huckleberry
Juniper berry
Marionberry
Mulberry
Raspberry
Salal berry
Salmonberry
Strawberry

[intro_to_linux]$ cat berries.txt | wc -l
18
Info

  • Although a simple example, it demonstrates the principle of piping four commands into a single call.

  • Without using the pipe command you would have had to create intermediate files, to store results, after each command.

  • As you get more confident, you can create more elaborate pipelines of

comands

  • commands.

Note

You might also be presented with a long pipeline/list of commands separated by the | character.

Now you know how to separate this pipeline into individual steps, dissecting by the | character, and then running each step individually.

...

Example: Pipe from ls command

...

Code Block
[intro_to_linux]$ ls -R
[intro_to_linux]$ ls -R | grep "Feb"
February
./data/2022/February:
Feb
./data/2023/Feb:

# Ignore case.
[intro_to_linux]$ ls -R | grep -i "Feb"
feb
./data/2021/feb:
february_01_2021.tx
February
./data/2022/February:
Feb
./data/2023/Feb:

...

Exercises: Pipeline

Info

Questions:

  1. How does the wc command work? What are its options?

  2. How does the sort command work? What are its options?

  3. How does the uniq command work? What are its options?

  4. How many unique varieties of beans are there in the vegetables.txt file?

...

Answers

Info

4

...

. How many unique varieties of beans are there in the vegetables.txt file?

  • How do you deal with “soy beans” vs “Soy Beans”

...

  • ?

Code Block
[intro_to_linux]$ cat vegatables.txt | grep -i beans | sort | uniq -i | wc -l
12

...

 Note
Make sure you are use the commands and options as you intend and understand and are able to describe and explain/justify.
Notice the difference between:
 Code Block
[salexan5@mblog1 intro_to_linux]$ cat vegatables.txt | grep beans | sort
...
kidney beans
...
soy beans
and
 Code Block
[salexan5@mblog1 intro_to_linux]$ cat vegatables.txt | grep -i beans | sort
...
kidney beans
kidney BEANS
...
soy beans
Soy Beans
  • Should “soy beans” and “Soy Beans” be treated the same or different? You need to decide with respect to the context and use case you are following, and intending.
  • What happens if you use bean instead of beans?
  • What does the uniq -i option do?
 Note
Note: Explore and understand how commands work, and the order they are run in. For example, in your own time, understand if there is a difference between sort | uniq versus uniq | sort?
 Expand
titleBrief Explanation:
 Info
The uniq command isn’t able to detect the duplicate lines unless they are adjacent to each other.
That is why we sort first.
Try it and see…
...
Previous
Search for a File
Workshop Home
Intro to Linux CLI: View, Find & Search
Next
CLI: Intermediate Features & Summary