...
View the content of text-based files.
Search a file for a string.
Search for a file/folder by name.
Redirect output from commands and pipe commands together.
Be very exercise based to allow practice of commands and concepts.
Title 01
|
|
---|
Title 02
hello | goodbyre |
---|---|
one | two |
no title
...
Table of Contents | ||
---|---|---|
|
Headers and Sections
...
Code Examples
...
Two Column Tables are nice ways to separate content/ Background info along with a code example on the same “Slide”. Please notice the table width. This should stop scroll bars from appearing
Bullets are nice to include for distinct points
yep
they
sure
are
This is 14 lines
...
Code Block |
---|
Please use the "code snippet" in the + button when creating code examples.
Also please do not go past the width of the table.
This is to prevent scroll bars appearing This is the Max number of code lines to show an example
|
Straight Code - No context
Code Block |
---|
Limit to 16 lines in the example. This is the end |
Same Thing With Images
...
Two Column Tables are nice ways to separate content/ Background info along with an image example on the same “Slide”. Please notice the table width. This should stop scroll bars from appearing
Bullets are nice to include for distinct points
yep
they
sure
are
This is 14 lines
...
...
Alternatively No Table
...
Finally The End
...
Link to Previous sub-module or Home Module
...
Table of Contents | ||
---|---|---|
|
...
01 View/Search a File
01.01 Setting Up
Question:
There is a folder called
intro_to_linux
within the/project/arccatrain/
folder.How would you copy this folder into your home folder?
...
01.02 Setting Up: Answer(s)
Answer: There are a number of ways…
Code Block |
---|
# Move to your home folder and copy int this location.
# The “.” marks the current working directory.
[arccanetrain]$ cd
[~]$ cp -r /project/arccanetrain/intro_to_linux/ .
cp: cannot open 'intro_to_linux/workshop_me.txt' for reading: Permission denied
[~]$ ls
Desktop Documents Downloads intro_to_linux
# Move into the /project/arccatrain/ folder and copy from there into your home.
# The “~” is short for your home folder.
[~]$ cd /project/arccanetrain/
[arccanetrain]$ cp -r intro_to_linux/ ~
cp: cannot open 'intro_to_linux/workshop_me.txt' for reading: Permission denied
[arccanetrain]$ ls ~
Desktop Documents Downloads intro_to_linux
# Why do we see the cp related permission denied?
# -rw------- 1 arcc-t05 arccanetrain 23 Oct 5 07:20 workshop_me.txt
# What happens to this file? It does not get copied. |
...
01.03a View the content of files.
Command | Description | ||
cat |
| ||
more |
| ||
head |
|
...
01.03b View the content of files.
Command | Description | ||
tail |
|
...
01.04 Exercises
Code Block |
---|
[]$ cd ~/intro_to_linux/
[intro_to_linux]$ cat software.csv
[intro_to_linux]$ cat -n software.csv
# Press spacebar to scroll through.
# Press ‘q’ to quit at any time.
[intro_to_linux]$ more software.csv
[intro_to_linux]$ head software.csv
[intro_to_linux]$ head –n 5 software.csv
[intro_to_linux]$ tail software.csv
[intro_to_linux]$ tail –n 5 software.csv |
...
01.05 Search for a string within a text file (grep)
Command | Description | ||
grep |
|
...
01.06 Examples: Search a file:
Code Block |
---|
# Remember: grep is case-sensitive
[intro_to_linux]$ grep NVIDIA software.csv
libraries and toolkits,cuDNN,cudnn,beartooth,The NVIDIA CUDA Deep...
libraries and toolkits,TensorRT,,beartooth,"NVIDIA TensorRT, an...
# Nothing is returned.
[intro_to_linux]$ grep nvidia software.csv
[intro_to_linux]$
# Neither of the above picked up “Nvidia”.
[intro_to_linux]$ grep -i NVidia software.csv
compiler,NVidia HPC SDK,nvhpc,"beartooth,teton"...
libraries and toolkits,cuDNN,cudnn,beartooth,The NVIDIA CUDA Deep...
libraries and toolkits,TensorRT,,beartooth,"NVIDIA TensorRT, an...
# Ignore the case of the word to search for.
[intro_to_linux]$ grep -n -i NVidia software.csv
145:compiler,NVidia HPC SDK,nvhpc,"beartooth,teton"...
152:libraries and toolkits,cuDNN,cudnn,beartooth,The NVIDIA CUDA Deep...
166:libraries and toolkits,TensorRT,,beartooth,"NVIDIA TensorRT, an... |
...
01.07 Examples: Search folders and files:
Code Block |
---|
[intro_to_linux]$ cd clusters/
[clusters]$ grep -i nvidia *
beartooth.html: .../788758554/NVidia+HPC+SDK">NVidia HPC SDK</a></td>
teton.html: .../788758554/NVidia+HPC+SDK">NVidia HPC SDK</a></td>
[clusters]$ cd ..
[intro_to_linux]$ grep -i nvidia *
grep: clusters: Is a directory
software.csv:compiler,NVidia HPC SDK,nvhpc,"beartooth,teton"...
software.csv:libraries and toolkits,cuDNN,cudnn,beartooth,The NVIDIA CUDA Deep...
software.csv:libraries and toolkits,TensorRT,,beartooth,"NVIDIA TensorRT, an...
[intro_to_linux]$ grep -r -i nvidia *
clusters/teton.html: .../788758554/NVidia+HPC+SDK">NVidia HPC SDK</a></td>
clusters/beartooth.html: .../788758554/NVidia+HPC+SDK">NVidia HPC SDK</a></td>
software.csv:compiler,NVidia HPC SDK,nvhpc,"beartooth,teton"...
software.csv:libraries and toolkits,cuDNN,cudnn,beartooth,The NVIDIA CUDA Deep...
software.csv:libraries and toolkits,TensorRT,,beartooth,"NVIDIA TensorRT, an... |
...
01.08 Exercises
Code Block |
---|
# The software.csv file takes the form:
[intro_to_linux]$ head software.csv
Type,Name,Module,Cluster,Description
application,Alphafold,alphafold,"beartooth,teton",AlphaFold...
application,Astral,astral,wildiris,ASTRAL is a tool...
application,Augustus,augustus,beartooth,AUGUSTUS is a program...
application,Avizo,avizo,loren-pre202308,Avizo is a general-purpose...
application,ANGSD,angsd,"beartooth,teton",ANGSD: is a software...
application,ANSYS,ansys,teton,"ANSYS is a general-purpose software... |
Questions:
Which named applications are related to the words “bayes”?
Which files contain reference to IPA?
...
01.09 Answers
Code Block |
---|
[intro_to_linux]$ grep -i bayes software.csv
application,Bayescan,bayescan,beartooth,"BayeScan aims...
application,Beast1,beast1,wildiris,BEAST is a cross-platform program for Bayesian...
application,Beast2,beast2,beartooth,"BEAST 2 is a cross-platform program for Bayesian...
application,Freebayes,freebayes,beartooth,"freebayes is a Bayesian genetic...
application,Jags,jags,"beartooth,teton",Just Another Gibbs Sampler. It is a program for analysis of Bayesian hierarchical...
application,RevBayes,revbayes,wildiris,Bayesian phylogenetic...
application,ROHan,rohan,teton,"ROHan is a Bayesian framework...
application,SourceTracker2,sourcetracker2,"beartooth,teton","SourceTracker, a Bayesian approach...
[intro_to_linux]$ grep -r IPA *
clusters/beartooth.html: .../pages/1893597185/IPA">IPA</a></td>
software.csv:application,IPA,ipa,beartooth,Improved Phased Assembler (IPA) is... |
...
02 Search for a File
...
02.01a Searching for Files: find
Let us look at a folder with many subfolders and files.
Code Block |
---|
[]$ cd ~/intro_to_linux
[intro_to_linux]$ ls
clusters data software.csv
[intro_to_linux]$ ls -R
.:
clusters data Intro_to_linux.pdf software.csv vegatables.txt workshop_all.txt workshop_me.txt
./clusters:
beartooth.html loren.html teton.html wildiris.html
./data:
2021 2022 2023 dd.tx
./data/2021:
Apr Nov Sep
./data/2021/Apr:
20210403.txt 20210427.txt 20210428.txt |
...
02.01b Searching for Files: find
Code Block |
---|
./data/2021/Nov:
20211114.txt 20211115.txt 20211116.txt hello.txt
./data/2021/Sep:
20210908.txt 20210921.txt
./data/2022:
Dec Feb Hello.csv Jul Jun
./data/2022/Dec:
20221207.txt 20221220.txt 20221230.txt 20221231.txt
./data/2022/Feb:
20220203.txt 20220223.txt
./data/2022/Jul:
20220720.txt 20220722.txt 20220723.TX
... |
...
02.02 Searching for Files: find
Command | Description | ||
find |
|
...
02.03 Examples
Code Block |
---|
[]$ cd ~/intro_to_linux/
[intro_to_linux]$ find . -name 20230121.txt
./data/2023/Jan/20230121.txt
# Check that this file is within the returned location.
[intro_to_linux]$ ls data/2023/Jan/
20230102.txt 20230108.txt 20230115.txt 20230121.txt
# Nothing returned – no file exists called “20230120.txt”
[intro_to_linux]$ find . -name 20230120.txt
[arcc-t05@blog1 intro_to_linux]$
[intro_to_linux]$ find . -name README.txt
./data/2021/README.txt
# find is case–sensitive: use –iname option
[intro_to_linux]$ find . -iname README.txt
./data/2021/README.txt
./data/2022/readme.txt
./data/2023/ReadMe.txt |
...
02.04 Examples
Code Block |
---|
# Use wildcards to find all files with the postfix .csv:
[intro_to_linux]$ find . -name "*.csv"
./software.csv
./data/2022/Hello.csv
# Find any files/folders that contain the string “dec”
[arcc-t05@blog1 intro_to_linux]$ find . -name "*dec*"
./data/2022/Dec/2022_dec_01.txt
[arcc-t05@blog1 intro_to_linux]$ find . -iname "*dec*"
./data/2022/Dec
./data/2022/Dec/2022_dec_01.txt
# Find only folders.
[arcc-t05@blog1 intro_to_linux]$ find . -type d -iname "*dec*"
./data/2022/Dec
# Find only files
[arcc-t05@blog1 intro_to_linux]$ find . -type f -iname "*dec*"
./data/2022/Dec/2022_dec_01.txt |
...
02.05 Exercises
Questions:
What do we notice about some of the
find
command options?Find any files that contain the string “
hello
”, regardless of case, within their filename.Find any folders or files that contain the string “
feb
” regardless of case.Can you list only the folders?
Find any files that have the postfix “
tx
” – must be lowercase.
...
02.06 Answers
1: What do we notice about some of the find command options?
That some of the single dash options (
-name
) are similar to long-names and not single letters.
2: Find any files that contain the string “hello
”, regardless of case, within their filename.
Code Block |
---|
[intro_to_linux]$ find . -name "hello“
[intro_to_linux]$ find . -name "hello.*"
./data/2021/Nov/hello.txt
[intro_to_linux]$ find . -iname "hello.*"
./data/2021/Nov/hello.txt
./data/2022/Hello.csv
./data/2023/Mar/HELLO.txt |
...
02.07 Answers
3: Find any folders or files that contain the string “feb
” regardless of case.
Can you list only the folders?
Code Block |
---|
[intro_to_linux]$ find . -name feb
./data/2021/feb
[intro_to_linux]$ find . -iname feb
./data/2021/feb
./data/2023/Feb
[intro_to_linux]$ find . -iname "*feb*"
./data/2021/feb
./data/2021/feb/february_01_2021.tx
./data/2022/February
./data/2023/Feb
[intro_to_linux]$ find . -type d -iname "*feb*"
./data/2021/feb
./data/2022/February
./data/2023/Feb |
...
02.08a Answers
4: Find any files that have the postfix “tx
” – must be lowercase.
Code Block |
---|
[intro_to_linux]$ find . -name "tx"
[intro_to_linux]$ find . -name "*tx*"
./data/2021/README.txt
./data/2021/Nov/20211115.txt
./data/2021/Nov/hello.txt
./data/2021/Nov/20211114.txt
…
[intro_to_linux]$ find . -name "*tx"
./data/dd.tx
./data/2021/feb/february_01_2021.tx
./data/2023/Jan/texttx
[intro_to_linux]$ find . -name "*.tx"
./data/dd.tx
./data/2021/feb/february_01_2021.tx |
...
02.08a Answers
Code Block |
---|
# dd.tx is actually a folder.
# Notices the ’d’ in the long format list.
[intro_to_linux]$ ls -l data
total 4
drwxrwxr-x 6 arcc-t05 arcc-t05 2021
drwxrwxr-x 6 arcc-t05 arcc-t05 2022
drwxrwxr-x 5 arcc-t05 arcc-t05 2023
drwxrwxr-x 2 arcc-t05 arcc-t05 dd.tx
[intro_to_linux]$ find . -type f -name "*.tx"
./data/2021/feb/february_01_2021.tx
# We explicitly want lowercase.
[intro_to_linux]$ find . -type f -iname "*.tx"
./data/2021/feb/february_01_2021.tx
./data/2022/20220723.TX |
...
03 Output Redirection and Pipes
...
03 Output Redirection and Pipes
Redirection of output: > vs >>
A redirect sends a channel of output to a file.
You can redirect a file as input to a command using
<
and<<
(not looked at).
Using pipe “|’
A pipe passes standard output as the standard input to another command
Examples of the form:
View a text file and pipe to grep.
Cat a list and sort by line.
Sort and then find unique items.
View folder contents and look for a specifically named name.
...
03.01 Redirection of output: > vs >>
Code Block |
---|
# Writes out to the command line.
[intro_to_linux]$ grep -i bayes software.csv
# Redirects the output to a file called apps.txt
[intro_to_linux]$ grep -i bayes software.csv > apps.txt
[intro_to_linux]$ ls
apps.txt clusters data software.csv
[intro_to_linux]$ cat apps.txt
# Overwrites any existing file called apps.txt
[intro_to_linux]$ grep -i IPA software.csv > apps.txt
[intro_to_linux]$ cat apps.txt
[intro_to_linux]$ rm apps.txt
# Overwrites existing apps.txt
[intro_to_linux]$ grep -i bayes software.csv > apps.txt
# Appends to the existing file.
[intro_to_linux]$ grep -i IPA software.csv >> apps.txt |
...
03.02 Example: Using pipe “|” from a file.
Code Block |
---|
[intro_to_linux]$ cat fruits.txt
Gooseberry
Apple
Apricot
Avocado
Strawberry
...
[intro_to_linux]$ cat fruits.txt | wc -l
97 |
...
03.03 Example continued:
Code Block |
---|
# The order of items is the same as listed within the fruits.txt file.
[intro_to_linux]$ cat fruits.txt | grep berry
Gooseberry
Strawberry
Bilberry
Blackberry
Marionberry
Blueberry
Boysenberry
Gooseberry
Cloudberry
Elderberry
Goji berry
Honeyberry
Juniper berry
Cranberry
Cranberry
Marionberry
Gooseberry
Mulberry
Salmonberry
Huckleberry
Raspberry
Salal berry |
...
03.04 Example continued:
Code Block |
---|
# Notice the duplicates.
[intro_to_linux]$ cat fruits.txt | grep berry | sort
Bilberry
Blackberry
Blueberry
Boysenberry
Cloudberry
Cranberry
Cranberry
Elderberry
Goji berry
Gooseberry
Gooseberry
Gooseberry
Honeyberry
Huckleberry
Juniper berry
Marionberry
Marionberry
Mulberry
Raspberry
Salal berry
Salmonberry
Strawberry |
...
03.05 Example continued:
Code Block |
---|
# Duplicates have been removed leaving only the unique names.
[intro_to_linux]$ cat fruits.txt | grep berry | sort | uniq
Bilberry
Blackberry
Blueberry
Boysenberry
Cloudberry
Cranberry
Elderberry
Goji berry
Gooseberry
Honeyberry
Huckleberry
Juniper berry
Marionberry
Mulberry
Raspberry
Salal berry
Salmonberry
Strawberry
[intro_to_linux]$ cat fruits.txt | grep berry | sort | uniq | wc –l
18 |
...
03.06 Example continued
Code Block |
---|
[intro_to_linux]$ cat fruits.txt | grep berry | sort | uniq > berries.txt
[intro_to_linux]$ cat berries.txt
Bilberry
Blackberry
Blueberry
Boysenberry
Cloudberry
Cranberry
Elderberry
Goji berry
Gooseberry
Honeyberry
Huckleberry
Juniper berry
Marionberry
Mulberry
Raspberry
Salal berry
Salmonberry
Strawberry
[intro_to_linux]$ cat berries.txt | wc -l
18 |
...
03.07 Example: Pipe from ls command
Code Block |
---|
[intro_to_linux]$ ls -R
[intro_to_linux]$ ls -R | grep "Feb"
February
./data/2022/February:
Feb
./data/2023/Feb:
[intro_to_linux]$ ls -R | grep -i "Feb"
feb
./data/2021/feb:
february_01_2021.tx
February
./data/2022/February:
Feb
./data/2023/Feb: |
...
03.08 Exercises
How does the
wc
command work? What are its options?How does the
sort
command work? What are its options?How does the
uniq
command work? What are its options?How many unique varieties of beans are there in the
vegetables.txt
file?
...
03.09 Answers
4: How many unique varieties of beans ae there in the vegetables.txt file?
How do you deal with “soy beans” vs “Soy Beans”?
What options does the
uniq
command provide?
Code Block |
---|
[intro_to_linux]$ cat vegatables.txt | grep -i beans | sort | uniq -i | wc -l
12 |
...
04 More Intermediate Features, Next Steps, Suggestions and Summary
...
04.01a More Intermediate Features
Environment Variables: Define the behavior of the environment: Try:
echo $HOME
echo $USER
echo $SHELL
echo $PATH
File searching/manipulation
sed: stream editor for filtering and transforming text
gawk: pattern scanning and processing language
Ability to update file permission and ownership:
chmod
/chown
User-case of sharing files/folders.
...
04.01b More Intermediate Features
Aliases in
.bashrc
.Create short-cuts of popular/frequently used commands.
Text editors: vi/vim/nano
vimtutor
touch
Remote access with ssh.
...
04.02 Next Steps, Suggestions
Next Steps on using Linux:
Practicing using Linux online.
Dual boot a Windows machine with Linux.
Run a container image.
UW Researcher? Create a project on the Beartooth cluster with your PI.
...
04.03 Further Trainings: UWYO LinkedIn
...
Introduction to Linux
Learning Linux Command Line
Linux: Files and Permissions
Linux: Over and Installation
Learning Linux Shell Scripting
...
04.04 Request an Account with ARCC
...
...
04.05 Summary
In this workshop we have:
How to search for a string within a file.
How to find a file.
How to redirect the output of a command into a file.
How to use pipes to direct the output of one command as the input into another command.