Unix plumbing

This exercise is about redirecting the standard streams of commands to files, and connecting commands to each other with pipes, to analyze text on your cloud server.

Do this exercise on your cloud server, through SSH: its first steps look at files that only exist on a server.

Legend

Parts of this exercise are annotated with the following icons:

• A task you MUST perform to complete the exercise
• Optional step that you may perform to make sure that everything is working correctly, or to set up additional tools that are not required but can help you, or optional extra exercises to explore the topic further
• Advanced tips on how to go further (or challenges!)
• The end of the exercise
• The architecture of the software you ran or deployed during this exercise
• Troubleshooting tips: how to fix common problems you might encounter

Where does each line go?

Connect to your cloud server and go to your home directory, where the files of this exercise will be created:

$> cd

Search the SSH configuration of your server for the word PasswordAuthentication:

$> grep -r PasswordAuthentication /etc/ssh

Some of the lines it displays are output data, and some are error messages. Which are which, and why are there errors?

The /etc/ssh directory holds the configuration of SSH on your server:

• sshd_config configures the SSH server, the program that accepts your connection when you run ssh from your computer.
• ssh_config configures the SSH client, used when you run ssh from this server to connect to another machine.
• The ssh_host_..._key files are the server’s private keys, which prove its identity to the clients that connect to it: the fingerprint you checked the first time you connected identifies one of these keys. These files can only be read by root, and you are not root (unless you use sudo).

PasswordAuthentication is the setting that decides whether the SSH server accepts passwords, or only keys like yours. A line starting with # is a comment, which the server ignores: these lines document the setting and its default value.

Would you like to know more?

Terrific! Always tell me more!

Stop telling me more...

Now run the same command three more times, redirecting:

• Its standard output stream to a file named found.txt.
• Its standard error stream to a file named errors.txt.
• Both output streams to a single file named all.txt.

Each time, what is still displayed in your terminal, and what ends up in the file? Display the file with cat to see.

Did it work?

Every command ends with an exit status, which the shell keeps in the special variable $?. Display it with echo $? right after each of these commands:

$> grep PasswordAuthentication /etc/ssh/sshd_config
$> echo $?

$> grep unicorn /etc/ssh/sshd_config
$> echo $?

$> grep -r PasswordAuthentication /etc/ssh
$> echo $?

What does each exit status mean? The last command found lines: why is its exit status not 0?

Download the song

In your home directory, download the song that the next pipelines work on:

$> curl -L https://tinyurl.com/archidep-otr > rainbow.txt

Tip

This command is already a redirection: curl prints what it downloads to its standard output stream, and > sends it into the rainbow.txt file instead of your terminal.

Display the file:

$> cat rainbow.txt
Somewhere over the rainbow
...

Pipe the song

Use command pipelines to answer questions about the song. For example, how many lines are there in the text?

$> cat rainbow.txt | wc -l
51

Now it’s your turn:

• Count the number of words in the text

• Print the lines of the text containing the word rainbow

• Do the same but without any duplicates

• Print the second word of each line in the text

• Count the number of times the letter e is used (case-insensitive)

Here are a few commands you might find useful for these pipelines. They all read the data from their standard input stream (or from a file you name), and print the result on their standard output stream, so they can be piped into each other:

Tip

If you want to know more about any of these commands or their options, type man <command>, e.g. man cut.

Challenge

What are the five most used words in the song (case-insensitive), and how many times is each one used?

Pipe your server’s users

The file /etc/passwd lists the users of your server, one per line, with fields separated by colons (:). The seventh and last field of each line is the user’s login shell, the program started when that user logs in.

Use a pipeline to count how many users have each login shell.

Tip

Remember the cut command you used to print the second word of each line of the song: its -d option gives the character that separates the fields (the delimiter), and its -f option the number of the field to keep.

Most users have /usr/sbin/nologin as their login shell. Why?

Discard the errors

In the first step, you ran this command, which displays both the lines it finds and error messages:

$> grep -r PasswordAuthentication /etc/ssh

Find out how to run it so that it displays only the lines it found, without creating any file. The optional section of the subject on how to discard an output stream explains how.

What have I done?

You took commands that each do one small job, and decided where their input comes from and where their output goes: into a file, or straight into another command. None of these commands was written to work with files or with each other: the shell connected them.

Every process has three standard streams: one input and two outputs. A program writes its data to the standard output stream and its errors to the standard error stream. In your terminal, both end up on the same screen, which is why they look alike. They only show that they are separate once you send one of them somewhere else: the data went into found.txt, and the errors kept reaching you.

Redirection is done by the shell, not by the program. The shell connects the streams before the program starts, and the program does not know: grep wrote to its standard output as usual, and it ended up in a file.

The exit status is a third answer, separate from both outputs. grep found lines and still reported a failure, because it also met errors. Scripts and tools that run commands for you rely on the exit status, not on what is printed, to decide whether a command worked.

A pipe connects the standard output stream of one process to the standard input stream of the next. Each command in a pipeline only knows how to do one job, and the pipeline does what none of them can do alone: count the uses of a word, or the users of each shell. The error messages are not part of the pipeline: they still go to your terminal, so that you see them.

This is the Unix philosophy: small programs that do one thing well, work together, and handle text streams. It works because every program speaks the same language, a stream of text in and a stream of text out.

Later in the course, the applications you deploy will also write to their standard streams, and that is where you will look for their logs.

Troubleshooting

Permission denied when redirecting to a file

You are in a directory where you are not allowed to create files, such as /etc. Go back to your home directory with cd or cd ~ and run the command again.

rainbow.txt: No such file or directory

You are not in the directory where you downloaded the file. Go back to your home directory with cd or cd ~, or download it again as in Download the song.

The grep lines on my server are not the same as the solution’s

The SSH configuration of your server may differ slightly from the one used to write the solution. What matters is which lines are output data and which are error messages, not their exact text.