Shell and Scripting

Reddit Discussion

The shell is an efficient, textual interface to your computer.

The shell prompt: what greets you when you open a terminal. Lets you run programs and commands; common ones are:

But the shell lets you do so much more; you can invoke any program on your computer, and command-line tools exist for doing pretty much anything you may want to do. And they’re often more efficient than their graphical counterparts. We’ll go through a bunch of those in this class.

The shell provides an interactive programming language (“scripting”). There are many shells:

In this class we’ll focus on the ubiquitous sh and bash, but feel free to play around with others. I like fish.

Shell programming is a very useful tool in your toolbox. Can either write programs directly at the prompt, or into a file. #!/bin/sh + chmod +x to make shell executable.

Working with the shell

Run a command a bunch of times:

for i in $(seq 1 5); do echo hello; done

There’s a lot to unpack:

We have variables:

for f in $(ls); do echo $f; done

Will print each file name in the current directory. Can also set variables using = (no space!):

foo=bar
echo $foo

There are a bunch of “special” variables too:

To only print directories

for f in $(ls); do if test -d $f; then echo dir $f; fi; done

More to unpack here:

But wait! This is wrong! What if a file is called “My Documents”?

Argument splitting

Bash splits arguments by whitespace; not always what you want!

Globbing is the answer!

Whitespace issues don’t stop there:

Composability

Shell is powerful in part because of composability. Can chain multiple programs together rather than have one program that does everything.

The key character is | (pipe).

All programs you launch (“processes”) have three “streams”:

Bash also provides a number of other ways to compose programs.

You can group commands with (a; b) | tac: run a, then b, and send all their output to tac, which prints its input in reverse order.

A lesser-known, but super useful one is process substitution. b <(a) will run a, generate a temporary file-name for its output stream, and pass that file-name to b. For example:

diff <(journalctl -b -1 | head -n20) <(journalctl -b -2 | head -n20)

will show you the difference between the first 20 lines of the last boot log and the one before that.

Job and process control

What if you want to run longer-term things in the background?

What about other stuff running on your computer?

Flags

Most command line utilities take parameters using flags. Flags usually come in short form (-h) and long form (--help). Usually running CMD -h or man CMD will give you a list of the flags the program takes. Short flags can usually be combined, running rm -r -f is equivalent to running rm -rf or rm -fr. Some common flags are a de facto standard and you will seem them in many applications:

Also, a double dash -- is used in built-in commands and many other commands to signify the end of command options, after which only positional parameters are accepted. So if you have a file called -v (which you can) and want to grep it grep pattern -- -v will work whereas grep pattern -v won’t. In fact, one way to create such file is to do touch -- -v.

Exercises

  1. If you are completely new to the shell you may want to read a more comprehensive guide about it such as BashGuide. If you want a more indepth introduction The Linux Command Line is a good resource.

  2. PATH, which, type

    We briefly discussed that the PATH environment variable is used to locate the programs that you run through the command line. Let’s explore that a little further

    • Run echo $PATH (or echo $PATH | tr -s ':' '\n' for pretty printing) and examine its contents, what locations are listed?
    • The command which locates a program in the user PATH. Try running which for common commands like echo, ls or mv. Note that which is a bit limited since it does not understand shell aliases. Try running type and command -v for those same commands. How is the output different?
    • Run PATH= and try running the previous commands again, some work and some don’t, can you figure out why?
  3. Special Variables
    • What does the variable ~ expands as? What about .? And ..?
    • What does the variable $? do?
    • What does the variable $_ do?
    • What does the variable !! expand to? What about !!*? And !l?
    • Look for documentation for these options and familiarize yourself with them
  4. xargs

    Sometimes piping doesn’t quite work because the command being piped into does not expect the newline separated format. For example file command tells you properties of the file.

    Try running ls | file and ls | xargs file. What is xargs doing?

  5. Shebang

    When you write a script you can specify to your shell what interpreter should be used to interpret the script by using a shebang line. Write a script called hello with the following contentsmake it executable with chmod +x hello. Then execute it with ./hello. Then remove the first line and execute it again? How is the shell using that first line?

       #! /usr/bin/python
    
       print("Hello World!")
    

    You will often see programs that have a shebang that looks like #! usr/bin/env bash. This is a more portable solution with it own set of advantages and disadvantages. How is env different from which? What environment vairable does env use to decide what program to run?

  6. Pipes, process substitution, subshell

    Create a script called slow_seq.sh with the following contents and do chmod +x slow_seq.sh to make it executable.

       #! /usr/bin/env bash
    
       for i in $(seq 1 10); do
               echo $i;
               sleep 1;
       done
    

    There is a way in which pipes (and process substitution) differ from using subshell execution, i.e. $(). Run the following commands and observe the differences:

    • ./slow_seq.sh | grep -P "[3-6]"
    • grep -P "[3-6]" <(./slow_seq.sh)
    • echo $(./slow_seq.sh) | grep -P "[3-6]"
  7. Misc
    • Try running touch {a,b}{a,b} then ls what did appear?
    • Sometimes you want to keep STDIN and still pipe it to a file. Try running echo HELLO | tee hello.txt
    • Try running cat hello.txt > hello.txt what do you expect to happen? What does happen?
    • Run echo HELLO > hello.txt and then run echo WORLD >> hello.txt. What are the contents of hello.txt? How is > different from >>?
    • Run printf "\e[38;5;81mfoo\e[0m\n". How was the output different? If you want to know more, search for ANSI color escape sequences.
    • Run touch a.txt then run ^txt^log what did bash do for you? In the same vein, run fc. What does it do?
  8. Keyboard shortcuts

    As with any application you use frequently is worth familiarising yourself with its keyboard shortcuts. Type the following ones and try figuring out what they do and in what scenarios it might be convenient knowing about them. For some of them it might be easier searching online about what they do. (remember that ^X means pressing Ctrl+X)

    • ^A, ^E
    • ^R
    • ^L
    • ^C, ^\ and ^D
    • ^U and ^Y

Edit this page.