This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Shell

Some useful information about Shells

Some useful information about Shells.

1 - Linux Shell

Basic Linux commands in the shell.

Learning Objectives

  • Be able to know the basic commands to work in a Linux terminal.
  • Get familiar with Linux Commands

Topics covered


We introduce you to a number of useful shell commands. You may ask:

“Why is he so keen on telling me all about shells as I do have a beautiful GUI?”

You will soon learn that A GUI may not be that suitable if you like to manage 10, 100, 1000, 10000, … virtual machines. A commandline interface could be mcuh simpler and would allow scripting.

History

LINUX is a reimplementation by the community of UNIX which was developed in 1969 by Ken Thompson and Dennis Ritchie of Bell Laboratories and rewritten in C. An important part of UNIX is what is called the kernel which allows the software to talk to the hardware and utilize it.

In 1991 Linus Torvalds started developing a Linux Kernel that was initially targeted for PC’s. This made it possible to run it on Laptops and was later on further developed by making it a full Operating system replacement for UNIX.

Shell

One of the most important features for us will be to access the computer with the help of a shell. The shell is typically run in what is called a terminal and allows interaction to the computer with commandline programs.

There are many good tutorials out there that explain why one needs a linux shell and not just a GUI. Randomly we picked the first one that came up with a google query. This is not an endorsement for the material we point to, but could be a worth while read for someone that has no experience in Shell programming:

Certainly you are welcome to use other resources that may suite you best. We will however summarize in table form a number of useful commands that you may als find even as a RefCard.

We provide in the next table a number of useful commands that you want to explore. For more information simply type man and the name of the command. If you find a useful command that is missing, please add it with a Git pull request.

CommandDescription
man commandmanual page for the command
apropos textlist all commands that have text in it
lsDirectory listing
ls -lisalist details
treelist the directories in graphical form
cd dirnameChange directory to dirname
mkdir dirnamecreate the directory
rmdir dirnamedelete the directory
pwdprint working directory
rm fileremove the file
cp a bcopy file a to b
mv a bmove/rename file a to b
cat aprint content of filea
cat -n filenameprint content of filea with
line numbers
less aprint paged content of file a
head -5 aDisplay first 5 lines of file a
tail -5 aDisplay last 5 lines of file a
du -hs .show in human readable form the space used by the current directory
df -hshow the details of the disk file system
wc filenamecounts the word in a file
sort filenamesorts the file
uniq filenamedisplays only uniq entries in the file
tar -xvf dirtars up a compressed version of the directory
rsyncfaster, flexible replacement for rcp
gzip filenamecompresses the file
gunzip filenamecompresses the file
bzip2 filenamecompresses the file with
block-sorting
bunzip2 filenameuncompresses the file with block-sorting
clearclears the terminal screen
touch filenamechange file access and modification times or if file does not exist creates file
whodisplays a list of users that are currently logged on, for each user the login name, date and time of login, tty name, and hostname if not local are displayed
whoamidisplays the users effective id see also id
echo -n stringwrite specified arguments to standard output
datedisplays or sets date & time, when invoked without arguments the current date and time are displayed
logoutexit a given session
exitwhen issued at the shell prompt the shell will exit and terminate any running jobs within the shell
killterminate or signal a process by sending a signal to the specified process usually by the pid
psdisplays a header line followed by all processes that have controlling terminals
sleepsuspends execution for an interval of time specified in seconds
uptimedisplays how long the system has been running
time commandtimes the command execution in seconds
find / [-name] file-name.txtsearches a specified path or directory with a given expression that tells the find utility what to find, if used as shown the find utility would search the entire drive for a file named file-name.txt
diffcompares files line by line
hostnameprints the name of the current host system
whichlocates a program file in the users path
taildisplays the last part of the file
headdisplays the first lines of a file
topdisplays a sorted list of system processes
locate filenamefinds the path of a file
grep ‘word’ filenamefinds all lines with the word in it
grep -v ‘word’ filenamefinds all lines without the word in it
chmod ug+rw filenamechange file modes or Access Control Lists. In this example user and group are changed to read and write
chownchange file owner and group
historya build-in command to list the past commands
sudoexecute a command as another user
susubstitute user identity
unameprint the operating system name
set -o emacstells the shell to use Emacs commands.
chmod go-rwx filechanges the permission of the file
chown username filechanges the ownership of the file
chgrp group filechanges the group of a file
fgrep text filenamesearches the text in the given file
grep -R text .recursively searches for xyz in all files
find . -name *.pyfind all files with .py at the end
pslist the running processes
kill -9 1234kill the process with the id 1234
atque commands for later execution
crondaemon to execute scheduled commands
crontabmanage the time table for execution commands with cron
mount /dev/cdrom /mnt/cdrommount a filesystem from a cd rom to /mnt/cdrom
userslist the logged in users
whodisplay who is logged in
whoamiprint the user id
dmesgdisplay the system message buffer
lastindicate last logins of users and ttys
unameprint operating system name
dateprints the current date and time
time commandprints the sys, real and user time
shutdown -h “shut down”shutdown the computer
pingping a host
netstatshow network status
hostnameprint name of current host system
tracerouteprint the route packets take to network host
ifconfigconfigure network interface parameters
hostDNS lookup utility
whoisInternet domain name and network number directory service
digDNS lookup utility
wgetnon-interactive network downloader
curltransfer a URL
sshremote login program
scpremote file copy program
sftpsecure file transfer program
watch commandrun any designated command at regular intervals
awkprogram that you can use to select particular records in a file and perform operations on them
sedstream editor used to perform basic text transformations
xargsprogram that can be used to build and execute commands from STDIN
cat some_file.json | python -m json.toolquick and easy JSON validator
————————————————-

The command man

On Linux you find a rich set of manual pages for thes commands. Try to pick one and execute:

$ man ls

You will see somthing like this


LS(1)                     BSD General Commands Manual                    LS(1)

NAME
     ls -- list directory contents

SYNOPSIS
     ls [-ABCFGHLOPRSTUW@abcdefghiklmnopqrstuwx1] [file ...]

DESCRIPTION
     
     For each operand that names a file of a type other than directory,
     ls displays its name as well as any requested, associated
     information.  For each operand that names a file of type directory,
     ls displays the names of files contained within that directory, as
     well as any requested, associated information.

     If no operands are given, the contents of the current directory are
     displayed.  If more than one operand is given, non-directory
     operands are displayed first; directory and non-directory operands
     are sorted separately and in lexicographical order.

     The following options are available:

     -@      Display extended attribute keys and sizes in long (-l) output.

     -1      (The numeric digit ``one''.)  Force output to be one entry
             per line.  This is the default when output is not to a terminal.

     -A      List all entries except for . and ...  Always set for the
             super-user.

     -a      Include directory entries whose names begin with a dot (.).


     ... on purpose cut ... instead try it yourslef

Multi-command execution

One of the important features is that one can execute multiple commands in the shell.

To execute command 2 once command 1 has finished use

command1; command2

To execute command 2 as soon as command 1 forwards output to stdout use

command1; command2

To execute command 1 in the background use

command1 &

Keyboard Shortcuts

These shortcuts will come in handy. Note that many overlap with emacs short cuts.

KeysDescription
Up ArrowShow the previous command
Ctrl + zStops the current command
Resume with fg in the foreground
Resume with bg in the background
Ctrl + cHalts the current command
Ctrl + lClear the screen
Ctrl + aReturn to the start of the line
Ctrl + eGo to the end of the line
Ctrl + kCut everything after the cursor to a special clipboard
Ctrl + yPaste from the special clipboard
Ctrl + dLogout of current session, similar to exit

bashrc, bash_profile or zprofile

Usage of a particular command and all the attributes associated with it, use man command. Avoid using rm -r command to delete files recursively. A good way to avoid accidental deletion is to include the following in the file .bash_profile or .zprofile on macOS or .bashrc on other platforms:

alias rm='rm -i'
alias mv='mv -i'
alias h='history'

Makefile

Makefiles allow developers to coordinate the execution of code compilations. This not only includes C or C++ code, but any translation from source to a final format. For us this could include the creation of PDF files from latex sources, creation of docker images, and the creation of cloud services and their deployment through simple workflows represented in makefiles, or the coordination of execution targets.

As makefiles include a simple syntax allowing structural dependencies they can easily adapted to fulfill simple activities to be executed in repeated fashion by developers.

An example of how to use Makefiles for docker is provided at

An example on how to use Makefiles for LaTeX is provided at

Makefiles include a number of rules that are defined by a target name. Let us define a target called hello that prints out the string “Hello World”.

hello:
    @echo "Hello World"

Important to remember is that the commands after a target are not indented just by spaces, but actually by a single TAB character. Editors such as emacs will be ideal to edit such Makefiles, while allowing syntax highlighting and easy manipulation of TABs. Naturally other editors will do that also. Please chose your editor of choice. One of the best features of targets is that they can depend on other targets. Thus, iw we define

hallo: hello
    @echo "Hallo World"

our makefile will first execute hello and than all commands in hallo. As you can see this can be very useful for defining simple dependencies.

In addition we can define variables in a makefile such as

HELLO="Hello World"

hello:
    @echo $(HELLO)

and can use them in our text with $ invocations.

Moreover, in sophisticated Makefiles, we could even make the targets dependent on files and a target rules could be defined that only compiles those files that have changed since our last invocation of the Makefile, saving potentially a lot of time. However, for our work here we just use the most elementary makefiles.

For more information we recommend you to find out about it on the internet. A convenient reference card is available at

chmod

The chmod command stand for change mode and changes the access permissions for a given file system object(s). It uses the following syntax: chmod [options] mode[,mode] file1 [file2…]. The option parameters modify how the process runs, including what information is outputted to the shell:

Option:Description:
-f, --silent, --quietForces process to continue even if errors occur
-v, --verboseOutputs for every file that is processed
-c, --changesOutputs when a file is changed
--reference=RFileUses RFile instead of Mode values
-R, --recursiveMake changes to objects in subdirectories as well
--helpShow help
--versionShow version information

Modes specify which rights to give to which users. Potential users include the user who owns the file, users in the file’s Group, other users not in the file’s Group, and all, and are abbreviated as u, g, o, and a respectively. More than one user can be specified in the same command, such as chmod –v ug(operator)(permissions) file.txt . If no user is specified, the command defaults to a. Next, a + or - indicates whether permissions should be added or removed for the selected user(s). The permissions are as follows:

Permission:Description:
rRead
wWrite
xExecute file or access directory
XExecute only if the object is a directory
sSet the user or group ID when running
tRestricted deletion flag or sticky mode
uSpecifies the permissions the user who owns the file has
gSpecifies the permissions of the group
oSpecifies the permissions of users not in the group

More than one permission can be also be used in the same command as follows:

$ chmod –v o+rw file.txt

Multiple files can also be specified:

$ chmod a-x,o+r file1.txt file2.txt

Exercises

E.Linux.1

Familiarize yourself with the commands

E.Linux.2

Find more commands that you find useful and add them to this page.

E.Linux.3

Use the sort command to sort all lines of a file while removing duplicates.

E.Linux.4

Should there be other commands listed in the table with the Linux commands If so which? Create a pull request for them.

E.Linux.5

Write a section explaining chmod. Use letters not numbers

E.Linux.6

Write a section explaining chown. Use letters not numbers

E.Linux.7

Write a section explaining su and sudo

E.Linux.8

Write a section explaining cron, at, and crontab

2 - Perl One Liners

Manipulating files with convenient perl one line commands.

Learning Objectives

  • Familiarize yourself with manipulating files by using Perl one line commands.

Topics covered

Perl is a programming language that used to be very popular with system administrators. It predates Python. It has some very powerful regular expression abilities allowing you to easily do things on the commandline that woul otherwise thake many hours. Here ar some useful perl one line commands.

Strip trailing whitespace from a file

$ perl -lpe 's/\s*$//' FILENAME

Replace wrong quote

$ perl -i -p -e "s/’/'/g;"  *.md

Remove ^M from file

$ perl -p -i -e 's/\r\n$/\n/g'

3 - ZSH

Introduction to using zsh.

Learning Objectives

  • Get to know about zsh
  • Use it if you must. It is default on macOS, but not the PI

You may use zsh on the PI, but we find it better to just use a default shell such as bash. The reaon we have included it is that it became default on macOS and many are accessing the Pis from the Mac.

Topics covered

Introduction

Z shell (zsh) is an alternative to bash. It is used as an interactive shell or command interpreter. Zsh has been chosen by apple as a replacement for bash. A large number of plugins for zsh is avalable at the Web site Oh My Zsh.

Features of zsh include:

  • commandline completion
  • global history that can be shared in shells
  • build in file globing
  • multiline commands
  • spell correction
  • compatibillity modes to impersonate other shells
  • themes for prompts
  • in addition to which a wher ecommand
  • shortcut to names directories with ~

In principal it does not matter much whcih shell you use as long as you use to set up your envireonment properly. While bash zupporst .bash_profile and .bashrc, zsh supports ~/.zprofile and ~/.zshrc

A good overview of the loading process is documented at

Setting up zsh on an older OSX is relatively simple.

$ brew install zsh

To add Oh My Zsh you can do:

$ sh -c "$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

To chnge the default shell to zsh you can than execute

$ chsh -s $(which zsh)

To activate the shell you can as usal do

$ source ~/.zshrc

However if you start a new terminal, you do not have to do this as it is added automatically.

To use a number of useful plugins, you can activate them with

$ plugins=(osx git colored-man colorize pip python brew zsh-syntax-highlighting zsh-autosuggestions)

If you like to change the theme you can find a large number at

Using zsh on your operating systems

For other operationg systems see

zsh on Windows 10

To install zsh on Windows 10, please look at