The Makers software development program I attended was prefaced by a pre-course, which went into some of basics that are required to navigate the terminal comfortably. There was game I’ve since gone a little further, though I decided to include some of the most useful commands here, as I still use them very frequently.

Pipes and Redirections

Pipes allow us to redirect streams. Streams are how a program sends outputs and receives inputs.

EG:

cat file.txt | less<1>
ls -lA | less<2>
cat file.txt > newCombined.txt<3>

1. This will make “less” take the input from the input stream. The output of cat file.txt will not be shown on the screen at all since we redirected the output stream to the input stream of less.
2. If we had lots of files in directory and wanted to be able to scroll and up down instead of just printing thousands of lines on the screen. We can redirect the output in this way.
3. We can also redirect the output stream of one program to the input stream of another one, if the file on the right doesn’t exist it will create one for us.

Wildcards

We can use this to search for files that match certain criteria.

ls *.txt
ls new*.txt ls * ls *n*
find . -name "*.txt" -print
cd ~ find . -name "*.txt" -print
cd ~/Downloads find . -name "*.pdf" -print > myPdfs.txt
grep binary combined
find ~ -name "*.txt" -print | grep README

Ag

As an improvement to using wildcards, we also have alternative tools such as The Silver Searcher, otherwise known as ag that searches for certain bits of data contained in files. It returns all the files and the lines on which the matching data can be found.

Regular Expressions

find ~/Documents -name "*" -print | grep "\d\+"
find ~ -name "*" -printf "%f\n" | grep "[0-9]"

Counting words (intermediate commands)

wc longText.txt
find ~ -name "*.txt" -print | grep README | wc -l

Permissions

.To give a user permission to EXECUTE a file:

chmod +x filname.rb

Shebang

A combination of a hash and an exclamation mark. If we wanted to be able run a ruby file without having to put “ruby” as the executable in front of the file, we might add this to the top of that file:

#!/usr/bin/env ruby

Superuser mode

To give some saftey against executing commands that may be considered dangerous, such as:

`rm -rf /*`<1>

1. Deletes all files in the root

It’s not considered good practice to work as root on a permanent basis. Instead we should only switch into superuser mode when necessary.

To execute a command as a super-user we can prefix it with sudo command. For instance, if we want to delete a file we haven’t got permissions to delete, procided we know the superuser password we can do:

sudo rm special_file_to_be_deleted

We’d then be prompted for the password.

Environement

We can view any single environment variable by typing ‘echo $ENV_VAR’, e.g.:

echo $HOME 
echo $USER
echo $PATH

Echo

We can print things to the terminal.

echo "Hello World" 

This might seem quite useless, since we wouldn’t use this in a program, although we might like to use it to check environement variables, as we did previously.

We can also use it however to save short strings to a file. For example:

echo "Save this string somewhere" > savedstring.txt

PATH

echo $PATH

Setting environement variables

We know how to read environment variables, but we can also set them:

export MOOD=good

We created a new environement variable called MOOD. We can now read back its value:

echo $MOOD

Why would we want to use environement variables?

One common reason is for storing sensitive data (like passwords)

For open source for instance, we can put the secret key in an environment variable, and then share that code but still read it locally using our program, for instance:

export SECRET_KEY=password_for_a_program12345

And from within our ruby code:

secret_key = ENV['SECRET_KEY']

A place where I have used this technique is when trying to access an API key (if it is shared the daily quota of API hits risks being hit), the code for the DataPicker module of a google API program I have is over here.

Profile files

Environment variables only last until the end of the current shell session. So if we export MOOD=good in the terminal, when we open a new one it should no longer be there if we check with echo $MOOD. However, if we want an environment variable to persist and be available in all new shell sessions as well, we can use Procfile files to do this.

The following example is for zsh. In order to have an environment variable persist across different terminal sessions, We can create it permanently inside the .zshrc file in root:

echo "export MOOD=good" >> ~/.zshrc <1>

1. The double angle brakcet ‘»’ means append; ‘>’ means overwrite.

We could also put it in manually, by simply going into the file with vi .zshrc.

The zshrc file will now look like this:

[...]
export PATH="$PATH:$HOME/.rvm/bin"
export MOOD=good

Other commands

The tools that follow are ones that we would like to know exist. They’re good examples of the UNIX philosophy:

“A tool in Unix should do one thing, and one thing well.”

We might never use them, but they’re nice tools for us to know nevertheless.

1. diff

(More to do with versioning). We can diff between files, repos; and they dont need to be inside the same for us to a diff.

EXAMPLE: diff code_file.rb ../similar_code_file.rb

Side Note: When we do git diff we’re actually doing git + -diff / git-diff. Same thing goes for log, git-log

2. grep

If looking for a particular word in a file, we could use grep to see whether it is contained inside the file:

grep -i something filename

the -i flag signifies case insensitive, otherwise a return will only match if it respects the case of the word/word.

If we want to ge the specific line on which it is, we do:

grep -n something filename

3. awk

Is similar to grep, ag - both of which are more or less identical, except that ag (from silversurfer) is faster, though one step above them in that you can perform actions on the things that you are searching for. awk can be used as a stream editor.

“Awk is mostly used for pattern scanning and processing. It searches one or more files to see if they contain lines that matches with the specified patterns and then perform the associated actions. Awk is abbreviated from the names of the developers”

4. cut

If I need to make a table of something that comes in as a string, they’re not tabular, but they’re table like because they’re more CSV than they are a table. I’ve got four headers. It’s more like a trick. Not used very often.

5. sed

Stream editor, when you want to edit multiple files with a similar “macro”. The key is that you’re editing them in a stream. It’s a little bit like a recording in vim.