LinuxBasics

Disclaimer: This is a very large topic, and much too broad to be covered on a single support page. There are many other sites (yes, entire sites) which cover the topic in more detail. We'll link so some of them below. This page is meant to be just the essentials.

Logging in for the first time

To login to Beocat, you first need an "SSH Client". SSH (short for "secure shell") is a protocol that allows secure communication between two computers. We recommend the following.

• Windows
  • PuTTY is by far the most common SSH client, both for Beocat and in the world.
  • MobaXterm is a fairly new client with some nice features, such as being able to SCP/SFTP (see below), and running X (which isn't terribly useful on Beocat, but might be if you connect to other Linux hosts).
  • Cygwin is for those that would rather be running Linux but are stuck on Windows. It's purely a text interface.
• Macintosh
  • OS-X has SSH a built-in application called "Terminal". It's not great, but it will work for most Beocat users.
  • iTerm2 is the terminal application we prefer.
• Others
  • There are many SSH clients for many different platforms available. While we don't have experience with many of these, any should be sufficient for access to Beocat.

You'll need to connect your client (via the SSH protocol, if your client allows multiple protocols) to headnode.beocat.ksu.edu.

For command-line tools, the command to connect is

ssh username@headnode.beocat.ksu.edu

Your username is your K-State eID name and the password is your eID password.

Note: When you type your password, nothing shows up on the screen, not even asterisks.

You'll know you are successfully logged in when you see a prompt that says

[username@machinename ~]$

where machinename is the name of the machine you've logged into (currently either 'clymene' or 'helios') and username is your eID username

Transferring files (SCP or SFTP)

Usually, one of the first things people want to do is to transfer files into or out of Beocat. To do so, you need to use SCP (secure copy) or SFTP (SSH FTP or Secure FTP). Again, there are multiple programs that do this.

• Windows
  • Putty (see above) has PSCP and PSFTP programs (both are included if you run the installer). It is a command-line interface (CLI) rather than a graphical user interface (GUI).
  • MobaXterm (see above) has a built-in GUI SFTP client that automatically changes the directories as you change them in your SSH session.
  • FileZilla (client) has an easy-to-use GUI. Be sure to use 'SFTP' mode rather than 'FTP' mode.
  • WinSCP is another easy-to-use GUI.
  • Cygwin (see above) has CLI scp and sftp programs.
• Macintosh
  • FileZilla is also available for OS-X.
  • Within terminal or iTerm, you can use the 'scp' or 'sftp' programs.
• Linux
  • FileZilla also has a GUI linux version, in addition to the CLI tools.

Using a Command-Line Interface (CLI)

You can safely ignore this section if you're using a graphical interface (GUI). We highly recommend using a GUI when first learning how to use Beocat.

First test case: transfer a file called myfile.txt in your current folder to your home directory on Beocat. For these examples, I use bold text to show what you type and plain text to show Beocat's response

Using SCP:

scp myfile.txt username@headnode.beocat.ksu.edu:
Password: (type your password here, it will not show any response on the screen)
myfile.txt                                                                            100%    0     0.0KB/s   00:00

Note the colon at the end of the 'scp' line.

Using SFTP

sftp username@headnode.beocat.ksu.edu
Password: (type your password here, it will not show any response on the screen)
Connected to headnode.beocat.ksu.edu.
sftp> put myfile.txt
Uploading myfile.txt to /homes/kylehutson/myfile.txt
myfile.txt                                                                            100%    0     0.0KB/s   00:00
sftp> exit

SFTP is interactive, so this is a two-step process. First, you connect to Beocat, then you transfer the file. As long as the system gives the sftp> prompt, you are in the sftp program, and you will remain there until you type 'exit'.

Second test case: transfer a file called myfile.txt in your current folder to a diretory named 'mydirectory' under your home directory on Beocat.

Here we run into one of the problems with scp - there is no easy way of creating 'mydirectory' if it doesn't already exist. If it does not already exist, you must login via ssh (as seen above) and create the directory using the 'mkdir' command (see Common Linux Commands) below.

scp myfile.txt username@headnode.beocat.ksu.edu:mydirectory
Password: (type your password here, it will not show any response on the screen)
myfile.txt                                                                            100%    0     0.0KB/s   00:00

An alternative version. If the colon is immediately followed by a slash, the directory name is taken from the root, rather than your home directory. So, given that your home directory on Beocat is /homes/username, we could instead type

scp myfile.txt username@headnode.beocat.ksu.edu:/homes/username/mydirectory
Password: (type your password here, it will not show any response on the screen)
myfile.txt                                                                            100%    0     0.0KB/s   00:00

Using SFTP:

sftp username@headnode.beocat.ksu.edu
Password: (type your password here, it will not show any response on the screen)
Connected to headnode.beocat.ksu.edu.
sftp> mkdir mydirectory
[Note, if this directory already exists, you will get the response "Couldn't create directory: Failure"]
sftp> cd mydirectory
sftp> put myfile.txt
Uploading myfile.txt to /homes/username/mydirectory/myfile.txt
myfile.txt                                                                            100%    0     0.0KB/s   00:00
sftp> quit

Third test case: copy myfile.txt from your home directory on Beocat to your current folder.

Using scp:

scp username@headnode.beocat.ksu.edu:myfile.txt .
Password: (type your password here, it will not show any response on the screen)
myfile.txt                                                                            100%    0     0.0KB/s   00:00

Using SFTP:

sftp username@headnode.beocat.ksu.edu
Password: (type your password here, it will not show any response on the screen)
Connected toheadnode.beocat.ksu.edu.
sftp> get myfile.txt
Fetching /homes/username/myfile.txt to myfile.txt
myfile.txt                                                                            100%    0     0.0KB/s   00:00
sftp> exit

Basic Linux Commands

Again, this guide is very limited, mostly limited to directory navigation and basic file commands. Here is a pretty decent tutorial if you want to dig deeper. If you want more, entire books have been written on the subject.

The Lingo

Term	Definition
Directory	A "Folder" in Windows or OS-X terms. A location where files or other directories are stored. The current directory is sometimes represented as `.` and the parent directory can be referenced as `..`
Shell	The interface or environment under which you can run commands. There is a section below on shells
SSH	Secure Shell. A protocol that encrypts data and can give access to another system, usually by a username and password
SCP	Secure Copy. Copying to or from a remote system using part of SSH
path	The list of directories which are searched when you type the name of a program. There is a section below on this
ownership	Every file and directory has an user and a group attached to it, called its owners. These affect permissions.
permissions	The ability to read, write, and/or execute a file. Permissions are based on ownership
switches	Modifiers to a command-line program, usually in the form of -(letter) or --``(word). Several examples are given below, such as the '-a' on the 'ls' command
pipes and redirects	Changes the input (often called 'stdin') and/or output (often called stdout) to a program or a file

Linux Command Line Cheat Sheet

File System Navigation

Command	What it does	Example Usage	Example Output
pwd	"Print working directory", Where am I now?	pwd	/homes/mozes
ls	Lists files and folders	ls ~/	NewFile NewFolder
ls -lh	Lists files and folders with perms size and ownership	ls -lh ~/	-rw-r--r-- 1 mozes mozes_users 1 Jul 13 2011 NewFile drwxr-xr-x 9 mozes mozes_users 9.0K Apr 12 2010 NewFolder
ls -a	Lists all files and folders	ls -a ~/	. .. .bashrc .bash_profile .tcshrc NewFile NewFolder
cd	Changes directory	cd NewFolder
cd ..	Changes to parent directory	cd ..
cd -	Changes to the previous directory you were in	cd -
cd ~	Changes to your home directory	cd ~

Working with files

Command'	What it does	Example Usage'	Example Output
file	Identifies the type of object a file is	file NewFile	NewFile: a /usr/bin/python script, ASCII text executable
cat	Prints the contents of one or more files	cat NewFile	This is line one This is line two
cp	copy a file	cp OldFile NewFile
cp -i	copy a file, ask to overwrite	cp -i OldFile NewFile}	overwrite NewFile? (y/n [n])
cp -r	copy a directory, including contents	cp -r OldFolder NewFolder
mv	move, or rename, a file	mv OldFile NewFile
mv -i	move, or rename, a file, ask to overwrite	mv -i OldFile NewFile	overwrite NewFile? (y/n [n])
rm	remove a file	rm NewFile
rm -i	remove a file, ask to be sure (useful with -r)	rm -i NewFile	remove NewFile? (y/n [n])
rm -r	remove a direcory and its contents	rm -r NewFolder
mkdir	creates a directory	mkdir TempFolder
rmdir	removes an empty directory	rmdir TempFolder
touch	creates an empty file	touch TempFile

Finding files and directories with find

Command	What it does	Example Usage
find <directory>	finds all files and folders within <directory>	find ~/
find <directory> -iname '<filename>'	finds all files and directories within <directory> that match <filename>	find ~/ -iname 'hello.qsub'
find <directory> -iname '*<partialmatch>*'	finds all files and directories within <directory> that partially match <partialmatch>	find ~/ -iname '*.qsub*'

Other useful commands include htop, less, and man. man followed by a command name above will give you the manual page for the specified command full of many other useful options for the command. htop will give you an overview of the processes currently being run on the host you are connected to. less allows you to page through files and see their contents using <PgUp> and <PgDn>.

Editing Text Files

If you're new to Linux, the editor you will probably want to use is 'nano'. It works much the same as 'Notepad' in Windows or 'textedit' on OS-X. Note that you cannot use your mouse to change position within the document as you can with your local computer. You must use the arrow keys, instead.

So, if I wanted to edit my .bashrc (as shown below), and I was already in my home directory (see above), I would type

nano .bashrc

While in nano, there is a list of actions you can take at the bottom of the screen. <Ctrl> is represented by a caret (`^`), so to exit (labeled as `^`X at the bottom of the screen), I would type <ctrl>-x. This action prompts you whether you want to save and exit (Y), lose changes and exit (N), or cancel and go back to editing (<ctrl>-c).

If you do a significant amount of text editing in Linux, you'll probably want to switch to a more powerful editor, such as vim. The usage of vim is beyond the scope of this document. It is not at all intuitive to the beginning user, but with a little practice it becomes a much faster way of editing text files. If you're interested in using vim, there is a nice tutorial here.

Shells

What is a Shell?

In this case, I don't believe I can do a better job explaining shells than this.

tcsh

Elsewhere at Kansas State University, the default Shell is set to tcsh. tcsh stands for "TENEX C SHell." It is considered a replacement for csh and uses many of the same features. If you have experience with either csh or tcsh you'll probably feel right at home. This was the default shell until July of 2013. If you had an account before then, it is probably still tcsh.

But what if you don't want or like tcsh, what can you do? Well, we have other shells available of Beocat as well.

bash

Bash seems to be the defacto standard shell in most Linux installs today. Bash is common and probably what most of you are used to. As of July 2013, bash is our new default shell. All new users will be set to bash initially. Software Carpentry teaches classes on several subjects specifically targeting researchers, including the bash shell. Their documentation is all freely available. Here is a link to their excellent tutorial on using BASH. Most of our documentation assumes you are using BASH.

bash configuration files:

This section gets into some minutiae with the way our job scheduler interacts with bash. If you're trying to solve a problem, read on, otherwise you can probably skip this section.

Bash has 3 user configurable configuration files, ~/.bashrc ~/.bash_profile and ~/.bash_logout. We'll look at the two more relevant ones ~/.bashrc and ~/.bash_profile

Bash has 3 ways of looking at things, login, interactive, or none.

Normally what happens is that shells that are interactive read ~/.bash_profile, shells that are login read ~/.bashrc. none shells read neither.

sbatch jobs are login, srun jobs are login+interactive, logging into Beocat in a way that you can enter commands is login+interactive. There are very few cases that you will get none. For any session that isn't interactive, your sourced files cannot output anything to the screen, or else it can break scp or sftp file transfers.

If they are quiet statements, and you want them in all shells, you can put them in your ~/.bashrc. If they are not quiet or they output anything to the screen, you must put them in your ~/.bash_profile.

zsh

zsh is an alternative to bash and tcsh. It tends to support more complex features than either of the other two while using a syntax remarkably similar to bash. Unless specifically noted, when we specify Change your shell to bash, zsh should work as well.

Changing Shells

Previously, we gave you the option of using a ~/.login to modify your shell. This is no longer supported, if you have issues with your shell/paths/environment variables we will ask you to delete your ~/.login file and change your shell via the method below.

You can change your shell is via chsh on either of the headnodes (clymene/helios). This does not need to be re-done if you've changed to it to your preferred shell in the past.

Use the appropriate of the following three lines:

/usr/local/bin/chsh -s bash && bash -l
/usr/local/bin/chsh -s tcsh && tcsh -l
/usr/local/bin/chsh -s zsh && zsh -l

Changing your PATH

Typically, you don't have to change your PATH, but it is useful to know what your PATH is and what it does. The PATH is the list of directories which are searched when you type the name of a program. Note that by default the current directory is NOT included in the path, so if you were wanting to run a program called MyProgram in the current directory, you could NOT simply type 'MyProgram', you would instead type './MyProgram' (where the '.' represents the current directory).

To find your PATH, we need to identify which shell you are using. If you do not know, run the following:

ps | awk '/sh/ {print $4}'

tcsh

You'll need to edit a file in your home directory called .tcshrc, replacing /usr/local/bin with the directory that you want added to your PATH using a text editor as shown above.

setenv PATH /usr/local/bin:$PATH

bash

You'll need to edit a file in your home directory called .bashrc, replacing /usr/local/bin with the directory that you want added to your PATH using a text editor as shown above.

export PATH=/usr/local/bin:$PATH

zsh

You'll need to edit a file in your home directory called .zshrc, replacing /usr/local/bin with the directory that you want added to your PATH using a text editor as shown above.

export PATH="/usr/local/bin:$PATH"

Ownership and Permissions

Every file and directory has a user and group associated with it. You can view ownership information by using the '-l' switch on ls. By default on Beocat, files you create have a user ownership of your username (i.e., your eID) and a group ownership of your username_users. So, if I were logged in as 'myusername' and I had a single file in my home directory called MyProgram, the result of typing 'ls -l' would be something like this:

total 0
-rwxr-x--- 1 myusername myusername_users 79 May 31  2011 MyProgram

This tells us several things.

• The first column ('-rwxr-x---') is permissions (covered below)
• The second column ('1') is the number of links to this file. You can safely ignore this (unless you're both masochistic and interested in filesystem details)
• The third column ('myusername') shows the user ownership
• The fourth column ('myusername_users') shows the group ownership
• The fifth column ('79') gives the size of the file in bytes
• The next columns ('May 31 2011'), as you have probably guessed, gives the date the file was last changed
• The final column ('MyProgram') is the name of the file

So why is this interesting to us? Because whenever things don't work, it's usually because of file ownership or permissions. Looking at these often gives us some useful diagnostic information.

The permissions field shows us who has permissions to do what with this file. It is always 10 characters. The first character (-) is usually either a '-' for a regular file or a 'd' for a directory. The next 9 characters are broken into three groups of three, with each group showing read (r), write (w), and execute (x) permissions for the owner, group, and world, in that order.

• The first group (rwx) shows permissions for the owner (myusername). The owner here has read, write, and execute permissions
• The next group (r-x) shows permissions for the group (myusername_users). The group here has read and execute permissions, but cannot write.
• The last group (---) shows permissions for the rest of the world. The world has no permissions to read, write, or execute.

When you create a shell script with a text editor, and sometimes when you copy programs to Beocat via SCP, the execute flag is not set. The permissions string may look more like (-rw-r--r--). To change this, you need to give yourself permission to execute this program. This is done with the 'chmod' (change mode) command. 'chmod' can have a long and confusing syntax, but since by far the most common problem is to give yourself execute permissions, here is the command to change that:

chmod u+x MyProgram

This changes the permissions so that the user ('u', i.e., the owner) adds ('+') execute permission ('x').

For more complex ownership or permissions changes, please feel free to contact the Beocat staff.

Access Control Lists

Access Control Lists build on our knowledge and use of basic Linux permissions, so we'll cover those again:

Linux permissions are typically broken down to (r)ead, (w)rite, and e(x)ecute split across 3 classes of accessors.

Files

read

Read the file, pretty straight forward

write

Write to the file, including overwrite, truncation, etc.

execute

Execute the file, this permission allows you to run the file.

Folders

read

List the directory, (ls)

write

Create new files and folders in the directory.

execute

Pass through the directory (cd into and through).

Those accessors are (u)ser, (g)roup, and (o)ther.

user

The user would typically be the user account that created the file or folder

group

The group would be that accounts primary group by default or can be changed by the user to be any group that they are a member of

other

Other is special, other is anything that doesn't meet either of the two other critera. We typically refer to them as world permissions, as they match everyone else.

Unfortunately, it is that "Other" permission that is a frequent problem. You may want to share some data with a colleague, but, from a security standpoint, you also may need to make sure that only that colleague has access to the data. If you aren't in the same group as the colleague, then, under standard Linux permissions, you have no other option except making the file "world" accessible.

This is where ACLs come into play. ACLs are like the standard Linux permissions, except you can apply many of them, and you can allow individual users and groups to access alongside your own.

ACLs can also do things that standard Linux permissions can't, like setting up "default" permissions for newly created files/folders within a directory.

One big thing to be aware of for any permissions scheme is that permissions are checked at every level in a directory hierarchy.

1. /
2. /homes
3. /homes/$USER
4. /homes/$USER/$SHARE

If at any point the accessing user is denied permission, the traversal and access attempt will stop.

Example 1

Let's say I have a file in a directory that I want the user billy to be able to read. This file is /homes/mozes/example/input.file. We'll look at the current permissions of the directory tree like so:

We'll assume everyone has requisite permissions for / and /homes.

First we'll check my home directory

$ getfacl -e /homes/mozes
# owner: mozes
# group: mozes_users
user::rwx
group::r-x                      #effective:r-x
group:beocat_support:r-x        #effective:r-x
mask::r-x
other::---
default:user::rwx
default:group::r-x              #effective:r-x
default:group:beocat_support:r-x        #effective:r-x
default:mask::r-x
default:other::---

If we make it past that permissions check, we'd go one level deeper.

$ getfacl -e /homes/mozes/example
# owner: mozes
# group: mozes_users
user::rwx
group::r-x
group:beocat_support:r-x        #effective:r-x
other::r-x
default:user::rwx
default:group::r-x              #effective:r-x
default:group:beocat_support:r-x        #effective:r-x
default:mask::r-x

Finally we'd check if we had permission to access the file itself:

$ getfacl -e /homes/mozes/example/input.file
# owner: mozes
# group: mozes_users
user::rw-
group::r--
group:beocat_support:r-x        #effective:r-x
other::r--

There is quite a lot of information contained in the the above output, so lets look at and attempt to understand the contents.

First, in each section, we see the POSIX owner and group as comments prefixed by '#' characters. These are what the respective user:: and group:: lines refer to when viewing the permissions.

Second, we have lines related to the permissions of each accessor. These do what they say, show the permissions that an accessor would be granted. Please note there is a catch here, it seems to be most specific permission wins. This can come into play with granting a certain group access and then a specific member of the group a differing level of access.

Third, on many of them there are lines prefixed with default: and then a permissions set. Default permissions are interesting. They are only set to directories and they define the starting set of acls that should be set when new files or folders are created within that folder.

Finally, there is a mask, I'm not covering it because there are very few cases that people would need to use them.

Back to the task at hand, We want billy to be able to read /homes/mozes/example/input.file. Checking /homes/mozes, we see that 'other' has no permissions, and billy has not been granted any special access.

So we grant billy access "through" /homes/mozes, granting the smallest set of permissions would be this:

$ setfacl -m u:billy:x /homes/mozes

Note, since I didn't give billy read access to my home directory, they wouldn't be able to ls /homes/mozes, they can simply cd into it and through it.

Then we check the rest of the permissions, /homes/mozes/example has an 'other' permission granting (r)ead and e(x)ecute, so that shouldn't be an issue. /homes/mozes/example/input.file allows 'other' to read it, so our job is done. Billy has access to read my file.

If we decide later that billy needs to write to my file, we can grant them specific read/write permissions to just that file with:

$ setfacl -m u:billy:rw /homes/mozes/example/input.file

Example 2

That's all well and good, but lets say we want all of my grad students to have read/write access to my example directory.

Looking at the acls that have been set so far:

$ getfacl -e /homes/mozes
# owner: mozes
# group: mozes_users
user::rwx
user:billy:--x
group::r-x                      #effective:r-x
group:beocat_support:r-x        #effective:r-x
mask::r-x
other::---
default:user::rwx
default:group::r-x              #effective:r-x
default:group:beocat_support:r-x        #effective:r-x
default:mask::r-x
default:other::---

$ getfacl -e /homes/mozes/example
# owner: mozes
# group: mozes_users
user::rwx
group::r-x
group:beocat_support:r-x        #effective:r-x
other::r-x
default:user::rwx
default:group::r-x              #effective:r-x
default:group:beocat_support:r-x        #effective:r-x
default:mask::r-x

We now want to grant my group of grad students the correct permissions to /homes/mozes/example

$ setfacl -R -m g:my_grad_students:rw -m d:g:my_grad_students:rw -m d:u:mozes:rw /homes/mozes/example

There are a few things to note there:

• We're setting multiple acls at once (note the multiple -m arguments)
• We're setting those permissions recursively (on all files/folders nested anywhere in that directory hierarchy). The -R option does this.
• We're setting some default permissions. Default permissions are prefixed with d:. Here we're saying that the (g)roup my_grad_students should be granted read/write permissions, we aslo set a default permission for ourselves. d:u:mozes:rw grants me read/write access to those files as if I were the owner, this is nice in the event that you're not a member of the my_grad_students group, it would make sure that you still retain a reasonable baseline of access.

That all looks good, right? Except my grad students are complaining that they can't access /homes/mozes/example. What did we forget?

Permissions are checked at every level of the directory hierarchy, and we forgot to grant my_grad_students access through my home directory.

$ setfacl -m g:my_grad_students:x /homes/mozes

Manual (man) pages

Most commands have a complex set of switches that will modify the amount or type of information they display. To find out what switches are available, or how a program expects data, you can use the manual pages by typing "`man` command". Using one of the most common Linux commands, take a look the output of 'man ls'. It shows that it has over 50 switches available, ranging from which files to include, to how to display file sizes, to sort order and more. (I'm not pasting it here, because it's over 200 lines long!) To navigate a 'manpage', use the up-arrow and down-arrow keys. Press 'q' to quit.

Pipes and Redirects

Typically a Linux program takes data from the keyboard and outputs data to the screen. In Unix and Linux terminology, the keyboard is the default 'stdin' (pronounced "standard in") and the screen is the default 'stdout' (pronounced "standard out"). Many times, we want to take data from somewhere else (like a file, or the output of another program) and send it to yet another location. These redirectors are:

cmd > filename	Redirect output from cmd to filename
cmd >> filename	Redirect output from cmd and append to filename
cmd < filename	Redirect input from cmd to filename
cmd1 | cmd2	Use the output from cmd1 as the input to cmd2

Here is a quick example. Let's say I have a thousands of files in a directory, and I want a list of those that end in '.sh' 'ls' by itself scrolls so far I can't see even a fraction of them. So, I redirect the output to a file

ls > ~/filelist.txt

That gives me all the files in the current folder and saves them in my home directory in 'filelist.txt'. A quick look through the file in my favorite editor tells me this is still going to take too long, so I need another step. The 'grep' program is a commonly-used program to perform pattern matching. The syntax of 'grep' is beyond the scope of this document, but take my word for it that

grep '\.sh$'

will return all lines that end in .sh.

We can now redirect the input from grep to the file we just created:

grep '\.sh$' < ~/filelist.txt

Great! We now have our list. However, we wanted to save this as filelist.txt, and instead we have another list that we have to copy-and-paste. Instead of redirecting to a file, we'll use the vertical bar '|' (which we often term a "pipe") to send the output of one command to another.

ls | grep '\.sh$' > ~/filelist.txt

This time the output of 'ls' is not redirected to a file, but is redirected to the next command (grep). The output of grep (which is all our .sh files) instead of being sent to the screen is redirected to the file ~/filelist.txt.

This example is a very simple demonstration of how pipes and redirects work. Many more examples with complex structures can be found at http://www.ibm.com/developerworks/linux/library/l-lpic1-v3-103-4/index.html