Category: HPC

This page lists commonly used Linux commands to help those who are not very familiar with Linux command environment. I have been collecting and recording those from my experience.

I was once a beginner, so I can understand the pain for Linux beginners. I have not yet seen any post that has done comprehensive collection of commands on Linux, so I thought I could help this out. That is why you see this post. Here you go. Happy Learning!

You can see further reading list at the end of this post.

Note: Do not contain space in your filename or directory name, use underscore instead of space.

======Basic commands:

• cd 

this command will goes back to the home directory of your account, no matter where your current directory is located in your terminal.

• cd ../
  

this command will go to the parent directory of your current directory in terminal

• rmdir 

remove/delete an empty folder.

example:

first cd into the parent folder of the empty folder to remove

rmdir test

• rm -rf ./*

  first cd to a directory, and this command will empty all of things under the current directory

• ls  

list all the files and folders under current path

• ls -l  

list all the files and folders with details like dates.

• ls -l -t

list files and folders ordered by time.

• ls -ltrh 

list all the csv files under the current directory in long format by time and in reverse order, the file size in human readable format (e.g., in mb, or gb, instead of byte size)

• -l List in long format. If the output is to a terminal, a total sum for all the file sizes is output on a line before the long listing.
• -r Reverse the order of the sort to get reverse lexicographical order or the oldest entries first (or largest files last, if combined with sort by size.
• -t Sort by time modified (most recently modified first) before sorting the operands by lexicographical order.

• ls -ltrh *.csv

list all the csv files under the current directory in long format by time and in reverse order, the file size in human readable format (e.g., in mb, or gb, instead of byte size)

• find

find -name ‘*.jpg’ -exec cp {} ./test/ \;

Find all jpg files  and then copy the found files to the folder test which is subfolder of current path.
Note that: the current path should be the path where the files to search are located. (i.e., use cd to locate to the directory where the files are in before type in the cmd below into terminal.)

• rm -r -f

-r means recursive, it will remove folders and subfolders and files within the folders and subfolders

-f means force

• mkdir [folder name]

create new folder

example:

mkdir image

• cp [filename] [new filename]

copy and rename file

• cp [filename] [path/to/new/lotcation/filename]

copy the file to another location

if you use this commond to copy a directory, you would meet this error:

cp: omitting directory ...

The error notice means you told cp to copy files and not directories. The warning is about cp finding a directory and informing you it will be skipped.

• cp -r [directory] [path/to/new/location/directory]

copy a directory to another location.

cp -r means recursive and this option will make cp also include sub-directories.

If you meet permission denied error, add sudo before the command, and it will ask your password.

• mv [directory] [path/to/new/location/directory]

If you meet permission denied error, add sudo before the command, and it will ask your password.

• nano [new file name or /path/to/new file/new file name]

example:

nano  myexample   #it will create a new empty file named “myexample” under the current directory

• nano [file name]

If the file name already exists, it will open the file and you can edit it.

Note: Ctrl+O to save the file, and then hit Enter, and then Ctrl +X to close the file.

 

 

======More advanced commands:

• cd into directory without having permission

When cd into a directory and the following error occurs

bash: cd: your-dirctory: Permission denied

The solution is:

Enter super user mode, and cd into the directory that you are not permissioned to go into. Sudo requires administrator password.

sudo su  
cd directory  # you will notice that your prompt changes after your enter your root password. now you can cd to the directory.

# to exit "super user" mode, type exit.

• lspci

check GPU information on Ubuntu

look for “VGA compatible controller:”…

• sudo nvidia-smi

check GPU info and GPU usage.

• sudo reboot -h now

reboot a server from terminal

• sudo shutdown -h now

shut down a server from terminal

Note: If your Ubuntu Server 16.o4 LTS has Black Screen after reboot or shut down, try pressing (simultaneously) Ctrl + Alt + F2 to see whether you can switch to different console

• vncserver -kill :1  

This is a vncserver command. It is used to kill a port of a GUI by VNC server, where 1 is the port you would like to kill.

• vncviewer -via username@yourserver_hostname :1

connect to a server via vncviewer from a linux-based client. you need to change the port number 1 to yours.

• echo

echo is a built-in command in the bash and C shells that writes its arguments to standard output.

See here, and here, and herefor example usage of it.

• cat

See here for example usage of cat command.

• chmod

see here for example usage of chmod command.

 

• check supercomputing Cluster’s Linux distribution and version

$ lsb_release -a
LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID:    RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 6.4 (Santiago)
Release:    6.4
Codename:    Santiago

• show the list of top processes ordered by RAM and CPU  use in descendant form

(remove the pipeline and head if you want to see the full list):

$ ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head

Brief explanation of the options used in the command above:

— The -o (or –format) option of ps allows us to specify the output format.

— the processes’ PIDs (pid), PPIDs (pid)

— the name of the executable file associated with the process (cmd), and

— the RAM and CPU utilization (%mem and %cpu, respectively).

We can use --sort to sort by either %mem or %cpu. By default, the output will be sorted in ascendant form, but usually we prefer to reverse that order by adding a minus sign in front of the sort criteria to make it list in descendant.

To add other fields to the output, or change the sort criteria, refer to the OUTPUT FORMAT CONTROL section in the man page of ps command.

======File Transfer: getting files to/from your account on a server

• On Linux generally the command line scp command.

scp -p file_name username@yourserver_hostname:destination/directory

• or for a full directory tree:

scp -pr dir_name username@yourserver_hostname:destination/directory

 

Note that if you want to transfer files from server to your client computer, just reverse the directory.

e.g., scp -pr username@yourserver_hostname:source/directory dir_name_on_your_client

 

======download files

• wget (tool for downloading files)  (pdf)
• See Linux wget command (pdf), which provides detailed and comprehensive different tags (options) to use with wget command.

======Save terminal output to a file

• sudo command -option | tee logThis command will show output on terminal and save to a file at the same time.
• Save terminal output to a fileredirect the output to a file: someCommand > someFile.txt Or if you want to append data: someCommand >> someFile.txt If you want stderr too use this: someCommand &> someFile.txt or this to append:  someCommand &>> someFile.txt
• Tail -f log.txt

Python related commands:

• enter python environment

type python in terminal, and it will show python 2.7 version info and also enter into python 2 environment

• enter python 3 environment

python3 

and it will show python 3 version info (e.g., python 3.5.2) and also enter into python 3 environment.

 

======Git related commands

• git clone the url to gitclone

for example:

first cd into the folder you want the models to be cloned to in your terminal, and then issue this command. it will clone the model foder from https://github.com/tensorflow/models under your current folder in your terminal.

git clone https://github.com/tensorflow/models

 

======Some useful shortcuts on linux

• you can open multiple terminals

open each terminal by pressing Ctrl + Alt + T.

• Shortcut to bring all open terminals to the front

After you bring one terminal window in the front, press Alt+~ to bring all other terminal windows in the front one by one:

• CTRL + C − terminate the current command. 

======References and further reading list:

• The 5-Minute Essential Shell Tutorial (pdf)

• Linux Commands (pdf) – it is a great post, has illustration examples.
• TOP MOST USED & COMMON SSH COMMANDS.
• An Introduction to Linux (Cornell Virtual Workshop, See here for all other topics from Cornell Virtual Workshop)
  This tutorial is for the beginning Linux user, intended to get the user acquainted with some of the basic principles of the Linux operating system.
• the basics of using a terminal in Linux.
• TOP MOST USED & COMMON SSH COMMANDS
• http://www.tecmint.com/echo-command-in-linux/ (see here for a pdf copy)
• http://www.linfo.org/echo.html (see herefor a pdf copy)
• http://www.computerhope.com/unix/uecho.htm  (see herefor a pdf copy)
• http://www.computerhope.com/unix/ucat.htm (see herefor a pdf copy)

• Unix ABCs
• Linux ABCs
• MS-DOS vs. Linux and Unix
• Unix and Linux commands
• Unix Top 10 commands
• Unix Shortcuts
• Unix and Linux history
• Unix and Linux tips
• Unix and Linux links
• Linux Q&A

  Linux and Unix top 10 commands (See here for links to more commands intro)

  Below is a listing of the top 10 Linux and Unix commands most commonly used and that you will most likely use during a normal Linux or Unix session.

  1. Linux and Unix cd command help and Linux and Unix chdir command help
  2. Linux and Unix ls command help
  3. Linux and Unix cp command help
  4. Linux and Unix mv command help
  5. Linux and Unix rm command help
  6. Linux and Unix chmod command help
  7. Linux and Unix pico command help and Linux and Unix ed command help
  8. Linux and Unix clear command help
  9. Linux and Unix pwd command help
  10. Linux and Unix pine command help

Linux and Unix top 10 command pages  (See here for links to more commands intro)

Below is a listing of the top 10 Unix command pages by the amount of times they have been accessed on the Computer Hope server.

1. Linux and Unix tar command help
2. Linux and Unix chmod command help
3. Linux and Unix ls command help
4. Linux and Unix find command help
5. Information about the Linux and Unix grep command
6. Linux and Unix cp command help
7. Linux and Unix vi command help
8. Linux and Unix ifconfig command help
9. Linux and Unix date command help
10. Linux and Unix kill command help

======TOC of the nice tutorial: Linux Shell Commands: A Tutorial Quick Reference for Desktop Users

Table of Contents

======The end of the TOC of the nice tutorial: Linux Shell Commands: A Tutorial Quick Reference for Desktop Users

======apt-get usages

• How To Manage Packages In Ubuntu and Debian With Apt-Get & Apt-Cache (pdf)

======curl command examples

cURL can be used in many different and useful ways. Using cURL, we can download, upload and manage files, check email address, or even update status on some of the social media websites or check the weather outside.

cURL is very useful command line tool to transfer data from / to a server. cURL supports various protocols, including FILE, HTTP, HTTPS, IMAP, IMAPS, LDAP, DICT, LDAPS, TELNET, FTP, FTPS, GOPHER, RTMP, RTSP, SCP, SFTP, POP3, POP3S, SMB, SMBS, SMTP, SMTPS, and TFTP.

• 5 basic cURL command examples (OCTOBER 21, 2016) (pdf)

This tutorial covers five of the most useful and basic uses of cURL tool:

–Check URL

One of the most common and simplest uses of cURL is typing the command itself, followed by the URL you want to check

curl https://example.com
#This command will display the content of the URL on your terminal

–Save the output of the URL to a file

The output of the cURL command can be easily saved to a file by adding the -o option to the command, as shown below

curl -o website https://example.com
#the output will be save to a file named ‘website’ in the current working directory

–Download files with cURL

curl -O https://example.com/file.zip

# the -O option used for saving files to current working directory without renaming
# e.g.,  the ‘file.zip’ zip archive will be downloaded to the current working directory.

curl -o archive.zip https://domain.com/file.zip

# the ‘file.zip’ archive will be downloaded and saved as ‘archive.zip’.

curl -O https://domain.com/file.zip -O https://domain.com/file2.zip

# cURL can be also used to download multiple files simultaneously

#cURL can be also used to download files securely via SSH

curl -u user sftp://server.domain.com/path/to/file

# Note that the full path of the file to be downloaded is required

–Get HTTP header information from a website

You can easily get HTTP header information from any website you want by adding the -I option (capital ‘i’) to cURL.

curl -I http://example.com

–Access an FTP server

#  access your FTP server with cURL 
curl ftp://ftp.domain.com --user username:password

# cURL will connect to the FTP server and list all files and directories in user’s home directory

curl ftp://ftp.domain.com/file.zip --user username:password
# download a file via FTP using curl

curl -T file.zip ftp://ftp.domain.com/ --user username:password
# upload a file to  the FTP server

–check cURL manual page to see all available cURL options and functionalities

man curl

• Linux curl command (04/26/2017) – pdf

This post covers detailed and comprehensive explanation of different options to use with curl command.

• 5 commands to check memory usage on Linux (pdf)

• $ free -m

Linux has the habit of caching lots of things for faster performance, so that memory can be freed and used if needed.

• $ cat /proc/meminfo
• $ vmstat -s

• How to Clear RAM Memory Cache, Buffer and Swap Space on Linux