Write and run a bash file

This post introduces how to writ and run a bash file from terminal on Ubuntu. (If you prefer video style tutorials, check here for a post that is video based.)

The bash ( Bourne Again Shell) is the most common shell installed with Linux distributions and Mac OS.

  • Write a bash file

the most common is to write a file, make sure the first line is

#!/bin/bash

Then save the file. Next mark it executable using chmod +x file

Then when you click (or run the file from the terminal) the commands will be executed. By convention these files usually have no extension, however you can make then end in .sh or any other way.

For example,

#!/bin/bash          
echo Hello World

A Simple Bash Example

#!/bin/bash  
echo "This is a shell script"  
ls -lah  
echo "I am done running ls"  
SOMEVAR='text stuff'  
echo "$SOMEVAR"  
  • Run a bash file

go to the folder where you bash file is located and type:

./yourbashfile.sh

./ just means that you should call the script located in the current directory. (Alternatively, just type the full path of the yourbashfile.sh). If it doesn’t work then, check if yourbashfile.sh has execute permissions.

You can add execute permission by the following command:

$ chmod +x yourbashfile.sh

References:

Why Bother?

Why do you need to learn the command line anyway? Well, let me tell you a story. A few years ago we had a problem where I used to work. There was a shared drive on one of our file servers that kept getting full. I won’t mention that this legacy operating system did not support user quotas; that’s another story. But the server kept getting full and it stopped people from working. One of our software engineers spent the better part of a day writing a C++ program that would look through all the user’s directories and add up the space they were using and make a listing of the results. Since I was forced to use the legacy OS while I was on the job, I installed a Linux-like command line environment for it. When I heard about the problem, I realized I could do all the work this engineer had done with this single line:

du -s * | sort -nr > $HOME/user_space_report.txt

Graphical user interfaces (GUIs) are helpful for many tasks, but they are not good for all tasks. I have long felt that most computers today are not powered by electricity. They instead seem to be powered by the “pumping” motion of the mouse! Computers were supposed to free us from manual labor, but how many times have you performed some task you felt sure the computer should be able to do but you ended up doing the work yourself by tediously working the mouse? Pointing and clicking, pointing and clicking.

I once heard an author say that when you are a child you use a computer by looking at the pictures. When you grow up, you learn to read and write. Welcome to Computer Literacy 101. Now let’s get to work.

Contents

  1. What Is “The Shell”?
  2. Navigation
  3. Looking Around
  4. A Guided Tour
  5. Manipulating Files
  6. Working With Commands
  7. I/O Redirection
  8. Expansion
  9. Permissions
  10. Job Control

Here Is Where The Fun Begins

With the thousands of commands available for the command line user, how can you remember them all? The answer is, you don’t. The real power of the computer is its ability to do the work for you. To get it to do that, we use the power of the shell to automate things. We write shell scripts.

What Are Shell Scripts?

In the simplest terms, a shell script is a file containing a series of commands. The shell reads this file and carries out the commands as though they have been entered directly on the command line.

The shell is somewhat unique, in that it is both a powerful command line interface to the system and a scripting language interpreter. As we will see, most of the things that can be done on the command line can be done in scripts, and most of the things that can be done in scripts can be done on the command line.

We have covered many shell features, but we have focused on those features most often used directly on the command line. The shell also provides a set of features usually (but not always) used when writing programs.

Scripts unlock the power of your Linux machine. So let’s have some fun!

Contents

  1. Writing Your First Script And Getting It To Work
  2. Editing The Scripts You Already Have
  3. Here Scripts
  4. Variables
  5. Command Substitution And Constants
  6. Shell Functions
  7. Some Real Work
  8. Flow Control – Part 1
  9. Stay Out Of Trouble
  10. Keyboard Input And Arithmetic
  11. Flow Control – Part 2
  12. Positional Parameters
  13. Flow Control – Part3
  14. Errors And Signals And Traps (Oh My!) – Part 1
  15. Errors And Signals And Traps (Oh My!) – Part 2

https://askubuntu.com/questions/223691/how-do-i-create-a-script-file-for-terminal-commands/223698

http://stackoverflow.com/questions/17015449/how-do-i-run-sh-or-bat-files-from-terminal

Ubuntu – Shell script to execute/run (pdf)

wikiHow to Write a Shell Script Using Bash Shell in Ubuntu

How to create & execute a script file [closed]

Advanced Bash-Scripting Guide (An in-depth exploration of the art of shell scripting) by Mendel Cooper

 

Install Oracle Java 8 with PPA on Ubuntu 16.04

This post provides the instructions to install Oracle JDK 8 on Ubuntu 16.04. (Notes: Do not install JDK 9 yet, JDK 8 is the latest most stable version.)

(If you are not sure which JDK — OpenJDK or Oracle JDK — to install, check this post for the main difference between them.)

The PPA of Oracle Java for Ubuntu is being maintained by Webupd8 Team. JAVA 8 is released with many of new features and security updates, read more about whats new in Oracle Java 8.

  • Add Oracle’s PPA, then update your package repository.

We need to add webupd8team Java PPA repository onto our system. Then install Oracle Java 8 by issuing the following commands.

$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer

Note that when issuing the command:
sudo add-apt-repository ppa:webupd8team/java
if you get the error:
sudo: add-apt-repository: command not found
do the following:
sudo apt-get install software-properties-common
And then rerun adding your repository.

Note that it is possible to install multiple Java installations on one machine, and set one of installed versions as the default. Check out How To Install Java with Apt-Get on Ubuntu 16.04 (April 23, 2016)  (pdf), in particular the “Managing Java” section.

  • Verify Installed Java Version

After successfully installing Oracle Java, use the following to verify what version we installed.

$ java -version 

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
  • Configuring Java Environment and Set the JAVA_HOME Environment Variable

We also need to install java configuration package. The package should come with the latest operating systems during installation of JAVA packages. But it does no harm to run the following command to be sure we have it installed on our machine.

$ sudo apt-get install oracle-java8-set-default

Many programs use the JAVA_HOME environment variable to determine the Java installation location.

Copy the path from your preferred installation and then open /etc/environment configuration file using  nano or your favorite text editor, to set JAVA_HOME environment variable.

sudo nano /etc/environment

At the end of this file, add the following line, making sure to replace the highlighted path with your own copied path.


JAVA_HOME=/usr/lib/jvm/java-8-oracle

Save and close the file and exit nano editor environment. (Note: Ctrl+O to save the file, and then hit Enter, and then Ctrl +X to close and exit the file.)

Use the following command to reload the file.

  • source /etc/environment

You can now test whether the environment variable has been set by issuing the following command:

echo $JAVA_HOME

This will return the path you just set.

  • Conclusion

We have now installed Java 8 on our system and set it as default. We can now install software which runs on Java, such as Tomcat and Solr.

 

References:

How To Install Java with Apt-Get on Ubuntu 16.04 (April 23, 2016)  (pdf)

This is a very good post, it introduced the installation of both OpenJDK and Oracle JDK 6/7/8/9

How to Install Oracle JAVA 8 (JDK/JRE 8u121) on Ubuntu & LinuxMint with PPA (Mar 29, 2017 by Rahul K.)  – pdf

 

Using Apache Solr with Python

This post provides the instructions to use Apache Solr with Python in different ways.

======using Pysolr

Below are two small python snippets that the author of the post used for testing writing to and reading from a new SOLR server.

The script below will attempt to add a document to the SOLR server.

# Using Python 2.X
from __future__ import print_function  
import pysolr

# Setup a basic Solr instance. The timeout is optional.
solr = pysolr.Solr('http://some-solr-server.com:8080/solr/', timeout=10)

# How you would index data.
solr.add([  
    {
        "id": "doc_1",
        "title": "A very small test document about elmo",
    }
])

The snippet below will attempt to search for the document that was just added from the snippet above.

# Using Python 2.X
from __future__ import print_function  
import pysolr

# Setup a basic Solr instance. The timeout is optional.
solr = pysolr.Solr('http://some-solr-server.com:8080/solr/', timeout=10)

results = solr.search('elmo')

print("Saw {0} result(s).".format(len(results)))  

 

======GitHub repos

pysolr is a lightweight Python wrapper for Apache Solr. It provides an interface that queries the server and returns results based on the query.

install Pysolr using pip

pip install pysolr

Multicore Index

Simply point the URL to the index core:

# Setup a Solr instance. The timeout is optional.
solr = pysolr.Solr('http://localhost:8983/solr/core_0/', timeout=10)

SolrClient is a simple python library for Solr; built in python3 with support for latest features of Solr.

Components of SolrClient

 

References:

 

Apache Solr resources

Elasticsearch and Apache Solr are open source search engines, and they are the most widely used search servers. This post provides resources about Apache Solr.

Apache Solr is a fast open-source Java search server.

Solr enables you to easily create search engines which searches websites, databases and files.

Solr (pronounced “solar”) is an open source enterprise search platform, written in Java, from the Apache Lucene project. Its major features include full-text search, hit highlighting, faceted search, real-time indexing, dynamic clustering, database integration, NoSQL features and rich document (e.g., Word, PDF) handling. Providing distributed search and index replication, Solr is designed for scalability and fault tolerance. Solr is the second-most popular enterprise search engine after Elasticsearch.

Solr runs as a standalone full-text search server. It uses the Lucene Java search library at its core for full-text indexing and search, and has REST-like HTTP/XML and JSON APIs that make it usable from most popular programming languages. Solr’s external configuration allows it to be tailored to many types of application without Java coding, and it has a plugin architecture to support more advanced customization.

An Elasticsearch / Apache Solr index is the equivalent of a SQL table.

An Elasticsearch or Solr server (aka Solr instance, aka Solr engine) can maintain several indexes.

(Elasticsearch index configuration is done with HTTP / JSON commands. No files required. You define types, mappings, analysis with simple commands.)

In Apache Solr, each index is defined by a schema.xml file (it’s not mandatory in Solr 5/6, but recommended in production), and a solrconfig.xml file. The index schema is equivalent to a SQL table schema definition.  (See this post for Solr Schema related resources.)

An index contains several documents, equivalent to SQL table rows. Each document contains fields, equivalent to SQL table columns.

When an index document is inserted/updated/deleted, we say it is “indexed”.

To retrieve documents from an index, Elasticsearch (json) / Apache Solr (xml, json) provide an http API, with a proprietary syntax.

Elasticsearch and Apache Solr are web applications. A client will use their http API to query or store data.

A full-text search engine is built from the ground to tackle problems that a SQL search find difficult or impossible. The list of those features is huge: multi-language, dedicated plugins to extend the engine, synonyms, stop words, facets, boosts, …

The core search engine of Elasticsearch and Apache Solr is Apache LuceneThe relationship between Elasticsearch / Apache Solr and Lucene, is like that of the relationship between a car and its engine.

You can access Solr admin from your browser: http://localhost:8983/solr/

use the port number used in installation.

See below for some useful Solr related resources:

Check out his Unofficial Solr Guide (e.g., Solr 6.5 Features)

Configuring

Integrating Solr

Parallel Programming using MPI in Python

This post introduces Parallel Programming using MPI in Python.

The library is mpi4py (MPI and python extensions of MPI), see here for its code repo on bitbucket.

Laurent Duchesne provides an excellent step-by-step guide for parallelizing your Python code using multiple processors and MPI. Craig Finch has a more practical example for high throughput MPI on GitHub. See here for more mpi4py examples from Craig Finch.

An example of TensorFlow using MPI can be found here.

References:

 

OpenJDK or Oracle JDK? What is the main difference?

This post introduces what is OpenJDK and Oracle JDK and what is the difference and which one should we use on Ubuntu.

Both OpenJDK and Oracle JDK are created and maintained currently by Oracle only.

OpenJDK is the default version of Java that Ubuntu uses and is the easiest to install while Oracle Java 7/8 is Oracle’s own version of Java 7/8.

It entirely depends on the target platform on which you want to run JDK. Technical differences are a consequence of the goal of each one (OpenJDK is meant to be the reference implementation, open to the community, while Oracle is meant to be a commercial one)

They both have “almost” the same code of the classes in the Java API; but the code for the virtual machine itself is actually different, and when it comes to libraries, OpenJDK tends to use open libraries while Oracle tends to use closed ones.

OpenJDK was reported to work better for large number of users with small request count, while it become worse for small number of user with prolonged. This is an undocumented behaviour, and never seen anywhere other than experienced on some J2EE containers.

My conclusion:
I choose to install Oracle JDK, since there were complaints about using OpenJDK would meet bugs sometimes. (See this post if you decide to install Oracle Java 8 with PPA on Ubuntu.)

References:

Which Java package should I use: OpenJDK or Oracle JDK?

Performance OracleJDK or OpenJDK (pdf)

OpenJDK – Oracle is better? (pdf)

Is there any advantage of installing OpenJDK instead of Oracle Java Platform, Standard Edition on Ubuntu? (pdf)

 

Run R scripts from the command line on Ubuntu

Running R scripts from the command line can be a powerful way to:

  • Automate your R scripts
  • Integrate R into production
  • Call R through other tools or systems
There are basically two Linux commands that are used:
  1. RScript (preferred)
  2. The older command is  R CMD BATCH.

A better way to run R scripts in batch mode is Rscript, and its comes with R.

See the example below to see the difference between using RScript and R CMD BATCH.

Save 

print("hello world")

as a r script file and name it helloworld.r, and then run it in your terminal:

(Be sure to first cd to the path where you saved the helloworld.r file.)

then type the commands (the lines below in bold) to your temrinal

$ Rscript helloworld.r
[1] "hello world"
$ R CMD BATCH helloworld.r
$

We can see that Rscript directly  output to the terminal , and R CMD BATCH has done nothing. But actually, R CMD BATCH has written its output to a file called helloworld.r.Rout(it is located at the same place where you put helloworld.r), and that output includes both the commands and output, just like in interactive mode, along with some runtime stats:

> print("hello world")
[1] "hello world"
> 
> 
> proc.time()
   user  system elapsed 
  0.080   0.004   0.113

You can call these directly from the command line or integrate them into a bash script. You can also call these from any job scheduler.

Note, these are R related tools. The RStudio IDE does not currently come with tools that enhance or manage the RScript and R CMD BATCH functions. However, there is a shell built into the IDE and you could conceivably call these commands from there.

 The alternative to the using the Linux command line is to use the source() function inside of R. The source function will also call a script, but you have to be inside an R session to use it.

References:

How to run R scripts from the command line (Nathan Stephens on January 02, 2017)

Running R batch mode on Linux (pdf)

RScript man page

Setup R environment on Ubuntu 16.04 (R-Base and RStudio)

This post provides instructions for installing R-Base and RStudio on Ubuntu 16.04.

  • Install R-Base

You can find R-Base in the Software Center; this would be the easy way to do it. However, the Software Center versions are often out of date, which can be a pain moving foward when your packages are based on the most current version of R Base. The easy fix is to download and install R Base directly from the Cran servers.

1. Add R repository

First, we’ve got to add a line to our /etc/apt/sources.list file. This can be accomplished with the following. Note the “xenial” in the line, indicating Ubuntu 16.04. If you have a different version, just change that.

sudo echo "deb http://cran.rstudio.com/bin/linux/ubuntu xenial/" | sudo tee -a /etc/apt/sources.list

2. Add R to Ubuntu Keyring

First:

 gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9

Then:

 gpg -a --export E084DAB9 | sudo apt-key add -

3. Install R-Base

sudo apt-get update
sudo apt-get install r-base r-base-dev

 

If you would like to use R in IDE like RStudio, See below for the instructions.

  • Installing RStudio

Use CTRL + ALT + T to open your terminal, then use the commands below. If you would like to install the latest version, just change the link info after the wget command. (Note that you can get latest RStudio download link at here. See the picture below the install commands to see how to get the latest version of RStudio for you. Be sure to revise the command part associated with the version you would like to install accordingly, which I highlight in red and italic below.)

# Download and Install RStudio
sudo apt-get install gdebi-core
wget https://download1.rstudio.org/rstudio-1.0.136-amd64.deb
sudo gdebi rstudio-1.0.136-amd64.deb
rm rstudio-1.0.136-amd64.deb

References:

How to Install R on Linux Ubuntu 16.04 Xenial Xerus (April 26, 2016 By Kris Eberwein)

Install R and RStudio on Ubuntu 12.04/14.04/16.04 (Michael Galarnyk on Dec 17, 2016 )

 

Parallel programming on Ubuntu using OpenMP with C/C++

This post introduces the basic of parallel programming on Ubuntu using OpenMP with C/C++. (MP in OpenMP refers to as Multi-processing.)

OpenMP, is an an API that enables direct multi-threaded, shared memory parallelism. See here for a nice OpenMP tutorial. A good introduction of OpenMP can be found here.

See here for wiki intro to OpenMP.

OpenMP uses a portable, scalable model that gives programmers a simple and flexible interface for developing parallel applications for platforms ranging from the standard desktop computer to the supercomputer.

An application built with the hybrid model of parallel programming can run on a computer cluster using both OpenMP and Message Passing Interface (MPI), such that OpenMP is used for parallelism within a (multi-core) node while MPI is used for parallelism between nodes. There have also been efforts to run OpenMP on software distributed shared memory systems, to translate OpenMP into MPI and to extend OpenMP for non-shared memory systems.

1: Create a text file and paste the follow code into it

Writing the hello world code in C is the same everywhere. There are only minor differences in the code.

#include<stdio.h>

int main( int ac, char **av)

{

#pragma omp parallel // specify the code between the curly brackets is part of an OpenMP parallel section.

{

printf("Hello World!!!\n");

}

return 0;

}

Save the code as anyname.c.  Now let’s compile the code.

2:  Open a terminal (CTRL + ALT + T)

3: Compile the code

In your terminal, cd to the path where the c script you just created is located.

To compile your code, simply type this in the terminal

gcc anyname.c -o anyname.out

If there is any problem in the code, you must fix it and compile again. Now to next (big) step .

4: Number of threads and running in OpenMp

When the code is compiled correctly, we can run it parallel form using OpenMP. firstly we determine number of threads we are going to use. type this in your terminal:

export OMP_NUM_THREADS=4

you can try changing the number of threads into numbers you like. (Note that how many maximum threads you can use depends on your machine’s hardware.)

Then, we are going to compile it using the OpenMP. It is similar to normal compiling but with addition of a few words. The format is like this:

gcc -fopenmp anyname.c -o anyname.out

after that, you can run the program. To run the program, type this into your terminal:

./anyname.out

you will find out that the hello world prints out as many times as your thread number.

 

Notes for c++ with OpenMP :

Here I will just talk about how to compile cpp code, the other part with OpenMP is the same as c code, except for: need to change .c to .cpp and gcc to g++.

If you don’t have C/C++ compiler then first install it by  (Normally this comes with an installed Ubuntu OS):
sudo apt-get build-essential
Then open Gedit or any editor , write your code and save it as
“myprogram.cpp”. (See below for a simple example of c++ code)

#include <iostream>
int main()
{
  #pragma omp parallel 
  {
      std::cout << "hello world \n";
    }
    return 0;
}

Next step is to compile:
Make sure your terminal is cd to the folder where you have saved your .cpp file.
cd /path to/the cpp file
Once done you can compile it using G++
g++ myprogram.cpp
This will output a.out (by default) which is your executable code that can be run as
 ./a.out
If you want to name the output instead of a.out, you can pass a parameter to G++
g++ myprogram.cpp -o myprogram
This time it will output a file “myprogram” which again can be run in the same way
./myprogram

 

Notes for OpenMP:

You don’t install OpenMP. OpenMP is a feature of the compiler. Check that the compiler you are using implements OpenMP, which is an API for parallel programming in C/C++/Fortran.

Check with the OpenMP website, openmp.org for a list of compilers and platforms that support OpenMP programming.

OpenMP Directives:

  1. Directive Format
  2. C/C++ Directive Format
  3. Directive Scoping
  4. PARALLEL Construct

(Thanks Weiming for introducing this cool feature of OpenMP to the lab.)

Posts referenced:

Commonly used Linux commands (Ubuntu)

This page lists commonly used Linux commands to help those who are not very familiar with Linux command environment. I have been collecting and recording those from my experience.

I was once a beginner, so I can understand the pain for Linux beginners. I have not yet seen any post that has done comprehensive collection of commands on Linux, so I thought I could help this out. That is why you see this post. Here you go. Happy Learning!

You can see further reading list at the end of this post.

Note: Do not contain space in your filename or directory name, use underscore instead of space.

======Basic commands:

  • cd 

this command will goes back to the home directory of your account, no matter where your current directory is located in your terminal.

  • cd ../

this command will go to the parent directory of your current directory in terminal

  • rmdir 

remove/delete an empty folder.

example:

first cd into the parent folder of the empty folder to remove

rmdir test

  • rm -rf ./*

    first cd to a directory, and this command will empty all of things under the current directory

  • ls  

list all the files and folders under current path

  • ls -l  

list all the files and folders with details like dates.

  • ls -l -t

list files and folders ordered by time.

  • ls -ltrh 

list all the csv files under the current directory in long format by time and in reverse order, the file size in human readable format (e.g., in mb, or gb, instead of byte size)

  • -l List in long format. If the output is to a terminal, a total sum for all the file sizes is output on a line before the long listing.
  • -r Reverse the order of the sort to get reverse lexicographical order or the oldest entries first (or largest files last, if combined with sort by size.
  • -t Sort by time modified (most recently modified first) before sorting the operands by lexicographical order.
  • ls -ltrh *.csv

list all the csv files under the current directory in long format by time and in reverse order, the file size in human readable format (e.g., in mb, or gb, instead of byte size)

  • find

find -name ‘*.jpg’ -exec cp {} ./test/ \;

Find all jpg files  and then copy the found files to the folder test which is subfolder of current path.
Note that: the current path should be the path where the files to search are located. (i.e., use cd to locate to the directory where the files are in before type in the cmd below into terminal.)

  • rm -r -f

-r means recursive, it will remove folders and subfolders and files within the folders and subfolders

-f means force

  • mkdir [folder name]

create new folder

example:

mkdir image

  • cp [filename] [new filename]

copy and rename file

  • cp [filename] [path/to/new/lotcation/filename]

copy the file to another location

if you use this commond to copy a directory, you would meet this error:

cp: omitting directory ...

The error notice means you told cp to copy files and not directories. The warning is about cp finding a directory and informing you it will be skipped.

  • cp -r [directory] [path/to/new/location/directory]

copy a directory to another location.

cp -r means recursive and this option will make cp also include sub-directories.

If you meet permission denied error, add sudo before the command, and it will ask your password.

  • mv [directory] [path/to/new/location/directory]

If you meet permission denied error, add sudo before the command, and it will ask your password.

  • nano [new file name or /path/to/new file/new file name]

example:

nano  myexample   #it will create a new empty file named “myexample” under the current directory

  • nano [file name]

If the file name already exists, it will open the file and you can edit it.

Note: Ctrl+O to save the file, and then hit Enter, and then Ctrl +X to close the file.

 

 

======More advanced commands:

  • cd into directory without having permission

When cd into a directory and the following error occurs

bash: cd: your-dirctory: Permission denied

The solution is:

Enter super user mode, and cd into the directory that you are not permissioned to go into. Sudo requires administrator password.

sudo su  
cd directory  # you will notice that your prompt changes after your enter your root password. now you can cd to the directory.

# to exit "super user" mode, type exit.
  • lspci

check GPU information on Ubuntu

look for “VGA compatible controller:”…

  • sudo nvidia-smi

check GPU info and GPU usage.

  • sudo reboot -h now

reboot a server from terminal

  • sudo shutdown -h now

shut down a server from terminal

Note: If your Ubuntu Server 16.o4 LTS has Black Screen after reboot or shut down, try pressing (simultaneously) Ctrl + Alt + F2 to see whether you can switch to different console

  • vncserver -kill :1  

This is a vncserver command. It is used to kill a port of a GUI by VNC server, where 1 is the port you would like to kill.

  • vncviewer -via username@yourserver_hostname :1

connect to a server via vncviewer from a linux-based client. you need to change the port number 1 to yours.

echo is a built-in command in the bash and C shells that writes its arguments to standard output.

See here, and here, and herefor example usage of it.

  • cat

See here for example usage of cat command.

  • chmod

see here for example usage of chmod command.

 

  • check supercomputing Cluster’s Linux distribution and version

$ lsb_release -a
LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID:    RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 6.4 (Santiago)
Release:    6.4
Codename:    Santiago

  • show the list of top processes ordered by RAM and CPU  use in descendant form

(remove the pipeline and head if you want to see the full list):

$ ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head

Brief explanation of the options used in the command above:

— The -o (or –format) option of ps allows us to specify the output format.

— the processes’ PIDs (pid), PPIDs (pid)

— the name of the executable file associated with the process (cmd), and

— the RAM and CPU utilization (%mem and %cpu, respectively).

We can use --sort to sort by either %mem or %cpu. By default, the output will be sorted in ascendant form, but usually we prefer to reverse that order by adding a minus sign in front of the sort criteria to make it list in descendant.

To add other fields to the output, or change the sort criteria, refer to the OUTPUT FORMAT CONTROL section in the man page of ps command.


======File Transfer: getting files to/from your account on a server

  • On Linux generally the command line scp command.
 Examples of using the command line are:

scp -p file_name username@yourserver_hostname:destination/directory

  • or for a full directory tree:

scp -pr dir_name username@yourserver_hostname:destination/directory

 

Note that if you want to transfer files from server to your client computer, just reverse the directory.

e.g., scp -pr username@yourserver_hostname:source/directory dir_name_on_your_client

 

======download files

  • wget (tool for downloading files)  (pdf)
  • See Linux wget command (pdf), which provides detailed and comprehensive different tags (options) to use with wget command.

======Save terminal output to a file

  • sudo command -option | tee logThis command will show output on terminal and save to a file at the same time.
  • Save terminal output to a fileredirect the output to a file: someCommand > someFile.txt Or if you want to append data: someCommand >> someFile.txt If you want stderr too use this: someCommand &> someFile.txt or this to append:  someCommand &>> someFile.txt
  • Tail -f log.txt

Python related commands:

  • enter python environment

type python in terminal, and it will show python 2.7 version info and also enter into python 2 environment

  • enter python 3 environment

python3 

and it will show python 3 version info (e.g., python 3.5.2) and also enter into python 3 environment.

 

======Git related commands

  • git clone the url to gitclone

for example:

first cd into the folder you want the models to be cloned to in your terminal, and then issue this command. it will clone the model foder from https://github.com/tensorflow/models under your current folder in your terminal.

git clone https://github.com/tensorflow/models

 

======Some useful shortcuts on linux

  • you can open multiple terminals

open each terminal by pressing Ctrl + Alt + T.

  • Shortcut to bring all open terminals to the front

After you bring one terminal window in the front, press Alt+~ to bring all other terminal windows in the front one by one:

  • CTRL + C − terminate the current command. 

======References and further reading list:

Linux and Unix top 10 command pages  (See here for links to more commands intro)

Below is a listing of the top 10 Unix command pages by the amount of times they have been accessed on the Computer Hope server.

  1. Linux and Unix tar command help
  2. Linux and Unix chmod command help
  3. Linux and Unix ls command help
  4. Linux and Unix find command help
  5. Information about the Linux and Unix grep command
  6. Linux and Unix cp command help
  7. Linux and Unix vi command help
  8. Linux and Unix ifconfig command help
  9. Linux and Unix date command help
  10. Linux and Unix kill command help

======TOC of the nice tutorial: Linux Shell Commands: A Tutorial Quick Reference for Desktop Users

Table of Contents

1. A Short Intro to the Command Line

This chapter will acquaint you with the basics of the command line. To maximize your learning, you should follow along by typing in the example commands given. Every major Linux distribution has a menu item called “shell”, “console”, “terminal” or the like, which will give you a window with a command line interface. In this book, I assume that readers work in a graphical desktop environment and use the Bash shell in a terminal window. Bash is the default shell in all major Linux distributions.

2. Getting Information

The commands presented in this chapter provide valuable information on the state and configuration of your system.

3. Managing Files and Directories

The command line offers you great flexibility in creating, copying, moving and editing files and directories, as this chapter shows.

  • cd (change directory)  (pdf)
  • chgrp (change group ownership)  (pdf)
  • chmod (change file permissions)  (pdf)
  • chown (change file ownership)  (pdf)
  • cp (copy files and directories)  (pdf)
  • dd (write data to devices)  (pdf)
  • find (search for files)  (pdf)
  • ln (make links between files)  (pdf)
  • locate (find files by name)  (pdf)
  • mkdir (create a directory)  (pdf)
  • mount (mount file systems)  (pdf)
  • mv (rename files)  (pdf)
  • rm (remove files or directories)  (pdf)
  • rmdir (remove empty directories)  (pdf)
  • shred (delete a file securely)  (pdf)
  • touch (change file timestamps)  (pdf)
  • umount (unmount file systems)  (pdf)

4. Managing Processes

Linux provides powerful tools for controlling the execution of your programs. Some of the most important tools are presented in this chapter.

  • disown (detach a job from the shell)   (pdf)
  • kill (terminate a process)  (pdf)
  • ps (list running processes)  (pdf)
  • pstree (display a tree of processes)  (pdf)
  • shutdown (halt or reboot the system)  (pdf)
  • sudo (execute a command as root)  (pdf)

5. Working with Text

Processing plain text files is a big strength of Linux. The commands presented in this chapter allow you to display particular parts of files (e.g. head, tail), reorder their contents (e.g. sort), carry out search/replace operations (e.g. grep, sed), and much more.

  • cat (concatenate and output files)  (pdf)
  • cut (output columns from files)  (pdf)
  • diff (show differences between files)  (pdf)
  • grep (print lines matching a pattern)  (pdf)
  • head (output the first part of files)  (pdf)
  • less (view file by pages)  (pdf)
  • pdftk (manipulate PDF files)  (pdf)
  • sed (search and replace text)  (pdf)
  • sort (sort lines of text files)  (pdf)
  • tail (output the last part of files)  (pdf)
  • wc (count lines, words and characters)  (pdf)

6. Being Productive

This chapter collects some commands that can help you accomplish everyday tasks quickly and efficiently. Many of the commands are faster or more reliable replacements for popular graphical applications. For example, wget can replace a graphical download manager.

  • alias (define command shortcuts)  (pdf)
  • alsamixer (audio mixer)  (pdf)
  • bc (command line calculator)  (pdf)
  • history (display command history)  (pdf)
  • rsync (fast, versatile file copying tool)  (pdf)
  • tar (Linux archiving utility)  (pdf)
  • unrar (extract files from RAR archives)  (pdf)
  • unzip (extract files from ZIP archives)  (pdf)
  • wget (tool for downloading files)  (pdf)
  • xmodmap (change key bindings)  (pdf)

======The end of the TOC of the nice tutorialLinux Shell Commands: A Tutorial Quick Reference for Desktop Users

======apt-get usages

======curl command examples

cURL can be used in many different and useful ways. Using cURL, we can download, upload and manage files, check email address, or even update status on some of the social media websites or check the weather outside.

cURL is very useful command line tool to transfer data from / to a server. cURL supports various protocols, including FILE, HTTP, HTTPS, IMAP, IMAPS, LDAP, DICT, LDAPS, TELNET, FTP, FTPS, GOPHER, RTMP, RTSP, SCP, SFTP, POP3, POP3S, SMB, SMBS, SMTP, SMTPS, and TFTP.

This tutorial covers five of the most useful and basic uses of cURL tool:

–Check URL

One of the most common and simplest uses of cURL is typing the command itself, followed by the URL you want to check

curl https://example.com
#This command will display the content of the URL on your terminal

–Save the output of the URL to a file

The output of the cURL command can be easily saved to a file by adding the -o option to the command, as shown below

curl -o website https://example.com
#the output will be save to a file named ‘website’ in the current working directory

–Download files with cURL

curl -O https://example.com/file.zip

# the -O option used for saving files to current working directory without renaming
# e.g.,  the ‘file.zip’ zip archive will be downloaded to the current working directory.
curl -o archive.zip https://domain.com/file.zip

# the ‘file.zip’ archive will be downloaded and saved as ‘archive.zip’.
curl -O https://domain.com/file.zip -O https://domain.com/file2.zip

# cURL can be also used to download multiple files simultaneously
#cURL can be also used to download files securely via SSH

curl -u user sftp://server.domain.com/path/to/file

# Note that the full path of the file to be downloaded is required

–Get HTTP header information from a website

You can easily get HTTP header information from any website you want by adding the -I option (capital ‘i’) to cURL.

curl -I http://example.com

–Access an FTP server

#  access your FTP server with cURL 
curl ftp://ftp.domain.com --user username:password

# cURL will connect to the FTP server and list all files and directories in user’s home directory
curl ftp://ftp.domain.com/file.zip --user username:password
# download a file via FTP using curl
curl -T file.zip ftp://ftp.domain.com/ --user username:password
# upload a file to  the FTP server

–check cURL manual page to see all available cURL options and functionalities

man curl

This post covers detailed and comprehensive explanation of different options to use with curl command.

  • $ free -m

Linux has the habit of caching lots of things for faster performance, so that memory can be freed and used if needed.

  • $ cat /proc/meminfo
  • $ vmstat -s