Install Ubuntu 16.04 on Oracle VirtualBox that runs on Windows or Mac

This post provides some notes and useful resources about installing Ubuntu 16.04 on Oracle VirtualBox that runs on your Mac or Windows.

Note: check the RAM and hard disk size of your machine before creating a virtual machine on it.

Notes about which version of Ubuntu to download and install:

For Ubuntu, it is not always a wise choice to pick the newest version. My suggestion is that (unless you are aware that you need to install a particular version), download and install the latest LTS (Long Term Support) version (see the picture below from Ubuntu wiki page). Every two years, a Ubuntu LTS version is released, which will be supported for updates for five years. For example, as of now, Ubuntu 16.04 LTS is the latest LTS version.

The two main things you need to pay attention to when you create a virtual machine:

  • Memory allocation for your virtual machine.

You can set it as half of your RAM (e.g., if your RAM is 8 G, set it as 4 G or 5G for your virtual machine should be fine.)

  • Storage type:  Select “Dynamically allocated” if you are not sure how large storage you actually will need.

There are already several very good tutorials about this along with snapshots, so I won’t create a tutorial for this. See below for some useful resources I collected. (See some notes I wrote below for some posts.)

My notes: This one is very good (with snapshots), including  Guest additions and Shared folders settings. (Note that Guest additions are required if you want to set Shared folder, so be sure to install Guest additions first).

You can use the following command to check whether Guest additions were installed on your Ubuntu virtual machine if you are not sure because you installed your Ubuntu VM a while ago. (Note: even though you may find Guest additions was installed, you will still need to install Guest additions for your newly installed VM, otherwise the Shared folders wont work for you.)

Use lsmod from the command line, as it will tell you not only if it’s installed, but properly loaded:

$ lsmod | grep vboxguest
vboxguest             282624  6 vboxsf

I have tested Shared folders instructions (with pictures) in this tutorial on my Ubuntu 16.04 VM, and it works. The only difference is that on Ubuntu 16.04 VM, after you issued the following command on your terminal and  restart the Ubuntu guest machine, you do not need to do anything as the tutorial said, the shared folder is automatically mounted each time you start you Ubuntu VM. (After you restart, click the Files icon on the task bar, and you will see the shared folder you just set just now is automatically mounted there:))

  • sudo adduser brb vboxsf   # Replace 'brb' with your account name on Ubuntu. 

One more note: Although Shared Folder setting in VM is very convenient, using VirtualBox shared folder directly for fastq data, annotation or output directory can significantly reduce the performance compared to a native (Ubuntu) system or VirtualBox native system, so my recommendation is only use the folder to transfer files between windows/mac and your Ubuntu VM.

P.S. If you see some tutorials tell you that you need to enter some command like “sudo mount -t vboxsf sharing /mnt/share” to automatically mount the shared folder each time you start your Ubuntu VM, that is outdated instructions.

Fortunately, new VirtualBox version (4.x +) has a (GUI) Auto-mount option (see pics below) when you set your shared folder. (Note that you can choose your customized folder to share, instead of using a system predefined folder such as Documents or Downloads.)

If you want to share the clipboard between your host and your virtual machine, check out the picture below.

 

Answers to some frequently asked questions:

Q: Do I need to backup my files when I upgrade my VirtualBox to newer version.

A: just install the latest version and you will have all your files in the new one. You need not have to uninstall the old virtual machine.

Q: After I install the updates of Windows 10, my VirtualBox won’t start…

A: just install the latest version and you will have all your files in the new one. You need not have to uninstall the old virtual machine.

 

My notes:  this one is very good (with snapshots) on Mac. My notes above about VM settings running on Windows work the same for VM settings running on Mac.

 

How to install a graphics card

This post introduces how to install a graphics card and related resources.

Notes: When installing or removing a graphics card, handle it only by its edges, not by its connectors or components.

Before starting with graphics card installation, check out the following two things.

Step 1: make sure your computer has the proper hardware to support your new card.

The most common problem that people run into is an inadequate power supply:

  • Either it can’t supply enough wattage,
  • or it doesn’t have enough available PCI-E power connectors.

As a rule of thumb, your power supply should be rated from double the power consumption of your graphics card. For example if you purchased a R9 290X—a video card that draws 300 watts—you should have a power supply that can provide at least 600 watts of power and has both 8-pin and 6-pin PCI-E power connectors.

To find out how much wattage your power supply pumps out, open your case and look for the standard identification sticker all power supplies have, which lists their basic info. While you’re there you can also identify how many 6-pin and 8-pin PCI-E connectors are available.

Picking the right power supply is even more important if you’re upgrading to a multi-card configuration, because you’ll likely need to buy a power supply rated for one or more kilowatts. For more on multi-GPU SLI and CrossFireX setups, check out PCWorld’s guide to tricking out your PC with multiple graphics cards.

Step 2: is there enough room inside of your case to fit your new graphics card? 

Some high-end graphics card can be over a foot long, and two or even three expansion slots wide. You can find the physical dimensions for a graphics on its product page or on the manufacturer’s website.

With all of those questions resolved, it’s time to install the new card.

Installing a graphics card

Installing a graphics card requires three things:

  • a new graphics card,
  • your computer, and
  • a Phillips-head screw driver.

Notes: Be sure to turn off your PC and unplug it from the wall before you begin.

Start by removing the side of your computer’s case, then locate and remove your current graphics card. Some PCs will not have a graphics card installed. Instead, you need to locate the PCI-E x16 slot closest to the heat sink of your processor. This will either be the first or second expansion slot on your motherboard.

You install a graphics card into a PCI-E x16 slot on your computer’s motherboard (the long, blue slots in this picture given below.)

gpu install 1 of 7(By )

Make sure that there are no loose wires blocking your access to this slot. If you’re replacing an existing graphics card, unplug any cables connected to it, remove the screw from its retention bracket, and then remove the card. Most motherboards also have a small plastic latch on the end of the PCI-E slot that locks the graphics card in place. Make sure you toggle this latch to unlock your old graphics card so you can remove it.

You can now install your new graphics card into the open and unobstructed PCI-E x16 slot. Firmly insert the card into the slot, then push down the plastic lock on the end of the PCI-E slot to hold it in place. Next, use a screw to secure the graphic card’s metal retention bracket to your PC’s case. You can reuse the same screw(s) that held the cover bracket or your former graphics card in place.

Notes: Don’t forget to lock the latch at the end of the PCI-E slot after firmly inserting your graphics card! (This is very important, because your expensive graphics card needs this to secure its positioning. You should hear a “ta” sound when the latch is locked. It should be very firm and steadily locked, if you feel it is loose, that indicates the latch is not locked properly.)

Most gaming-level graphics cards require additional power connectors. If yours does, make sure you connect those PCI-E power cables. Your graphics card will not function correctly without properly supplied power. In fact, if you don’t connect those PCI-E power cables your PC may be unable to boot.

 

Wrapping up

With your graphics card secured and powered up, finish the job by sliding your case’s side panel back into position and plugging your display cable into your new graphics card. Turn on your computer.

Now it’s time to take care of the software side of upgrading your graphics card.

If your new graphics card is the same brand as your old card, this process is simple. Just go to the manufacturer’s website and download the latest driver package for you operating system. Keep in mind that graphics drivers are quite large, generally about 300MB in size, and it may take some time to download depending on the speed of your Internet connection. Install the driver, restart your computer.

If you’re switching manufacturers (from Intel to AMD, from AMD to Nvidia, or vice-versa), uninstall your old graphics driver and restart your computer before installing the driver for your new graphics card. If you don’t uninstall the old driver it may conflict with the new driver.

References:

Tips

  • If you intend to use more than one monitor with your video cards, be aware that while SLI is enabled, only one monitor is supported. A workaround for this involves installing additional hardware.
  • With Nvidia’s SLI, at least for the time being, you must connect two video cards with the same chipset. Example, 1 bfg 7600 gt and 1 evga 7600 gt can be connected.

Warnings

  • Make sure to ground yourself before handling hardware as ESD (electrostatic discharge) can fry your components. Static electricity remains a threat to all computer components. It is recommended that you wear clothing that does not produce static charges, that you keep in near constant contact with the computer’s chassis, and that you avoid touching the metallic traces of circuit boards inside or out of the computer.
  • Always unplug your system before installing any hardware

Apache Solr schema explained

Elasticsearch and Apache Solr are open source search engines, and they are the most widely used search servers. This page provides some explanations about Apache Solr schema. (See this post for Solr related resources.)

Let is first look at what (XML) schema means. (XML schema, a way to define the structure, content, and to some extent, the semantics of XML documents)

(Elasticsearch index configuration is done with HTTP / JSON commands. No files required. You define types, mappings, analysis with simple commands.)

Solr index configuration is done through 2 files: schema.xml and solrconfig.xml.

  • schema.xml— it defines the schema of the documents that are indexed/ingested into Solr (i.e. the set of fields that they contain). A news article may contain title, body, tags, article date etc.  It also defines the datatype of those fields. It configures the document structure (a document is made of fields with field types), and how field types are processed during indexing and querying.
  • solrconfig.xml — it contains the request handlers and other config options. It configures the “handlers”. Handlers are urls , executing plugins (java code) with their default configuration.

 

See below for some good explanations about Solr basic concepts, including Solr schema.

This section discusses how Solr organizes its data into documents and fields, as well as how to work with a schema in Solr.

This section includes the following topics:

Overview of Documents, Fields, and Schema Design: An introduction to the concepts covered in this section.

Solr Field Types: Detailed information about field types in Solr, including the field types in the default Solr schema.

Defining Fields: Describes how to define fields in Solr.

Copying Fields: Describes how to populate fields with data copied from another field.

Dynamic Fields: Information about using dynamic fields in order to catch and index fields that do not exactly conform to other field definitions in your schema.

Schema API: Use curl commands to read various parts of a schema or create new fields and copyField rules.

Other Schema Elements: Describes other important elements in the Solr schema.

Putting the Pieces Together: A higher-level view of the Solr schema and how its elements work together.

DocValues: Describes how to create a docValues index for faster lookups.

Schemaless Mode: Automatically add previously unknown schema fields using value-based field type guessing.

Check out his Unofficial Solr Guide for more useful tutorials and resources (e.g., Solr 6.5 Features)

 

 

Change port for Apache Solr from the default port 8983 on Ubuntu 16.04

This post introduces how to change the default port on which Apache Solr runs on Ubuntu 16.04.  (See my post if you have not installed Solr on your Ubuntu.)

The default port for Solr is 8983, but there are circumstances where you may want to change this. For example, if you wish to experiment with a new release, or you want your various Sitecore development instances to hit separate instances of Solr.  See below for two options for changing the port number on Ubuntu.

Step 1: use sudo service solr status to check your Solr status and the port it is running on.

yourusername@yourservername:~$ sudo service solr status
[sudo] password for yourusername: 
● solr.service - LSB: Controls Apache Solr as a Service
 Loaded: loaded (/etc/init.d/solr; bad; vendor preset: enabled)
 Active: active (exited) since Sun 2017-04-30 11:08:43 EDT; 1 weeks 0 days ago
 Docs: man:systemd-sysv-generator(8)

Apr 30 11:08:34 yourservername systemd[1]: Starting LSB: Controls Apache Solr as a Service...
Apr 30 11:08:34 yourservername su[2655]: Successful su for solr by root
Apr 30 11:08:34 yourservername su[2655]: + ??? root:solr
Apr 30 11:08:34 yourservername su[2655]: pam_unix(su:session): session opened for user solr by (uid=0)
Apr 30 11:08:42 yourservername solr[2652]: [194B blob data]
Apr 30 11:08:42 yourservername solr[2652]: Started Solr server on port 8983 (pid=2861). Happy searching!
Apr 30 11:08:43 yourservername solr[2652]: [14B blob data]
Apr 30 11:08:43 yourservername systemd[1]: Started LSB: Controls Apache Solr as a Service.

Step 2: use sudo service solr stop to  stop your Solr first before we go ahead and change  its default port.

yourusername@yourservername:/opt/solr-6.5.1/bin$ sudo service solr stop
yourusername@yourservername:/opt/solr-6.5.1/bin$ sudo service solr status
● solr.service - LSB: Controls Apache Solr as a Service
   Loaded: loaded (/etc/init.d/solr; bad; vendor preset: enabled)
   Active: inactive (dead) since Sun 2017-05-07 15:40:57 EDT; 17s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 15132 ExecStop=/etc/init.d/solr stop (code=exited, status=0/SUCCESS)

Apr 30 11:08:42 yourservername solr[2652]: Started Solr server on port 8983 (pid=2861). Happy searching!
Apr 30 11:08:43 yourservername solr[2652]: [14B blob data]
Apr 30 11:08:43 yourservername systemd[1]: Started LSB: Controls Apache Solr as a Service.
May 07 15:40:55 yourservername systemd[1]: Stopping LSB: Controls Apache Solr as a Service...
May 07 15:40:55 yourservername su[15135]: Successful su for solr by root
May 07 15:40:55 yourservername su[15135]: + ??? root:solr
May 07 15:40:55 yourservername su[15135]: pam_unix(su:session): session opened for user solr by (uid=0)
May 07 15:40:55 yourservername solr[15132]: Sending stop command to Solr running on port 8983 ... waiting up to 180 seconds to allow
May 07 15:40:57 yourservername solr[15132]: [56B blob data]
May 07 15:40:57 yourservername systemd[1]: Stopped LSB: Controls Apache Solr as a Service.

Step 3: Change config files

Check out all the following files for the port:

  • cd to /opt/solr-6.5.1/server/solr/
#the file path: /opt/solr-6.5.1/server/solr/solr.xml
yourusernmae:/opt/solr-6.5.1/server/solr$ sudo nano solr.xml
#change port here:  ${jetty.port:8983}
  • cd to /var/
# the file path: /var/solr/data/solr.xml
yourusernmae:/var$ sudo nano /solr/data/solr.xml
# change port here:  ${jetty.port:8983}
  • cd to /etc/default/
# the file path: /etc/default/solr.in.sh
yourusernmae:/etc/default$ sudo nano solr.in.sh
# change port here:  SOLR_PORT=8983

Once you save and close the solr.in.sh file you can return to your terminal and type this command to reload the file 

yourusernmae:/etc/default$ source solr.in.sh

Step 4: Start your solr service again using  sudo service solr start, you will see your solr is now running on the new port your changed just now in the step 3 above.

yourusername@yourservername:/etc/default$ sudo service solr start
yourusername@yourservername:/etc/default$ sudo service solr status
● solr.service - LSB: Controls Apache Solr as a Service
 Loaded: loaded (/etc/init.d/solr; bad; vendor preset: enabled)
 Active: active (exited) since Sun 2017-05-07 16:11:32 EDT; 3s ago
 Docs: man:systemd-sysv-generator(8)
 Process: 16988 ExecStop=/etc/init.d/solr stop (code=exited, status=1/FAILURE)
 Process: 17121 ExecStart=/etc/init.d/solr start (code=exited, status=0/SUCCESS)

May 07 16:11:29 yourservername systemd[1]: Starting LSB: Controls Apache Solr as a Service...
May 07 16:11:29 yourservername su[17125]: Successful su for solr by root
May 07 16:11:29 yourservername su[17125]: + ??? root:solr
May 07 16:11:29 yourservername su[17125]: pam_unix(su:session): session opened for user solr by (uid=0)
May 07 16:11:32 yourservername solr[17121]: [98B blob data]
May 07 16:11:32 yourservername solr[17121]: Started Solr server on port 8985 (pid=17327). Happy searching!
May 07 16:11:32 yourservername solr[17121]: [14B blob data]
May 07 16:11:32 yourservername systemd[1]: Started LSB: Controls Apache Solr as a Service.

Now you can reference Step 5: Creating a Solr search collection in my another post to create a Solr search collection for this port.

References:

 

 

Install Apache Solr 6 on Ubuntu 16.04

This post provides the tutorial to set up Apache Solr 6 on Ubuntu 16.04. (install Solr as a service that auto-starts when (re)boot Ubuntu.)

What is Apache Solr? Apache Solr is an open source enterprise-class search platform written in Java which enables you to create custom search engines that index databases, files, and websites. It has back end support for Apache Lucene. It can, for example, be used to search in multiple websites and can show recommendations for the searched content. Solr uses an XML (Extensible Markup Language) based query and result language. There are APIs (Applications program interfaces) available for Python, Ruby and JSON (Javascript Object Notation).

Some other features that Solr provides are:

  • Full-Text Search.
  • Snippet generation and highlighting.
  • Custom Document ordering/ranking.
  • Spell Suggestions.

This tutorial will show you how to install the latest Solr version on Ubuntu 16.04 LTS. The steps will most likely work with later Ubuntu versions as well.

Before Solr 5, Solr doesn’t work alone; it needs a Java servlet container such as Tomcat or Jetty. But after Solr 5, it does not need to run on Tomcat.  

Running Solr on Tomcat (No Longer Supported)

Beginning with Solr 5.0, Support for deploying Solr as a WAR in servlet containers like Tomcat is no longer supported.

For information on how to install Solr as a standalone server, please see Installing Solr.

To give an example:

Things need to do when installing Solr version before 6.

Download and install Tomcat (or some other servlet container)
Setup Tomcat as a service
Download and unpack Solr
Create a SOLR_HOME folder with correct content
copy solr.war into tomcat/webapps
set CATALINA_OPTS=“-Dsolr.solr.home=/path/to/home -Dsolr.x.y=z…. GC-flags etc”
Setup  Tomcat as a service
service tomcat start

With Solr 6.x, we just need to do:

Download Solr and unpack the install-script
solr/bin/install_solr_service solr-6.2.0.tgz  # Install
Tune /etc/default/solr.in.sh to your likings (mem, port, solr-home, Zk etc)
service solr start (or bin/solr start [options])

Your client would talk to Solr on typically http://host.name:8983/solr/ as a standalone server, not as one out of many webapps on 8080.

Apache Solr 6 required Java 8 or greater to run.

 There had been lots of scaling improvements in Solr 6.

Now let’s get started with the installation.

 

Step 1: Update your System

Use a non-root sudo user to login into your Ubuntu server. Through this user, you will have to perform all the steps and use the Solr later.

To update your system, execute the following command to update your system with latest patches and updates.

$ sudo apt-get update 
$ sudo apt-get upgrade -y   #note that this will update your ubuntu OS, skip this if you do not want to update your system.

Step 2: Install Java 

(Apache Solr 6 required Java 8 or greater to run. If you have installed Java 8 or greater on your machine, skip this.)

Solr is a Java application, so Java needs to be installed first in order to set up Solr. See my post for detailed Java 8 installation on Ubuntu 16.04.

Check the version of Java installed by running the following command

$ java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

Step 3: (Manually) install Solr 

Solr can be installed on Ubuntu in different ways, in this tutorial,we will install the latest package.  (If you would like to install the latest package from the source, check out How to install and configure Solr 6 on Ubuntu 16.04.)

Now Let’s download the required Solr version from its official site or mirrors.

First go to this Solr Download page, click the link to the latest version.

You would probably see something looks like the pic shown below. Get the download link you prefer. (for my case, I used this one http://apache.cs.utah.edu/lucene/solr/6.5.1). Click the download link you selected, and then you would see something like the pic shown below.

#If you do not have sudo privilege
#cd /path to one folder under your account 
# and you do not need to add "sudo" in the following commands
  
cd /opt
sudo wget http://apache.cs.utah.edu/lucene/solr/6.5.1/solr-6.5.1.tgz

Now extract solr service installer shell script from the downloaded Solr archive file and run installer using following commands.

sudo tar xzf solr-6.5.1.tgz solr-6.5.1/bin/install_solr_service.sh --strip-components=2

Then install Solr as a service using the script:

sudo ./install_solr_service.sh solr-6.5.1.tgz

The output will be similar to this: [Note that this installation will make Solr as a service that auto-starts when you (re)boot Ubuntu.]

myusername@myserver:/opt$ sudo ./install_solr_service.sh solr-6.5.1.tgz
id: ‘solr’: no such user
Creating new user: solr
Adding system user `solr’ (UID 117) …
Adding new group `solr’ (GID 126) …
Adding new user `solr’ (UID 117) with group `solr’ …
Creating home directory `/var/solr’ …

Extracting solr-6.5.1.tgz to /opt

Installing symlink /opt/solr -> /opt/solr-6.5.1 …

Installing /etc/init.d/solr script …

Installing /etc/default/solr.in.sh …

Service solr installed.
Customize Solr startup configuration in /etc/default/solr.in.sh
● solr.service – LSB: Controls Apache Solr as a Service
Loaded: loaded (/etc/init.d/solr; bad; vendor preset: enabled)
Active: active (exited) since Sun 2017-04-30 11:08:43 EDT; 5s ago
Docs: man:systemd-sysv-generator(8)
Process: 2652 ExecStart=/etc/init.d/solr start (code=exited, status=0/SUCCESS)

Apr 30 11:08:34 myserver systemd[1]: Starting LSB: Controls Apache Solr as a Service…
Apr 30 11:08:34 myserver su[2655]: Successful su for solr by root
Apr 30 11:08:34 myserver su[2655]: + ??? root:solr
Apr 30 11:08:34 myserver su[2655]: pam_unix(su:session): session opened for user solr by (uid=0)
Apr 30 11:08:42 myserver solr[2652]: [194B blob data]
Apr 30 11:08:42 myserver solr[2652]: Started Solr server on port 8983 (pid=2861). Happy searching!
Apr 30 11:08:43 myserver solr[2652]: [14B blob data]
Apr 30 11:08:43 myserver systemd[1]: Started LSB: Controls Apache Solr as a Service.
myusername@myserver:/opt$


Step 4:  Start / Stop Solr Service

Use the following command to check the status of the service

$ sudo service solr status

See below for a sample output:

myusername@myserver:/opt$ sudo service solr status
● solr.service - LSB: Controls Apache Solr as a Service
   Loaded: loaded (/etc/init.d/solr; bad; vendor preset: enabled)
   Active: active (exited) since Sun 2017-04-30 11:08:43 EDT; 13min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 2652 ExecStart=/etc/init.d/solr start (code=exited, status=0/SUCCESS)

Apr 30 11:08:34 myserver systemd[1]: Starting LSB: Controls Apache Solr as a Service...
Apr 30 11:08:34 myserver su[2655]: Successful su for solr by root
Apr 30 11:08:34 myserver su[2655]: + ??? root:solr
Apr 30 11:08:34 myserver su[2655]: pam_unix(su:session): session opened for user solr by (uid=0)
Apr 30 11:08:42 myserver solr[2652]: [194B blob data]
Apr 30 11:08:42 myserver solr[2652]: Started Solr server on port 8983 (pid=2861). Happy searching!
Apr 30 11:08:43 myserver solr[2652]: [14B blob data]
Apr 30 11:08:43 myserver systemd[1]: Started LSB: Controls Apache Solr as a Service.

 

Use the following commands to Start, Stop and check status of Solr service.

$ sudo service solr stop
$ sudo service solr start
$ sudo service solr status

 

Step 5: Creating a Solr search collection

(Before we create a Solr search collection, check out this post first if you want to change the default port 8983 to another port.)

Using Solr, we can create multiple collections. Run the following command, give the name of your collection (here mysolrcollection) and specify its configurations.

$ sudo su - solr -c "/opt/solr/bin/solr create -c mysolrcollection -n data_driven_schema_configs"

Sample output:

myusername@myserver:/opt$ sudo su - solr -c "/opt/solr/bin/solr create -c mysolrcollection -n data_driven_schema_configs"
 [sudo] password for myusername:

Copying configuration to new core instance directory:
 /var/solr/data/mysolrcollection

Creating new core 'mysolrcollection' using command:
 http://localhost:8983/solr/admin/cores?action=CREATE&name=mysolrcollection&instanceDir=mysolrcollection

{
 "responseHeader":{
 "status":0,
 "QTime":1422},
 "core":"mysolrcollection"}


The new core directory for our first collection has been created. To view the default schema file, got to:

cd /opt/solr/server/solr/configsets/data_driven_schema_configs/conf

You will see some files shown in the picture below.

To view other configuration options , got to:

cd /opt/solr/server/solr/configsets/

 

Step 6: Use the Solr Web Interface (i.e., Access Solr Admin Panel)

Default Solr runs on port 8983. You can access Solr port in your web browser and you will get Solr dashboard.

The Apache Solr is now accessible on the default port, which is 8983. The admin UI should be accessible at http://your_server_ip:8983/solr. The port should be allowed by your firewall to run the links. 

(If you do not know your IP, check my post to find it out.)

For example:

http://192.168.1.100:8983/solr/

Or use your machine’s host name if you have one.

http://example.org:8983/solr/

 

Here you can view statics of created collection in previous steps named “mycollection”. Click on “Core Selector” on left sidebar and select created collection.

To see the details of the first collection that we created earlier, select the “mysolrcollection” collection in the left menu.

After you selected the “mysolrcollection” collection, select Documents in the left menu. There you can enter real data in JSON format that will be searchable by Solr. To add more data, copy and paste the following example JSON onto Document field:

{
"id": 1,
"name":"John",
"age":30,
"cars":[ "Ford", "BMW", "Fiat" ]
}

Note: You can add other formats of data such as CSV etc to Solr. (See the pic below)

Click on the submit document button after adding the data.

Status: success
Response:
{
 "responseHeader": {
 "status": 0,
 "QTime": 758
 }
}

Now we can click on Query on the left side then click on Execute Query,

We will see something like this:

Conclusion

After successfully installing the Solr Web Interface on Ubuntu, you can now insert the data or query the data with the Solr API and Web Interface.

You can write code to add a large set of documents into Solr. See my post for using Solr with Python. See this post for some useful Solr resources I collected.

 

References:

Install Tomcat & Solr (You can’t avoid this one) – This is for Solr before version 5, after Solr 5, Tomcat is not required to install Solr.

Apache Solr Reference Guide/ Installing Solr  & Running Solr  & Solr Quick Start (pdf. a very good concise intro, including some basic usages and indexing xml, json, csv files)

Configuring a schema.xml for Solr

First, rename the /opt/solr/solr/collection1 to an understandable name like apples (use whatever name you’d like). (This can be skipped if you installed it using apt-get. In that case, you can execute the following command instead: cd /usr/share/solr):

cd /opt/solr/solr
mv collection1 apples
cd apples

Also, if you installed Solr manually, open the file core.properties (nano core.properties) and change the name to the same name.

Then, remove the data directory and change the schema.xml:

rm -R data
nano conf/schema.xml

Paste your own schema.xml in here.

 

 

 

Choose proper GeForce GPU(s) according to your machine

This post introduces how to choose proper NVIDIA GeForce GPU(s) according to your desktop or workstation.

We gratefully acknowledge the support of NVIDIA Corporation with the donation of (1) Titan X Pascal GPU used for our machine learning and deep learning based research.

It is very important to choose the proper GPUs according to your Desktop / Workstation (The Power Specs of your machine that will house), and also according to the overall computation performance efficiency, including the GPU Engine Specs (esp. how many NVIDIA CUDA Cores) and Memory Specs (e.g., Memory Speed, Standard Memory Config, Memory Bandwidth (GB/sec))GPU(s)) , as well as financial cost.

  • Full Specifications (The Compute Capability of the four GPU graphics cards listed below are all 6.1.)

NVIDIA TITAN Xp (VIEW FULL SPECS)

NVIDIA TITAN X (VIEW FULL SPECS)

GeForce GTX 1080 Ti

GeForce GTX 1080 (VIEW FULL SPECS)

  • Price

NVIDIA TITAN Xp

NVIDIA TITAN X

GeForce GTX 1080 Ti

GeForce GTX 1080

When you choose GeForce GPU(s) for your machine, be sure to consider both the power specs of your machine and also the GPU Engine Specs (esp. how many NVIDIA CUDA Cores) and Memory Specs (e.g., Memory Speed, Standard Memory Config, Memory Bandwidth (GB/sec)).

For example, if your machine has one 8pin and two 6pin PCIe power cables, and you have budge around $1200, I would recommend go for two GeForce GTX 1080 cards. In this case, purchasing two GeForce GTX 1080 cards will cost you a little bit less and more importantly it will give you much more computation power comparing with one single NVIDIA TITAN Xp

(Note that two 6pin PCIe power cables can be used as one 8pin PCIe power cable.)

If you machine has one 8pin and one 6pin if you have $700 budget, go for  GeForce GTX 1080 Ti

If you have two 6 pins or one 8pin, or one 8pin and  one 6pin, and you have budge around $600, the best choice would be one GeForce GTX 1080.

In this post I just compared the GPU card above GeForce GTX 1080. For more (combination) options, check the table I given below to find the best configuration according to your machine and the cost that best suitable for you.

(Thanks for Scott and Bob’s help with this.)

References:

GPU Compute Capability
NVIDIA TITAN Xp 6.1
NVIDIA TITAN X 6.1
GeForce GTX 1080 Ti 6.1
GeForce GTX 1080 6.1
GeForce GTX 1070 6.1
GeForce GTX 1060 6.1
GeForce GTX 1050 6.1
GeForce GTX TITAN X 5.2
GeForce GTX TITAN Z 3.5
GeForce GTX TITAN Black 3.5
GeForce GTX TITAN 3.5
GeForce GTX 980 Ti 5.2
GeForce GTX 980 5.2
GeForce GTX 970 5.2
GeForce GTX 960 5.2
GeForce GTX 950 5.2
GeForce GTX 780 Ti 3.5
GeForce GTX 780 3.5
GeForce GTX 770 3.0
GeForce GTX 760 3.0
GeForce GTX 750 Ti 5.0
GeForce GTX 750 5.0
GeForce GTX 690 3.0
GeForce GTX 680 3.0
GeForce GTX 670 3.0
GeForce GTX 660 Ti 3.0
GeForce GTX 660 3.0
GeForce GTX 650 Ti BOOST 3.0
GeForce GTX 650 Ti 3.0
GeForce GTX 650 3.0
GeForce GTX 560 Ti 2.1
GeForce GTX 550 Ti 2.1
GeForce GTX 460 2.1
GeForce GTS 450 2.1
GeForce GTS 450* 2.1
GeForce GTX 590 2.0
GeForce GTX 580 2.0
GeForce GTX 570 2.0
GeForce GTX 480 2.0
GeForce GTX 470 2.0
GeForce GTX 465 2.0
GeForce GT 740 3.0
GeForce GT 730 3.5
GeForce GT 730 DDR3,128bit 2.1
GeForce GT 720 3.5
GeForce GT 705* 3.5
GeForce GT 640 (GDDR5) 3.5
GeForce GT 640 (GDDR3) 2.1
GeForce GT 630 2.1
GeForce GT 620 2.1
GeForce GT 610 2.1
GeForce GT 520 2.1
GeForce GT 440 2.1
GeForce GT 440* 2.1
GeForce GT 430 2.1
GeForce GT 430* 2.1

NVIDIA TITAN X Pascal vs GTX 1080

This post introduces NVIDIA TITAN X Pascal, GTX 1080, and the comparisons between them.

In order to use TensorFlow with GPU support you must have a NVIDIA graphic card with a minimum compute capability of 3.0.

A single NVIDIA TITAN X Pascal is apparently much more powerful than a GTX 1080 graphics card if we do not consider their cost. But two GTX 1080 GPU cards will outperform a single NVIDIA Ttian X Pascal and from the financial cost perspective, two GTX 1080 will save your some money as well comparing to purchasing a single NVIDIA TITAN X. See my post, choose proper GeForce GPU(s) according to your machine,  for some detailed explanations.

References:

  • NVIDIA Titan X – The fastest accelerator for deep neural network training on a desktop PC based on the revolutionary NVIDIA Pascal architecture

Watch Now: Deep Learning Demystified – by NVIDIA

Watch Now: Deep Learning Demystified  (YouTubeUploaded on Mar 30, 2017) 

Artificial Intelligence (AI) is solving problems that seemed well beyond our reach just a few years back. Using deep learning, the fastest growing segment of AI, computers are now able to learn and recognize patterns from data that were considered too complex for expert written software. Today, deep learning is transforming every industry, including automotive, healthcare, retail and financial services.
This introduction to deep learning will explore key fundamentals and opportunities, as well as current challenges and how to address them.
Highlights include:
  1. Demystifying Artificial Intelligence, Machine Learning and Deep Learning
  2. Key challenges organizations face in adopting this new approach
  3. How GPU deep learning and software, along with training resources, can deliver breakthrough results

PRESENTED BY

Add Presenter 1's Head Shot Image URL (ex: http://info.nvidianews.com/rs/156-OFN-742/images/dan_m.jpg)
WILL RAMEY
Director, Developer Marketing, NVIDIA
Will Ramey is NVIDIA’s director of developer marketing. Prior to joining NVIDIA in 2003, he managed an independent game studio and developed advanced technology for the entertainment industry as a product manager and software engineer. He holds a BA in computer science from Willamette University and completed the Japan Studies Program at Tokyo International University. Outside of work, Will learns something new every day, usually from his two kids. He enjoys hiking, camping, open water swimming, and playing The Game.
HOSTED BY
Host1
======Below are some main screenshots from the video:

 

 

 

 

 

Install Oracle Java 8 with PPA on Ubuntu 16.04

This post provides the instructions to install Oracle JDK 8 on Ubuntu 16.04. (Notes: Do not install JDK 9 yet, JDK 8 is the latest most stable version.)

(If you are not sure which JDK — OpenJDK or Oracle JDK — to install, check this post for the main difference between them.)

The PPA of Oracle Java for Ubuntu is being maintained by Webupd8 Team. JAVA 8 is released with many of new features and security updates, read more about whats new in Oracle Java 8.

  • Add Oracle’s PPA, then update your package repository.

We need to add webupd8team Java PPA repository onto our system. Then install Oracle Java 8 by issuing the following commands.

$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer

Note that when issuing the command:
sudo add-apt-repository ppa:webupd8team/java
if you get the error:
sudo: add-apt-repository: command not found
do the following:
sudo apt-get install software-properties-common
And then rerun adding your repository.

Note that it is possible to install multiple Java installations on one machine, and set one of installed versions as the default. Check out How To Install Java with Apt-Get on Ubuntu 16.04 (April 23, 2016)  (pdf), in particular the “Managing Java” section.

  • Verify Installed Java Version

After successfully installing Oracle Java, use the following to verify what version we installed.

$ java -version 

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
  • Configuring Java Environment and Set the JAVA_HOME Environment Variable

We also need to install java configuration package. The package should come with the latest operating systems during installation of JAVA packages. But it does no harm to run the following command to be sure we have it installed on our machine.

$ sudo apt-get install oracle-java8-set-default

Many programs use the JAVA_HOME environment variable to determine the Java installation location.

Copy the path from your preferred installation and then open /etc/environment configuration file using  nano or your favorite text editor, to set JAVA_HOME environment variable.

sudo nano /etc/environment

At the end of this file, add the following line, making sure to replace the highlighted path with your own copied path.


JAVA_HOME=/usr/lib/jvm/java-8-oracle

Save and close the file and exit nano editor environment. (Note: Ctrl+O to save the file, and then hit Enter, and then Ctrl +X to close and exit the file.)

Use the following command to reload the file.

  • source /etc/environment

You can now test whether the environment variable has been set by issuing the following command:

echo $JAVA_HOME

This will return the path you just set.

  • Conclusion

We have now installed Java 8 on our system and set it as default. We can now install software which runs on Java, such as Tomcat and Solr.

 

References:

How To Install Java with Apt-Get on Ubuntu 16.04 (April 23, 2016)  (pdf)

This is a very good post, it introduced the installation of both OpenJDK and Oracle JDK 6/7/8/9

How to Install Oracle JAVA 8 (JDK/JRE 8u121) on Ubuntu & LinuxMint with PPA (Mar 29, 2017 by Rahul K.)  – pdf

 

Using Apache Solr with Python

This post provides the instructions to use Apache Solr with Python in different ways.

======using Pysolr

Below are two small python snippets that the author of the post used for testing writing to and reading from a new SOLR server.

The script below will attempt to add a document to the SOLR server.

# Using Python 2.X
from __future__ import print_function  
import pysolr

# Setup a basic Solr instance. The timeout is optional.
solr = pysolr.Solr('http://some-solr-server.com:8080/solr/', timeout=10)

# How you would index data.
solr.add([  
    {
        "id": "doc_1",
        "title": "A very small test document about elmo",
    }
])

The snippet below will attempt to search for the document that was just added from the snippet above.

# Using Python 2.X
from __future__ import print_function  
import pysolr

# Setup a basic Solr instance. The timeout is optional.
solr = pysolr.Solr('http://some-solr-server.com:8080/solr/', timeout=10)

results = solr.search('elmo')

print("Saw {0} result(s).".format(len(results)))  

 

======GitHub repos

pysolr is a lightweight Python wrapper for Apache Solr. It provides an interface that queries the server and returns results based on the query.

install Pysolr using pip

pip install pysolr

Multicore Index

Simply point the URL to the index core:

# Setup a Solr instance. The timeout is optional.
solr = pysolr.Solr('http://localhost:8983/solr/core_0/', timeout=10)

SolrClient is a simple python library for Solr; built in python3 with support for latest features of Solr.

Components of SolrClient

 

References: