Month: April 2017

This post provides the tutorial to set up Apache Solr 6 on Ubuntu 16.04. (install Solr as a service that auto-starts when (re)boot Ubuntu.)

What is Apache Solr? Apache Solr is an open source enterprise-class search platform written in Java which enables you to create custom search engines that index databases, files, and websites. It has back end support for Apache Lucene. It can, for example, be used to search in multiple websites and can show recommendations for the searched content. Solr uses an XML (Extensible Markup Language) based query and result language. There are APIs (Applications program interfaces) available for Python, Ruby and JSON (Javascript Object Notation).

Some other features that Solr provides are:

• Full-Text Search.
• Snippet generation and highlighting.
• Custom Document ordering/ranking.
• Spell Suggestions.

This tutorial will show you how to install the latest Solr version on Ubuntu 16.04 LTS. The steps will most likely work with later Ubuntu versions as well.

Before Solr 5, Solr doesn’t work alone; it needs a Java servlet container such as Tomcat or Jetty. But after Solr 5, it does not need to run on Tomcat.  

Running Solr on Tomcat (No Longer Supported)

Things need to do when installing Solr version before 6.

Download and install Tomcat (or some other servlet container)
Setup Tomcat as a service
Download and unpack Solr
Create a SOLR_HOME folder with correct content
copy solr.war into tomcat/webapps
set CATALINA_OPTS=“-Dsolr.solr.home=/path/to/home -Dsolr.x.y=z…. GC-flags etc”
Setup  Tomcat as a service
service tomcat start

With Solr 6.x, we just need to do:

Download Solr and unpack the install-script
solr/bin/install_solr_service solr-6.2.0.tgz  # Install
Tune /etc/default/solr.in.sh to your likings (mem, port, solr-home, Zk etc)
service solr start (or bin/solr start [options])

Your client would talk to Solr on typically http://host.name:8983/solr/ as a standalone server, not as one out of many webapps on 8080.

Apache Solr 6 required Java 8 or greater to run.

 There had been lots of scaling improvements in Solr 6.

Now let’s get started with the installation.

Step 1: Update your System

Use a non-root sudo user to login into your Ubuntu server. Through this user, you will have to perform all the steps and use the Solr later.

To update your system, execute the following command to update your system with latest patches and updates.

$ sudo apt-get update 
$ sudo apt-get upgrade -y   #note that this will update your ubuntu OS, skip this if you do not want to update your system.

Step 2: Install Java 

(Apache Solr 6 required Java 8 or greater to run. If you have installed Java 8 or greater on your machine, skip this.)

Solr is a Java application, so Java needs to be installed first in order to set up Solr. See my post for detailed Java 8 installation on Ubuntu 16.04.

Check the version of Java installed by running the following command

$ java -version

java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

Step 3: (Manually) install Solr 

Solr can be installed on Ubuntu in different ways, in this tutorial,we will install the latest package.  (If you would like to install the latest package from the source, check out How to install and configure Solr 6 on Ubuntu 16.04.)

Now Let’s download the required Solr version from its official site or mirrors.

First go to this Solr Download page, click the link to the latest version.

You would probably see something looks like the pic shown below. Get the download link you prefer. (for my case, I used this one http://apache.cs.utah.edu/lucene/solr/6.5.1). Click the download link you selected, and then you would see something like the pic shown below.

#If you do not have sudo privilege
#cd /path to one folder under your account 
# and you do not need to add "sudo" in the following commands
  
cd /opt
sudo wget http://apache.cs.utah.edu/lucene/solr/6.5.1/solr-6.5.1.tgz

Now extract solr service installer shell script from the downloaded Solr archive file and run installer using following commands.

sudo tar xzf solr-6.5.1.tgz solr-6.5.1/bin/install_solr_service.sh --strip-components=2

Then install Solr as a service using the script:

sudo ./install_solr_service.sh solr-6.5.1.tgz

The output will be similar to this: [Note that this installation will make Solr as a service that auto-starts when you (re)boot Ubuntu.]

myusername@myserver:/opt$ sudo ./install_solr_service.sh solr-6.5.1.tgz
id: ‘solr’: no such user
Creating new user: solr
Adding system user `solr’ (UID 117) …
Adding new group `solr’ (GID 126) …
Adding new user `solr’ (UID 117) with group `solr’ …
Creating home directory `/var/solr’ …

Extracting solr-6.5.1.tgz to /opt

Installing symlink /opt/solr -> /opt/solr-6.5.1 …

Installing /etc/init.d/solr script …

Installing /etc/default/solr.in.sh …

Service solr installed.
Customize Solr startup configuration in /etc/default/solr.in.sh
● solr.service – LSB: Controls Apache Solr as a Service
Loaded: loaded (/etc/init.d/solr; bad; vendor preset: enabled)
Active: active (exited) since Sun 2017-04-30 11:08:43 EDT; 5s ago
Docs: man:systemd-sysv-generator(8)
Process: 2652 ExecStart=/etc/init.d/solr start (code=exited, status=0/SUCCESS)

Apr 30 11:08:34 myserver systemd[1]: Starting LSB: Controls Apache Solr as a Service…
Apr 30 11:08:34 myserver su[2655]: Successful su for solr by root
Apr 30 11:08:34 myserver su[2655]: + ??? root:solr
Apr 30 11:08:34 myserver su[2655]: pam_unix(su:session): session opened for user solr by (uid=0)
Apr 30 11:08:42 myserver solr[2652]: [194B blob data]
Apr 30 11:08:42 myserver solr[2652]: Started Solr server on port 8983 (pid=2861). Happy searching!
Apr 30 11:08:43 myserver solr[2652]: [14B blob data]
Apr 30 11:08:43 myserver systemd[1]: Started LSB: Controls Apache Solr as a Service.
myusername@myserver:/opt$

Step 4:  Start / Stop Solr Service

Use the following command to check the status of the service

$ sudo service solr status

See below for a sample output:

myusername@myserver:/opt$ sudo service solr status
● solr.service - LSB: Controls Apache Solr as a Service
   Loaded: loaded (/etc/init.d/solr; bad; vendor preset: enabled)
   Active: active (exited) since Sun 2017-04-30 11:08:43 EDT; 13min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 2652 ExecStart=/etc/init.d/solr start (code=exited, status=0/SUCCESS)

Apr 30 11:08:34 myserver systemd[1]: Starting LSB: Controls Apache Solr as a Service...
Apr 30 11:08:34 myserver su[2655]: Successful su for solr by root
Apr 30 11:08:34 myserver su[2655]: + ??? root:solr
Apr 30 11:08:34 myserver su[2655]: pam_unix(su:session): session opened for user solr by (uid=0)
Apr 30 11:08:42 myserver solr[2652]: [194B blob data]
Apr 30 11:08:42 myserver solr[2652]: Started Solr server on port 8983 (pid=2861). Happy searching!
Apr 30 11:08:43 myserver solr[2652]: [14B blob data]
Apr 30 11:08:43 myserver systemd[1]: Started LSB: Controls Apache Solr as a Service.

Use the following commands to Start, Stop and check status of Solr service.

$ sudo service solr stop
$ sudo service solr start
$ sudo service solr status

Step 5: Creating a Solr search collection

(Before we create a Solr search collection, check out this post first if you want to change the default port 8983 to another port.)

Using Solr, we can create multiple collections. Run the following command, give the name of your collection (here mysolrcollection) and specify its configurations.

$ sudo su - solr -c "/opt/solr/bin/solr create -c mysolrcollection -n data_driven_schema_configs"

Sample output:

myusername@myserver:/opt$ sudo su - solr -c "/opt/solr/bin/solr create -c mysolrcollection -n data_driven_schema_configs"
 [sudo] password for myusername:

Copying configuration to new core instance directory:
 /var/solr/data/mysolrcollection

Creating new core 'mysolrcollection' using command:
 http://localhost:8983/solr/admin/cores?action=CREATE&name=mysolrcollection&instanceDir=mysolrcollection

{
 "responseHeader":{
 "status":0,
 "QTime":1422},
 "core":"mysolrcollection"}

The new core directory for our first collection has been created. To view the default schema file, got to:

cd /opt/solr/server/solr/configsets/data_driven_schema_configs/conf

You will see some files shown in the picture below.

To view other configuration options , got to:

cd /opt/solr/server/solr/configsets/

Step 6: Use the Solr Web Interface (i.e., Access Solr Admin Panel)

Default Solr runs on port 8983. You can access Solr port in your web browser and you will get Solr dashboard.

The Apache Solr is now accessible on the default port, which is 8983. The admin UI should be accessible at http://your_server_ip:8983/solr. The port should be allowed by your firewall to run the links. 

(If you do not know your IP, check my post to find it out.)

For example:

http://192.168.1.100:8983/solr/

Or use your machine’s host name if you have one.

http://example.org:8983/solr/

Here you can view statics of created collection in previous steps named “mycollection”. Click on “Core Selector” on left sidebar and select created collection.

To see the details of the first collection that we created earlier, select the “mysolrcollection” collection in the left menu.

After you selected the “mysolrcollection” collection, select Documents in the left menu. There you can enter real data in JSON format that will be searchable by Solr. To add more data, copy and paste the following example JSON onto Document field:

{
"id": 1,
"name":"John",
"age":30,
"cars":[ "Ford", "BMW", "Fiat" ]
}

Note: You can add other formats of data such as CSV etc to Solr. (See the pic below)

Click on the submit document button after adding the data.

Status: success
Response:
{
 "responseHeader": {
 "status": 0,
 "QTime": 758
 }
}

Now we can click on Query on the left side then click on Execute Query,

We will see something like this:

Conclusion

After successfully installing the Solr Web Interface on Ubuntu, you can now insert the data or query the data with the Solr API and Web Interface.

You can write code to add a large set of documents into Solr. See my post for using Solr with Python. See this post for some useful Solr resources I collected.

References:

• Installing Solr
• Solr tutorial
• How To Install Apache Solr 6.2 on Ubuntu, Debian & LinuxMint (Aug 27, 2016 by Rahul K.) (pdf) – Nice tutorial. It didn’t say directly in the text, but this will install Solr as a service that auto-starts when you (re)boot Ubuntu.
• How to install Latest version of Apache Solr on Ubuntu and how to uninstall (pdf)
• Configure Solr on Ubuntu, the quickest way  (pdf)

Install Tomcat & Solr (You can’t avoid this one) – This is for Solr before version 5, after Solr 5, Tomcat is not required to install Solr.

Apache Solr Reference Guide/ Installing Solr  & Running Solr  & Solr Quick Start (pdf. a very good concise intro, including some basic usages and indexing xml, json, csv files)

• How to install and configure Solr 6 on Ubuntu 16.04 (pdf)
• Switching Solr from (the default) port 8983 (pdf)
• How To Install Solr on Ubuntu 14.04  (including a short introduction to configuring a schema.xml for Solr)

Configuring a schema.xml for Solr

First, rename the /opt/solr/solr/collection1 to an understandable name like apples (use whatever name you’d like). (This can be skipped if you installed it using apt-get. In that case, you can execute the following command instead: cd /usr/share/solr):

cd /opt/solr/solr
mv collection1 apples
cd apples

Also, if you installed Solr manually, open the file core.properties (nano core.properties) and change the name to the same name.

Then, remove the data directory and change the schema.xml:

rm -R data
nano conf/schema.xml

Paste your own schema.xml in here.

• How To Install Apache Tomcat 8 on Ubuntu 16.04
• How to start Solr automatically? 
• https://www.howtoinstall.co/en/ubuntu/xenial/solr-tomcat
• SolrTomcat – Solr Wiki
• How to install Solr on Ubuntu and auto start at boot
• Using Python

Elasticsearch and Apache Solr are open source search engines, and they are the most widely used search servers. This post provides resources about Apache Solr.

Apache Solr is a fast open-source Java search server.

Solr enables you to easily create search engines which searches websites, databases and files.

Solr (pronounced “solar”) is an open source enterprise search platform, written in Java, from the Apache Lucene project. Its major features include full-text search, hit highlighting, faceted search, real-time indexing, dynamic clustering, database integration, NoSQL features and rich document (e.g., Word, PDF) handling. Providing distributed search and index replication, Solr is designed for scalability and fault tolerance. Solr is the second-most popular enterprise search engine after Elasticsearch.

Solr runs as a standalone full-text search server. It uses the Lucene Java search library at its core for full-text indexing and search, and has REST-like HTTP/XML and JSON APIs that make it usable from most popular programming languages. Solr’s external configuration allows it to be tailored to many types of application without Java coding, and it has a plugin architecture to support more advanced customization.

An Elasticsearch / Apache Solr index is the equivalent of a SQL table.

An Elasticsearch or Solr server (aka Solr instance, aka Solr engine) can maintain several indexes.

(Elasticsearch index configuration is done with HTTP / JSON commands. No files required. You define types, mappings, analysis with simple commands.)

In Apache Solr, each index is defined by a schema.xml file (it’s not mandatory in Solr 5/6, but recommended in production), and a solrconfig.xml file. The index schema is equivalent to a SQL table schema definition.  (See this post for Solr Schema related resources.)

An index contains several documents, equivalent to SQL table rows. Each document contains fields, equivalent to SQL table columns.

When an index document is inserted/updated/deleted, we say it is “indexed”.

To retrieve documents from an index, Elasticsearch (json) / Apache Solr (xml, json) provide an http API, with a proprietary syntax.

Elasticsearch and Apache Solr are web applications. A client will use their http API to query or store data.

A full-text search engine is built from the ground to tackle problems that a SQL search find difficult or impossible. The list of those features is huge: multi-language, dedicated plugins to extend the engine, synonyms, stop words, facets, boosts, …

The core search engine of Elasticsearch and Apache Solr is Apache Lucene. The relationship between Elasticsearch / Apache Solr and Lucene, is like that of the relationship between a car and its engine.

You can access Solr admin from your browser: http://localhost:8983/solr/

use the port number used in installation.

See below for some useful Solr related resources:

• Apache Solr Reference Guide 

• Getting Started, 
• Using the Solr Administration User Interface,
• Client APIs,
• Understanding Analyzers, Tokenizers, and Filters,
• Indexing and Basic Data Operations, 
• Searching, 
• The Well-Configured Solr Instance, 
• Documents, Fields, and Schema Design, 
• Solr Glossary,
• Managing Solr

• Install Apache Solr on Ubuntu 16.04
• Using Apache Solr with Python
• Configuring the Solr 6 using Admin Console
• Apache Solr Tutorial on Tutorialspoint
• Solr Concept and Architecture (pdf)

• Solr Tutorial (pdf) — this tutorial is very good, by the Solr creator Yonik Seeley

Check out his Unofficial Solr Guide (e.g., Solr 6.5 Features)

• Solr in 5 minutes (pdf) (check more Solr resources on this website, see the right navigation bars. See below for some I selected.)

• Basic Concepts
• Solr Query Syntax
• Lucene Query Builder

Configuring

• Configuring Solr
• schema.xml
• solrconfig.xml

Integrating Solr

• SolrJ

• Top 10 Performance Tips for Apache SOLR (pdf)
• Apache Solr resources on Khai’s personal knowledge vault (pdf) — It contains many pretty good resources and concise explanations of some Solr usages such as fuzzy search and facet.
• What is an Elasticsearch / Apache Solr index ? (pdf)
• What are Elasticsearch and Apache Solr ? (pdf)
• How does SOLR work? What is an explanation for the principle in layman’s terms?  (pdf) — It gave pretty good explanation how Solr works, including schema explanation.
• Basic Elasticsearch Concepts (pdf)
• Importing/Indexing database (MySQL or SQL Server) in Solr using Data Import Handler