Configuring Apache Solr Multi-core With Drupal and Tomcat on Ubuntu 9.10

One of the things I've had to do recently is set up a development server that has multiple sites with Apache Solr so that each site has its own index in Solr.  After searching the internets for information on how to do this, I found two sources that contained part of the process, but not put together in a way that worked for me.  
 

Nick's post got me through the Tomcat setup with Solr for one site, but his step for multi-site involved making copies of the solr.war file. I noticed Peter Wolanin's comment that it would be better to use the solr multi core feature to serve the 2 indexes at different paths, so I decided to try and implement it that way.  I found the handbook page, but it was not written for Tomcat on Ubuntu, so I had to combine them.  And then of course, I knew I had to blog about it. 

All of these steps (except for a couple minor modifications) are copied from the two sources listed above.  I am doing this on Ubuntu 9.10, the same as Nick.

So here we go.  Please feel free to raise your hand if you have any questions.
 

Step 1. Installing Tomcat

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install tomcat6 tomcat6-admin tomcat6-common tomcat6-user tomcat6-docs tomcat6-examples

This command downloads and install the following packages
 

tomcat6 : Servlet and JSP engine
tomcat6-admin : Admin web applications
tomcat6-common : Common files
tomcat6-user : Tools to create user instances
tomcat6-docs : Example web applications
tomcat6-examples : Example web applications

Start tomcat by typing
 

/etc/init.d/tomcat6 start

Step 2. Security

If you are using IP tables and installing Apache Solr on an external server, modify or add the following line to accept the port 8080:
 

-A INPUT -p tcp -m tcp --dport 8080 -j ACCEPT

After installation go to http://localhost:8080 or http://serverip:8080 in your browser. Now you should see the Tomcat welcome page.

PS: As noticed by Robert Douglass, be careful when opening ports in public servers. Apache Solr Does NOT provide security by default so anyone is able to post data towards your system. Best thing to do is to add another rule where you only accept 8080 connections from a certain IP address.
 

Step 3. Downloading Solr

Download apachesolr to your home directory for temporary reasons.  Get it from one of the mirrors at http://www.apache.org/dyn/closer.cgi/lucene/solr/.  To use the recommended mirror:
 

cd ~/
wget <a href="http://www.motorlogy.com/apachemirror/lucene/solr/1.4.0/apache-solr-1.4.0.tgz
tar" title="http://www.motorlogy.com/apachemirror/lucene/solr/1.4.0/apache-solr-1.4.0.tgz
tar">http://www.motorlogy.com/apachemirror/lucene/solr/1.4.0/apache-solr-1.4....</a> -zxvf apache-solr-1.4.0.tgz

Step 4. Linking Tomcat6 with Apache Solr application

This should give you an idea on where your distribution installed tomcat6.
Attention: If your path is different, do not forget to also adjust this in the next steps.  whereis tomcat6 should show you tomcat6:
 

/etc/tomcat6 /usr/share/tomcat6

Copy the solr.war file to the webapps directory:
 

sudo cp apache-solr-1.4.0/dist/apache-solr-1.4.0.war /usr/share/tomcat6/webapps/solr.war

Copy the example solr application to a new directory called solr. We will change this example solr application later on to be viable for Drupal 6:
 

sudo cp -R apache-solr-1.4.0/example/solr/ /usr/share/tomcat6/solr/

Create our config file

sudo nano /etc/tomcat6/Catalina/localhost/solr.xml

And fill it with the following configuration :

<Context docBase="/usr/share/tomcat6/webapps/solr.war" debug="0" privileged="true" allowLinking="true" crossContext="true">
<Environment name="solr/home" type="java.lang.String" value="/usr/share/tomcat6/solr" override="true" />
</Context>

Step 5. Managing Tomcat6 application

We want to be able to see how and/or if our Solr application is running, and we can do this by using the manager application. By default you don't have access to this application so we have to modify the permissions:
 

sudo nano /etc/tomcat6/tomcat-users.xml

And modify it so it more or less reflects the same information as shown here.
 

<tomcat-users>
<role rolename="admin"/>
<role rolename="manager"/>
<user username="nick" password="ateneatech" roles="admin,manager"/>
</tomcat-users>

Drop Tomcat security so Solr can access /usr/share/tomcat6/solr:
 

sudo nano /etc/default/tomcat6

And modify it so our security is disabled. Be careful if you are running on a server which you do not control 100%!
 

TOMCAT6_SECURITY=no

We'll restart Tomcat after we got Solr multi-core installed.
 

Step 6. Linking Drupal 6 with a running Apache Solr

I assume you have Drush installed, so we continue with downloading the apachesolr module. Execute this command in the designated website:
 

drush dl apachesolr

You will also need to get the Solr PHP Client library (available at http://code.google.com/p/solr-php-client/downloads/list) and unpack it so it is under the apachesolr module directory as SolrPhpCLient (i.e. sites/all/modules/apachesolr/SolrPHPClient)

Let's copy our schema files that will customize our Apache Solr instance so it fits the "Drupal" bill.
 

sudo cp ~/apachesolr/schema.xml /usr/share/tomcat6/solr/conf/schema.xml
sudo cp ~/apachesolr/solrconfig.xml /usr/share/tomcat6/solr/conf/solrconfig.xml

Step 7. Set up up the multi-core functionality

First, we copy the solr.xml that is configured for mutli-core to the /solr directory:
 

sudo cp ~/apache-solr-1.4.0/example/multicore/solr.xml /usr/share/tomcat6/solr/solr.xml

Now create a directory within the /usr/share/tomcat6/solr directory for each site you want to use Solr with.  For example, if I had two sites "site1.com" and "site2.com" and I wanted to use Solr with them both, I might create the following directories:
 

/usr/share/tomcat6/solr/site1
/usr/share/tomcat6/solr/site2

Copy the /usr/share/tomcat6/solr/conf directory into each directory you just created. When you're done, each site's directory should have a copy of /conf in them:
 

sudo cp /usr/share/tomcat6/solr/conf /usr/share/tomcat6/solr/site1/conf
sudo cp /usr/share/tomcat6/solr/conf /usr/share/tomcat6/solr/site2/conf

Note: DO NOT delete the /usr/share/tomcat6/solr/conf directory.  Even though there is a copy for each site, it is still needed by the Solr core.  For stock Solr, this directory can be deleted, but it is needed when running it in Drupal.

Now alter the solr.xml file you copied over to list both sites.  Your solr.xml should look like this:
 

<?xml version="1.0" encoding="UTF-8" ?>
<solr persistent="false">
<cores adminPath="/admin/cores">
  <core name="site1" instanceDir="site1" />
  <core name="site2" instanceDir="site2" />
</cores>
</solr>

OK, let's restart our Tomcat service:
 

sudo /etc/init.d/tomcat6 restart

Surf to http://localhost:8080/manager/html and log in with your username and password from above to check if the Solr instance is started. If not, start and it and check wether or not you receive an error code.  If your application is started, surf to http://localhost:8080/solr/admin and you should see a nice screen!
 

Step 8. Configure the Drupal modules

Enable the "Apache Solr framework" and "Apache Solr search" modules. Also, enable the core Drupal Search module if you haven't already. Do this for both sites.

Navigate to the Solr Settings page for each site (admin/settings/apachesolr/settings):
- Solr host name: localhost
- Solr port: 8080
- Solr path for site1.com: /solr/site1
- Solr path for site2.com: /solr/site2

Under "Advanced Configuration", enabled the "Make Apache Solr Search the default" option if you want to replace the default search with Solr (and I assume you do).

Click "Save Configuration."  The first time around it'll probably tell you it can't reach the server, but if you refresh the page you'll be good to go.

Now, whenever you add a new site, you will just need to do the following steps to enable Solr search:

  1. Copy /usr/share/tomcat6/solr/conf to /usr/share/tomcat6/solr/mysite
  2. Add the site to /usr/share/tomcat6/solr/solr.xml
  3. Configure Solr settings in Drupal at admin/settings/apachesolr

And that's it.  Happy searching!

 

Yay, happy to see somebody

Yay, happy to see somebody writing a good post about this - took me some time figuring this out by myself and (sadly) I never got to writing a post like this.

Something some people will need to know: in the last step, the "solr hostname" should be "nick:ateneatech@localhost"

Cheers for this tutorial! I'm sure it will help a lot of people!

Hmmm, not in my experience. I

Hmmm, not in my experience. I have this running on my local laptop as well as an Ubuntu server, and in each case, the value for "Solr host name" is just "localhost."

Thanks for writing this up! I

Thanks for writing this up! I was willing to do this but had some time shortage!
Also I included your post in the multiple solr part of my tutorial.

Thanks again!

I`ll try this on my weekend.

I`ll try this on my weekend. Thanks for tutorial.

Great write up Steve. It

Great write up Steve. It inspired me to finally blog my notes on multi core solr with Ubuntu 10.04 (aka Lucid Lynx).
While I was at it I wrote a custom core admin handler so I can have Solr automagically create cores on the fly. The post is available at http://davehall.com.au/blog/dave/2010/06/26/multi-core-apache-solr-ubunt...

Execelent post. There is

Execelent post.
There is only one thing: I'm missing the part about the config file for solr (/etc/tomcat6/Catalina/localhost/solr.xm).

Oops, good catch. I updated

Oops, good catch. I updated the post.

I can't see the update in the

I can't see the update in the text body: did you put it in?

For other readers' reference, the XML file needs to live at /etc/tomcat6/Catalina/localhost/solr.xml (not solr.xm) and here's information about what should go in it: http://www.nickveenhof.be/blog/setup-drupal-6-apache-solr-tomcat6-and-ub...

There's also some typos of tomcast6 and /mysite1, /mysite2 that people might stumble across.

Finally there.

Finally there.

Thank you so much for the

Thank you so much for the great write up. It really helped!

great Post, but I also miss

great Post, but I also miss the configuration part with the Catalina path (/etc/tomcat6/Catalina/localhost/solr.xml). Is this not necessary or why did you not mentioned it above?

If you get an error 500 and

If you get an error 500 and something like "can´t find solrconfig.xml in classpath", you´ll have to uncomment the setting for the environment variable in tomcat/webapps/solr/WEB-INF/web.xml like:

solr/home
/put/your/solr/home/here
java.lang.String

I tried all of the steps

I tried all of the steps above, but still get an error 404 - The requested resource (/solr/admin) is not available when I try to access localhost:8080/solr/admin.

Tomcat is working fine (I can access the manager), but am having no luck getting solr to appear. Any troubleshooting tips?

it is showing me the sites

it is showing me the sites which i have coded in solr.xml

<?xml version="1.0" encoding="UTF-8" ?>

only,
- Solr host name: localhost
- Solr port: 8080

not display the lines below there
- Solr path for site1.com: /solr/site1
- Solr path for site2.com: /solr/site2
- Solr path for site3.com: /solr/site3

If you've gone through this

If you've gone through this tutorial and Drupal is telling you that it can't connect to Solr, you may have a permissions issue. Sniff around in /var/logs/tomcat6, particularly catalina.out and look for issues. Tomcat may not have the write permissions necessary to create the index within each core.

Another giveaway is if you're seeing the message "missing core name in path" when you try to view the Solr instances from the Tomcat Manager (http://localhost:8080/solr)

I had to execute the following to get things working on Ubuntu 10.04LTS:

Set the solr directory to be part of the tomcat6 group
sudo chgrp -R tomcat6 /usr/share/tomcat6/solr

Change the read/write perms in the solr directory
sudo chmod -R 2750 /usr/share/tomcat6/solr

Make each site's folder writable
sudo chmod -R 2770 /usr/share/tomcat6/solr/site1

Thanks for the helpful tips,

Thanks for the helpful tips, Chris.  The last time I did this setup, I ran into some permissions problems just like that, but didn't think to update this post.

Nice how to and very detail

Nice how to and very detail articles.
still get some error in my end but i try to do review it from above
thanks :)

Thanks for great detailed

Thanks for great detailed post.
I got up and running tomcat but not SOLR, i think there are some issues with solrconfig.xml, but dont know how to fix this. I am getting following error. Can you please help

HTTP Status 500 - Severe errors in solr configuration. Check your log files for more detailed information on what may be wrong. If you want solr to continue after configuration errors, change: false in solr.xml ------------------------------------------------------------- java.lang.RuntimeException: org.apache.lucene.index.IndexNotFoundException: no segments* file found in org.apache.lucene.store.MMapDirectory@/usr/share/tomcat6/solr/site1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@a4133b0: files: [] at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1099) at org.apache.solr.core.SolrCore.(SolrCore.java:585) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:463) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:316) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:207) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:130) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:94) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(Application

and 10-15 more lines

Attribution, please. It's

Attribution, please. It's kind of obvious that much of this was paraphrased or plagiarized from the article I published on the Four Kitchens wiki.

Specifically, from the Four Kitchens page:
"On Ubuntu, drop Tomcat security so Solr can access /var/solr."

And on here:
"Drop Tomcat security so Solr can access /usr/share/tomcat6/solr:"

You just don't get such similar phrasing by coincidence.

Thanks,
David

David, I made it very clear

David,

I made it very clear at the beginning of the article where I got my information, and even linked to them. If you go back and read the two articles, you will see that everything was taken from there.  I'm not sure how much clearer I could have been.  In particular if you go back and look at Nick's blog post, step 5 has the (trademarked?) phrase.  And, if you look at the "Sources" section at the bottom of Nick's post, you will see that he specifically lists your wiki as one of his sources.

So, am I supposed to provide attributions for Nick's post, too?

Steve

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd><img>
  • Lines and paragraphs break automatically.

More information about formatting options

By submitting this form, you accept the Mollom privacy policy.