Java memory heap limits causing issues with Solr crawls completing

  • Updated

In some cases, you may see crawls not complete after first registering a site or after it has been working for some time.  Here is how to identify the cause and fix the issue.

First, check the Solr log files for OutOfMemory and lock exceptions. Recursively searching using the find in files option of Notepad++ is good for this. 

 

If you see file lock errors, check your anti-virus whitelisting.
Antivirus and Solr


If you see memory heap exceeded or database disconnect errors (found in by searching for OutOfMemory), there may be insufficient memory allocated to the java heap and the following steps can be taken depending on which process the out of memory occurs in.

 

Tomcat

  1. On your Solr server, navigate to 'C:\Program Files (x86)\Ektron\Search2.0\Tomcat\bin'.
  2. Locate the file 'setenv.bat', create a back-up, then open it for editing.
  3. Locate the 'JAVA_OPTS=%JAVA_OPTS%' string, identify the '-Xmx' key (ie: -Xmx1280m).
  4. Increase the value by at least 500MB.
  5. Stop then start the Ektron Solr Process Manager 2.0 service.

ManifoldCF

  1. On your Solr server, navigate to 'C:\Program Files (x86)\Ektron\Search2.0\ManifoldCF\core\options.env'.
  2. Locate the '-Xmx' key (lines with REM are commented out so be sure to ignore those).
  3. Increase the value by at least 500MB.
  4. Do the same thing for largeoptions.env, dbgoptions.env and lightoptions.env. 
  5. Stop then start the Ektron Solr Process Manager 2.0 service.

HSQL

  1. On your Solr server, navigate to 'C:\Program Files (x86)\Ektron\Search2.0\ManifoldCF\core\start-database.bat'.
  2. Locate the '-Xmx' key(lines with REM are commented out so be sure to ignore those).
  3. Increase the value by at least 500MB.
  4. Stop then start the Ektron Solr Process Manager 2.0 service.

Solr (Not in versions prior to 9.2)

  1. On your Solr server, navigate to 'C:\Solr\bin\solr.cmd' or if on an earlier version 'C:\Program Files (x86)\Ektron\Search2.0\Solr\bin\solr.cmd'.
  2. Locate the '-Xmx' key(lines with REM are commented out so be sure to ignore those).
  3. Increase the value by at least 500MB.
  4. Do the same process with '\bin\solr.in.cmd'
  5. Stop then start the Ektron Solr Process Manager 2.0 service.
  6. After doing those steps, re-evaluate if memory limits are being exceeded and react accordingly. 

Keep in mind that there must be sufficient memory on the server for these steps to be effective. If you cannot add 500MB due to the server's available memory, you may want to increase memory on the box or lower the -Xmx for Java processes that do not use much memory. 

Note: Only one restart of the Ektron Solr Process Manager 2.0 service is needed if multiple Java memory limits are changed.

Important: The above steps are for raising the memory, but the opposite may be needed if overall memory usage is too high on the server (identified through task manager). Lowering -Xmx values will free up memory on the server but make less memory available to that java process. Keep in mind that the -Xmx values should exceed the -Xms value (the amount of memory the process starts with). 


After restarting process manager and crawling, check the logs again. If the adjustments fail to resolve the OutOfMemory errors in the log, then a more precise method may be needed to get the heap size. 

The best way to determine the correct size is to analyze the garbage collection (GC) logs located in your logs directory. There are various tools that help analyze these logs and, in particular, show the amount of memory used after GC has completed (GCViewer and GCEasy are two). Also you can attach jconsole (distributed with most Java runtimes) to check memory consumption as Solr is running. This will show the absolute minimum amount of memory required; adding 25-50% "headroom" is a reasonable starting point.

source: JVM Settings | Apache Solr Reference Guide

Based on the heap size, raise the -xmx value with headroom.  

 

After fixing the heap values clear the crawl database and reregister the site. 

Clearing the Solr Crawl Database

To confirm resolution, check the heap memory while the crawl is running, or check the logs for OutOfMemory and raise as needed. If you still see the issue, run the log collector and provide the results to Optimizely Support.