Java memory heap limits causing issues with Solr crawls completing

  • Updated

In some cases, you see crawls not complete on the server and there is no obvious reason. For instance, on the Apache manifold page, you might see a start time for a crawl but no end time after 5+ hours. Here is how to identify the cause and fix the issue.

First check the Manifold logs on the search server(location below) and search for exception.

"C:\Program Files (x86)\Ektron\Search2.0\ManifoldCF\logs"

If you see file lock errors, check your anti-virus whitelisting.
Antivirus and Solr

If you see memory heap exceeded or database disconnect errors, there may be insufficient memory on the server. These can occur because of the Java heap memory limits on the 4 java processes (3 with versions prior to 9.2) used by Solr. There's one java process for HSQL, Manifold, Solr and Tomcat (Solr was added in 9.2). You can increase the memory limits of these processes to resolve this type of issue.

First, though, you need to identify which java process is having a memory limit issue. Once the problem occurs, install process explorer on the server.

https://technet.microsoft.com/en-us/sysinternals/processexplorer.aspx

Then do the following.

  • Run process explorer as an administrator on the Solr box.
    • Take note of each java process type(hsql, manifold, solr or tomcat) and its memory limit. Here are the steps to do that.
    • Mouse over each java process one at a time. A command line window appears.
    • At the end of each command line, you see references to files. Based on the references, you can determine what type of Java process is being used.
    • There is a -Xmx value at the beginning of the command line, which is the Java memory limit max in MB. 

Identify the working set memory of each java process.

If the working memory set amount exceeds the memory limit, the -Xmx value may need to be raised. For instance, if the -Xmx value for the Tomcat process is 1024M(M is MBs) and the working memory is 3,211,000K, the -Xmx value for Tomcat needs to be raised to at least 3220M. Likewise, if the -Xmx value is vastly above the working memory the -Xmx should be lowered. 

For better proof you can also check the Solr log files which will give an OutOfMemory exception if a limit has been exceeded. 

See below for how to raise and lower those memory limits. Keep in mind that there must be sufficient memory on the server for these steps to be effective. If you cannot add 500MB due to the server's available memory, you may want to increase memory on the box or lower the -Xmx for Java processes that do not use much memory. It is good to have some headroom in case the memory needs increasing as well.

Note: Only one restart of the Ektron Solr Process Manager 2.0 service is needed if multiple Java memory limits are changed.

Important: The below steps are for raising the memory but the opposite could be done if overall memory usage is too high on the server(identified through task manager). Lowering -Xmx values will free up memory on the server but make less memory available to that java process. Keep in mind that the -Xmx values should exceed the -Xms value(the amount of memory the process starts with). 

Tomcat

  1. On your Solr server, navigate to 'C:\Program Files (x86)\Ektron\Search2.0\Tomcat\bin'.
  2. Locate the file 'setenv.bat', create a back-up, then open it for editing.
  3. Locate the 'JAVA_OPTS=%JAVA_OPTS%' string, identify the '-Xmx' key (ie: -Xmx1280m).
  4. Increase the value by at least 500MB.
  5. Stop then start the Ektron Solr Process Manager 2.0 service.

ManifoldCF

  1. On your Solr server, navigate to 'C:\Program Files (x86)\Ektron\Search2.0\ManifoldCF\core\options.env'.
  2. Locate the '-Xmx' key (lines with REM are commented out so be sure to ignore those).
  3. Increase the value by at least 500MB.
  4. Do the same thing for largeoptions.env, dbgoptions.env and lightoptions.env. 
  5. Stop then start the Ektron Solr Process Manager 2.0 service.

HSQL

  1. On your Solr server, navigate to 'C:\Program Files (x86)\Ektron\Search2.0\ManifoldCF\core\start-database.bat'.
  2. Locate the '-Xmx' key(lines with REM are commented out so be sure to ignore those).
  3. Increase the value by at least 500MB.
  4. Stop then start the Ektron Solr Process Manager 2.0 service.

Solr(Not in versions prior to 9.2)

  1. On your Solr server, navigate to 'C:\Solr\bin\solr.cmd' or if on an earlier version 'C:\Program Files (x86)\Ektron\Search2.0\Solr\bin\solr.cmd'.
  2. Locate the '-Xmx' key(lines with REM are commented out so be sure to ignore those).
  3. Increase the value by at least 500MB.
  4. Do the same process with '\bin\solr.in.cmd'
  5. Stop then start the Ektron Solr Process Manager 2.0 service.
  6. After doing those steps, re-evaluate if memory limits are being exceeded and react accordingly. 

After fixing the heap values clear the crawl database and reregister the site. 

Clearing the Solr Crawl Database

While the crawl is running, check the memory in process explorer and verify the working set memory does not exceed the limits. If you still see the issue, run the log collector and provide the results to Episerver Support.