Elasticsearch Troubleshooting

This document provides troubleshooting guidance for common Elasticsearch issues encountered during installation, configuration, and operation in Relativity Server environments.

Windows service issues

Get-Service -Name elasticsearch-service-x64, kibana, apm-server | Format-Table -AutoSize

Get-Service -Name elasticsearch, kibana, apm-server | Format-Table -AutoSize

Status   Name           DisplayName
------   ----           -----------
Running  elasticsearch  elasticsearch
Running  kibana         kibana
Running  apm-server     apm-server

Elasticsearch service not starting

Symptoms:

• Elasticsearch service fails to start
• Service stops immediately after starting
• Error messages in Elasticsearch logs

Troubleshooting Steps:

1. Check Service Status:

   Copy

   Get-Service -Name elasticsearch-service-x64 | Select-Object Status, StartType, Name
   

   OR

   Copy

   Get-Service -Name elasticsearch | Select-Object Status, StartType, Name
   

   Expected output:

   Copy

   Status   StartType Name
   

Running Automatic elasticsearch

2. Verify Service Configuration:
```powershell
(Get-CimInstance Win32_Service -Filter "Name = 'elasticsearch-service-x64'").StartName

OR

(Get-CimInstance Win32_Service -Filter "Name = 'elasticsearch'").StartName

Expected output:

LocalSystem

3. Check Elasticsearch Logs:

   1. Navigate to the log directory (default: C:\elastic\elasticsearch-{version}\logs\).
   2. Review the Elasticsearch log file (elasticsearch.log) for error messages.
   3. Check the slow logs and garbage collection logs if present.
   4. For every error in the Elasticsearch log, provide troubleshooting for that specific error.
   • For detailed logging information, refer to the official Elasticsearch logging documentation
   • Elasticsearch includes a bundled Java runtime, so a separate Java installation is not required.
   • If the JAVA_HOME environment variable is defined, Elasticsearch will use the specified Java version instead of the bundled one.
   • If you want to use a specific Java version, ensure JAVA_HOME is set correctly.

4. Start Service Manually:

   Copy

   Start-Service elasticsearch-service-x64
   

   OR

   Copy

   Start-Service elasticsearch
   

   Expected output:

   Copy

   (No output if successful. Service status will be "Running" after execution.)
   

Service crashes or stops unexpectedly

Symptoms:

• Elasticsearch service starts but stops after a short period
• Service status shows "Stopped" unexpectedly

Troubleshooting Steps:

1. Check Elasticsearch Logs:

   1. Navigate to C:\elastic\elasticsearch\logs\
   2. Review the Elasticsearch log file (elasticsearch.log) for errors
   3. For every error in the Elasticsearch log, provide troubleshooting for that specific error.

      Always check the latest error in the Elasticsearch log and troubleshoot accordingly. This approach should be followed everywhere.

2. Verify Disk Space:

   1. Ensure sufficient disk space on data and log directories (minimum 15% free)

   2. Verify data and log files are on separate drives from the Operating System drive. If you store Elasticsearch data on a network share, ensure the share is accessible and has sufficient free space. Some environments may not use mapped drives.

      Copy

      # Check disk space
      Get-WmiObject -Class Win32_LogicalDisk | Select-Object DeviceID, @{Name="FreeSpace(%)";Expression={[math]::Round(($_.FreeSpace/$_.Size)*100,2)}}
      

      Expected output:

      Copy

      DeviceID FreeSpace(%)
      

   C:       22.15
   D:       48.92
   ```

Elasticsearch service installation fails with JavaVersionChecker ClassNotFoundException

Symptoms:

Running elasticsearch-service.bat install or elasticsearch.bat during a rolling upgrade fails with:

Error: Could not find or load main class org.elasticsearch.tools.java_version_checker.JavaVersionChecker
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.tools.java_version_checker.JavaVersionChecker

Troubleshooting Steps:

1. Verify the Java version being picked up by running:

   Copy

   java.exe -version
   

   Expected output:

   Copy

   java version "x.x.x" ...
   

   If the version shown does not match the JDK bundled at C:\elastic\elasticsearch\jdk, the wrong Java runtime is being used.

2. Verify that ES_JAVA_HOME, JAVA_HOME, and KCURA_JAVA_HOME all point to C:\elastic\elasticsearch\jdk:

   Copy

   $env:ES_JAVA_HOME
   $env:JAVA_HOME
   $env:KCURA_JAVA_HOME
   

3. Check the PATH variable for conflicting Java entries:

   Copy

   $env:PATH -split ';' | Where-Object { $_ -like '*java*' -or $_ -like '*jdk*' -or $_ -like '*jre*' }
   

   If multiple Java paths appear, take a backup of PATH and remove any entries that are not C:\elastic\elasticsearch\jdk\bin.

4. Run elasticsearch.bat directly (not the service installer) to capture more detailed output:

   Copy

   C:\elastic\elasticsearch\bin\elasticsearch.bat
   

   If no output or logs appear (not in Event Viewer, not in the logs folder, and not in the console), the files under C:\elastic\elasticsearch may be corrupted or incomplete.

5. Extract the Elasticsearch upgrade package to a separate temporary path. Copy the config folder from C:\elastic\elasticsearch\config into the extracted temp path, then run elasticsearch.bat from there:

   Copy

   C:\temp\elasticsearch-<version>\bin\elasticsearch.bat
   

   If Elasticsearch starts from the temp path but not from C:\elastic\elasticsearch, the C:\elastic\elasticsearch copy is corrupted.

6. Take a backup of C:\elastic\elasticsearch, then delete and re-copy the bin, lib, modules, and plugins directories cleanly from the upgrade package into C:\elastic\elasticsearch.

7. Re-run the service installation:

   Copy

   C:\elastic\elasticsearch\bin\elasticsearch-service.bat install
   

   Expected output:

   Copy

   The service 'Elasticsearch' has been installed.
   

Memory issues

Insufficient memory allocation

Symptoms:

• OutOfMemoryError in Elasticsearch logs
• Poor performance or slow queries
• Node becomes unresponsive

Troubleshooting Steps:

1. Check Current Memory Usage:

   Copy

   Get-WmiObject -Class Win32_PhysicalMemory | Measure-Object -Property Capacity -Sum
   

   Expected output:

   Copy

   Count    : 2
   Average  :
   Sum      : 34359738368
   Property : Capacity
   

   Sum is the total RAM in bytes (e.g., 34359738368 bytes = 32 GB).

2. Review JVM Heap Settings:

   1. Edit C:\elastic\elasticsearch\config\jvm.options file. (If the file does not exist, create it.)
      Copy

      # Recommended: Set Xms and Xmx to same value
      # Example for system with 8GB+ RAM:
      -Xms4g
      -Xmx4g
      

   Set heap to 50% of available RAM, maximum 32GB. Monitor current memory usage before making changes.

3. Check for Memory Leaks:

   1. Monitor heap usage over time
   2. Look for continuously increasing memory consumption
   3. Review application logs for memory-related warnings

Authentication issues

Username/Password authentication issues

Symptoms:

• Login failures
• "authentication failed" errors
• Cannot access Elasticsearch with credentials

Troubleshooting Steps:

1. Verify User Exists:

   Copy

   curl.exe -k -X GET "https://<hostname_or_ip>:9200/_security/user/<username>" -u <username>:<password>
   

   Expected output:

   Copy

   {
     "<username>": {
       "username": "<username>",
       "roles": [
         "superuser"
       ],
       ...
     }
   }
   

2. Reset Password:

   Copy

   C:\elastic\elasticsearch\bin\elasticsearch-reset-password.bat -u <username>
   

   Expected output:

   Copy

   Password for the [<username>] user successfully reset.
   

3. Verify Security Configuration:

   1. Check C:\elastic\elasticsearch\config\elasticsearch.yml:
      Copy

      xpack.security.enabled: true
      

   Also verify that the URL you are using is https://<username>:9200/

Audit issues

Audit write failures

Symptoms:

Audit write operations fail with HTTP 400 responses from Elasticsearch. The following exception appears in Relativity logs:

System.Exception: Error has occurred while writing to Data Grid. BatchID: <batch_id>, Range: <range_start> - <range_end> ---> Relativity.Audit.Elasticsearch.Exceptions.InvalidDataGridHeartbeatException: Unable to get or create an index template from the cluster. -> Invalid heartbeat reply, CheckIndexStatus failed. -> CreateIndex failed, Index: 'index_name' -> Http status code 400 did not indicate success:

Troubleshooting Steps:

1. Stop the Audit agents in Relativity.
2. Verify the ESIndexCreationSettings instance setting value is correct. In Relativity, go to Admin | Instance Settings and locate ESIndexCreationSettings. Confirm the setting value matches your cluster's intended configuration. If the value has been modified or reset, restore the correct value before proceeding. For details, refer to the Instance Setting Guide.
3. Delete the audit index template from the cluster:

curl.exe -k -u <username>:<password> -X GET "https://<hostname_or_ip>:9200/_template/audit"

curl.exe -k -u <username>:<password> -X DELETE "https://<hostname_or_ip>:9200/_template/audit"

4. Re-enable the Audit agents in Relativity.

Service verification

Verifying Elasticsearch health

Symptoms:

• Uncertainty about cluster status
• Need to confirm proper operation
• Performance monitoring

Troubleshooting Steps:

1. Check Cluster Health:

   Copy

   curl.exe -k -u <username>:<password> -X GET "https://<hostname_or_ip>:9200/_cluster/health?pretty"
   

   Expected response for healthy cluster: :

   Copy

   {
     "cluster_name": "elasticsearch",
     "status": "green",
     "timed_out": false,
     "number_of_nodes": 3,
     "number_of_data_nodes": 3
   }
   

2. Verify Node Status:

   Copy

   curl.exe -k -u <username>:<password> -X GET "https://<hostname_or_ip>:9200/_cat/nodes?v"
   

   Expected output:

   Copy

   ip            heap.percent ram.percent cpu load_1m load_5m load_15m     node.role      master name
   <ip>          14           95          28                               cdfhilmrstw *  <node name>
   

3. Check Index Health:

   Copy

   curl.exe -k -u <username>:<password> -X GET "https://<hostname_or_ip>:9200/_cat/indices?v"
   

   Expected output:

   Copy

   health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
   green  open   myindex  1a2b3c4d5e6f7g8h9i0j   1   1   1000       0            1.2mb      600kb
   

4. Monitor Performance:

   Copy

   curl.exe -k -u <username>:<password> -X GET "https://<hostname_or_ip>:9200/_nodes/stats?pretty"
   

   Expected output:

   Copy

   {
     "nodes": {
       "node_id": {
         "name": "node-1",
         "host": "10.0.0.1",
         ...
       }
     }
   }