Elasticsearch Troubleshooting

This document provides troubleshooting guidance for common Elasticsearch issues encountered during installation, configuration, and operation in Relativity Server environments.

This guide assumes a default Elasticsearch installation path of C:\elastic\elasticsearch. Adjust paths according to your actual installation directory.

1. Windows Service Issues

Before troubleshooting Elasticsearch, verify that all required Elastic Stack services are running. If any of these are not running, other troubleshooting steps may be irrelevant.

Check all required services:

Copy 
Get-Service -Name elasticsearch-service-x64, kibana, apm-server | Format-Table -AutoSize

OR

Copy 
Get-Service -Name elasticsearch, kibana, apm-server | Format-Table -AutoSize
Expected output
Copy 
Status   Name           DisplayName
------   ----           -----------
Running  elasticsearch  elasticsearch
Running  kibana         kibana
Running  apm-server     apm-server
  • The kibana Windows service may not exist if Kibana was not installed as a Windows service (e.g., via NSSM). NSSM is not required; Kibana can be run manually or as a scheduled task. Only check for the kibana service if you have installed it as a service.
  • If Kibana is not in running state, click here for Kibana troubleshooting.
  • If APM is not in running state, click here for APM troubleshooting

1.1 Elasticsearch Service Not Starting

Symptoms:

  • Elasticsearch service fails to start
  • Service stops immediately after starting
  • Error messages in Elasticsearch logs

Troubleshooting Steps:

  • Check Service Status:

    Copy 
    Get-Service -Name elasticsearch-service-x64 | Select-Object Status, StartType, Name

    OR

    Copy 
    Get-Service -Name elasticsearch | Select-Object Status, StartType, Name
    Expected output
    Copy 
    Status   StartType Name
    ------   --------- ----
    Running  Automatic elasticsearch

  • Verify Service Configuration:

    Copy 
    (Get-CimInstance Win32_Service -Filter "Name = 'elasticsearch-service-x64'").StartName

    OR

    Copy 
    (Get-CimInstance Win32_Service -Filter "Name = 'elasticsearch'").StartName
    Expected output
    Copy 
    LocalSystem

  • Check Elasticsearch Logs:

    • Navigate to the log directory (default: C:\elastic\elasticsearch-8.17.3\logs\).
    • Review the Elasticsearch log file (elasticsearch.log) for error messages.
    • Check the slow logs and garbage collection logs if present.
    • For every error in the Elasticsearch log, provide troubleshooting for that specific error.

For detailed logging information, refer to the official Elasticsearch logging documentation

  • Elasticsearch includes a bundled Java runtime, so a separate Java installation is not required.
  • If the JAVA_HOME environment variable is defined, Elasticsearch will use the specified Java version instead of the bundled one.
  • If you want to use a specific Java version, ensure JAVA_HOME is set correctly.

  • Start Service Manually:

    Copy 
    Start-Service elasticsearch-service-x64

    OR

    Copy 
    Start-Service elasticsearch
    Expected output
    Copy 
    (No output if successful. Service status will be "Running" after execution.)

1.2 Service Crashes or Stops Unexpectedly

Symptoms:

  • Elasticsearch service starts but stops after a short period
  • Service status shows "Stopped" unexpectedly

Troubleshooting Steps:

  • Check Elasticsearch Logs:
    • Navigate to C:\elastic\elasticsearch\logs\
    • Review the Elasticsearch log file (elasticsearch.log) for errors
    • For every error in the Elasticsearch log, provide troubleshooting for that specific error.

Always check the latest error in the Elasticsearch log and troubleshoot accordingly. This approach should be followed everywhere.

  • Verify Disk Space:

    • Ensure sufficient disk space on data and log directories (minimum 15% free)
    • Verify data and log files are on separate drives from the Operating System drive. If you store Elasticsearch data on a network share, ensure the share is accessible and has sufficient free space. Some environments may not use mapped drives.
    Copy 
    # Check disk space
    Get-WmiObject -Class Win32_LogicalDisk | Select-Object DeviceID, @{Name="FreeSpace(%)";Expression={[math]::Round(($_.FreeSpace/$_.Size)*100,2)}}
    Expected output
    Copy 
    DeviceID FreeSpace(%)
    -------- ------------
    C:       22.15
    D:       48.92

For port-related issues, see the Port Configuration Troubleshooting guide.

2. Memory Issues

2.1 Insufficient Memory Allocation

Symptoms:

  • OutOfMemoryError in Elasticsearch logs
  • Poor performance or slow queries
  • Node becomes unresponsive

Troubleshooting Steps:

  • Check Current Memory Usage:

    Copy 
    Get-WmiObject -Class Win32_PhysicalMemory | Measure-Object -Property Capacity -Sum
    Expected output
    Copy 
    Count    : 2
    Average  :
    Sum      : 34359738368
    Property : Capacity

Sum is the total RAM in bytes (e.g., 34359738368 bytes = 32 GB).

  • Review JVM Heap Settings:
    • Edit C:\elastic\elasticsearch\config\jvm.options file:
    • If the file does not exist, create it.
    Copy 
    # Recommended: Set Xms and Xmx to same value
    # Example for system with 8GB+ RAM:
    -Xms4g
    -Xmx4g

Set heap to 50% of available RAM, maximum 32GB. Monitor current memory usage before making changes.

  • Check for Memory Leaks:
    • Monitor heap usage over time
    • Look for continuously increasing memory consumption
    • Review application logs for memory-related warnings

3. Authentication Issues

3.1 Username/Password Authentication Problems

Symptoms:

  • Login failures
  • "authentication failed" errors
  • Cannot access Elasticsearch with credentials

Troubleshooting Steps:

  • Verify User Exists:

    Copy 
    curl.exe -k -X GET "https://<hostname_or_ip>:9200/_security/user/<username>" -u <username>:<password>
    Expected output
    Copy 
    {
       "<username>": {
         "username": "<username>",
         "roles": [
           "superuser"
         ],
         ...
       }
    }

  • Reset Password:

    Copy 
    C:\elastic\elasticsearch\bin\elasticsearch-reset-password.bat -u <username>
    Expected output
    Copy 
    Password for the [<username>] user successfully reset.

  • Verify Security Configuration:

    • Check C:\elastic\elasticsearch\config\elasticsearch.yml:
      Copy
      xpack.security.enabled: true

Also verify that the URL you are using is https://<username>:9200/

4. Service Verification

4.1 Verifying Elasticsearch Health

Symptoms:

  • Uncertainty about cluster status
  • Need to confirm proper operation
  • Performance monitoring

Troubleshooting Steps:

  • Check Cluster Health:

    Copy 
    curl.exe -k -u <username>:<password> -X GET "https://<hostname_or_ip>:9200/_cluster/health?pretty"
    Expected response for healthy cluster
    Copy 
    {
       "cluster_name": "elasticsearch",
       "status": "green",
       "timed_out": false,
       "number_of_nodes": 3,
       "number_of_data_nodes": 3
    }

  • Verify Node Status:

    Copy 
    curl.exe -k -u <username>:<password> -X GET "https://<hostname_or_ip>:9200/_cat/nodes?v"
    Expected output
    Copy 
    ip            heap.percent ram.percent cpu load_1m load_5m load_15m     node.role      master name
    <ip address>            14          95  28                          cdfhilmrstw *      <node name>

  • Check Index Health:

    Copy 
    curl.exe -k -u <username>:<password> -X GET "https://<hostname_or_ip>:9200/_cat/indices?v"
    Expected output
    Copy 
    health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
    green  open   myindex  1a2b3c4d5e6f7g8h9i0j   1   1   1000       0            1.2mb      600kb

  • Monitor Performance:

    Copy 
    curl.exe -k -u <username>:<password> -X GET "https://<hostname_or_ip>:9200/_nodes/stats?pretty"
    Expected output
    Copy 
    {
       "nodes": {
         "node_id": {
           "name": "node-1",
           "host": "10.0.0.1",
           ...
         }
       }
    }