Kibana Troubleshooting

This document provides troubleshooting guidance for common Kibana issues encountered during installation, configuration, and operation in Relativity environments.

1. Windows Service Issues

1.1 Kibana Service Not Starting

Symptoms:

• Kibana service fails to start
• Service stops immediately after starting

Troubleshooting Steps:

• Check Service Status and Configuration:

  Copy

  Get-Service -Name kibana
  

  Expected output

  Copy

  Status   Name   DisplayName
  ------   ----   -----------
  Running  kibana Kibana
  

  Copy

  Get-CimInstance -ClassName Win32_Service -Filter "Name='kibana'" | Select-Object Name, State, StartMode, StartName
  

  • Verify the service is running under Local System account (default configuration).

• Check Kibana Logs:

  • Navigate to C:\elastic\kibana\logs\
  • Review the latest log files (kibana.log) for error messages.
  • Look for configuration errors or Elasticsearch connection issues (for example: Unable to connect to Elasticsearch at https://<host or ip address>:9200).

• Start Service Manually:

  Copy

  Start-Service kibana
  

  Expected output

  Copy

  (No output if successful. Service status will be "Running" after execution.)
  

1.2 Service Crashes or Stops Unexpectedly

Symptoms:

• Kibana service starts but stops after a short period
• Service status shows "Stopped" unexpectedly
• Users lose access to Kibana interface

Troubleshooting Steps:

• Check Kibana Logs:

  • Navigate to C:\elastic\kibana\logs\
  • Review the latest log files (kibana.log) for crash details
  • Look for memory issues or connection failures

• Review Kibana Configuration:

  • Check C:\elastic\kibana\config\kibana.yml file
  • Verify Elasticsearch connection settings
  • Ensure all required configuration parameters are present

• Verify Elasticsearch Connectivity:

  Copy

  curl.exe -k -u <username>:<password> -X GET "https://<hostname_or_ip>:9200/"
  

  Expected output

  Copy

  {
      "name" : "EMTTEST",
      "cluster_name" : "elasticsearch",
      "cluster_uuid" : "PwBZoINKQjGZ53WH4gFfBg",
      "version" : {
         "number" : "8.17.3",
         "build_flavor" : "default",
         "build_type" : "zip",
         "build_hash" : "a091390de485bd4b127884f7e565c0cad59b10d2",
         "build_date" : "2025-02-28T10:07:26.089129809Z",
         "build_snapshot" : false,
         "lucene_version" : "9.12.0",
         "minimum_wire_compatibility_version" : "7.17.0",
         "minimum_index_compatibility_version" : "7.0.0"
      },
      "tagline" : "You Know, for Search"
  }
  

• Check Memory Usage:

  • Monitor Kibana process memory consumption using Task Manager or Resource Monitor.
  • Verify sufficient system memory is available.

2. Authentication Issues

2.1 Username/Password Authentication Issues

Symptoms:

• Login failures at Kibana interface
• "Invalid username or password" errors
• Users cannot access Kibana dashboards

Troubleshooting Steps:

• Verify User Configuration:

  • Ask the user to log in using the elastic username credentials.
  • Check C:\elastic\kibana\config\kibana.yml:
    Copy

    elasticsearch.username: "<username>"
    elasticsearch.password: "<password>"
    

• Test Elasticsearch Credentials Independently:

  • Use the elastic username and password from kibana.yml to verify connectivity to Elasticsearch:

    Copy

    curl.exe -k -u <username>:<password> -X GET "https://<hostname_or_ip>:9200/"
    

    Expected output

    Copy

    {
       "name" : "EMTTEST",
       "cluster_name" : "elasticsearch",
       ...
    }
    

3. Memory Issues

3.1 Insufficient Memory Allocation

Symptoms:

• Kibana becomes unresponsive
• Slow loading of dashboards and visualizations
• Out of memory errors in logs

Troubleshooting Steps:

• Check Current Memory Usage:

  Copy

  Get-Process -Name node | Where-Object {$_.ProcessName -eq "node"} | Select-Object WorkingSet, VirtualMemorySize
  

  Expected output

  Copy

  WorkingSet VirtualMemorySize
  ---------- ----------------
  12345678   23456789
  

• Review Out of Memory Errors in Logs:

  • Check for "out of memory" or "heap" errors in C:\elastic\kibana\logs\kibana.log.

• Verify Disk Space:
  • Insufficient disk space can also cause memory-related failures.
  • For disk space troubleshooting steps, see Verify Disk Space.

4. Kibana Encryption Keys Configuration

4.1 Missing or Invalid Encryption Keys

Symptoms:

• Kibana fails to start with encryption-related errors
• "Saved objects encryption key is missing" warnings
• Unable to save or retrieve saved objects

Troubleshooting Steps:

• Generate Encryption Keys:

  Copy

  cd C:\elastic\kibana\bin
  .\kibana-encryption-keys.bat generate
  

  Expected output

  Copy

  xpack.encryptedSavedObjects.encryptionKey: "<randomly-generated-key-1>"
  xpack.reporting.encryptionKey: "<randomly-generated-key-2>"
  xpack.security.encryptionKey: "<randomly-generated-key-3>"
  

• Configure Required Encryption Keys in kibana.yml:

  • Copy the generated lines above and paste them into your C:\elastic\kibana\config\kibana.yml file.

5. Service Verification

5.1 Verifying Kibana Health and Status

Symptoms:

• Need to confirm Kibana is operating correctly
• Performance monitoring requirements
• Health check automation

Troubleshooting Steps:

• Check Kibana Status:

  Copy

  curl.exe -k -u <username>:<password> -X GET "http://<hostname_or_ip>:5601/api/status"
  

  Expected output

  Copy

  {
     "name": "kibana",
     "version": {
       "number": "8.x.x"
     },
     "status": {
       "overall": {
         "level": "available"
       }
     }
  }
  

6. Additional Diagnostic Commands

6.1 Configuration Validation

Validate YAML syntax

C:\elastic\kibana\bin\kibana.bat 

# There is no built-in --validate-config option in Kibana 8.x.
# Configuration is validated when Kibana starts.
# If there are errors in kibana.yml, they will be shown in the console output and Kibana will fail to start.

7. Installation Issues

7.1 Long Path Issues During Extraction

Symptoms:

• Errors when extracting the kibana-8.xx.x-windows-x86_64.zip file.
• The extraction process fails due to file paths exceeding the Windows default limit.

Troubleshooting Steps:

1. Enable Long Paths in Windows: Windows must be configured to support long file paths. This can be enabled via the Local Group Policy Editor.

   1. Run gpedit.msc to open the Local Group Policy Editor.

   

   2. Navigate to Computer Configuration → Administrative Templates → System → Filesystem.

   

   3. Double-click on Enable Win32 long paths and set it to Enabled.

   

2. Re-extract the Kibana zip file: After enabling long path support, attempt to extract the kibana-8.xx.x-windows-x86_64.zip file again.