Custom JSON Configuration Troubleshooting

This document provides guidance for troubleshooting issues related to custom JSON configuration file in Relativity Server environments.

Prerequisites

• Install the latest Server Bundle to ensure support for custom JSON configuration. The most recent release can be downloaded from the Server Bundle releases page.

Common Issues

If the log message "The Environment Watch shared configuration object is not empty" does not appear in Kibana, review the following potential causes:

"The Environment Watch shared configuration object is not empty"

The custom JSON configuration file is not placed in the correct BCP path.

Verify that the custom JSON configuration file is located in the correct BCP path.

Base path: BCPPath

Folder: EnvironmentWatch

File name: environment-watch-configuration.json

Environment Watch automatically reads this file and applies the defined monitoring rules to the relevant instances, products, and hosts.

To identify the BCP path for the environment, navigate to the "Server" tab in the Relativity front end as below:

Complete path to the custom JSON configuration file will be:

\\EMTTEST\BCPPath\EnvironmentWatch\environment-watch-configuration.json

An example of the BCPPath and folder structure is shown below:

Activate the BCP path if it is not active.

• Log in to the Relativity application.
• Navigate to Servers.
• Filter by SQL - Primary in the Type column.

• Ensure the BCP path is active and Visible in Dropdown is set to 'Yes'.

• Click on 'Edit' and toggle the Status to 'Inactive' and save it.
• Again, click on 'Edit' and toggle the Status to 'Active' and save it.
• Restart the Environment Watch Windows Service to apply the changes.

Check the custom JSON configuration file for syntax errors.

• Download the sample JSON file from the link below and compare it with your custom JSON configuration file to identify any syntax errors.

  Sample Custom JSON Configuration File

• Restart the Environment Watch Windows Service to apply the changes.

Update the Environment Watch Windows Service to the latest version.

• Verify the installed version of the Environment Watch Windows Service.
• Ensure the service is updated to latest version, as custom JSON configuration file support was introduced in latest version.
• If the service is running an earlier version, review the Prerequisites section to confirm you’re on the latest Server Bundle, then upgrade the Environment Watch Windows Service to the latest available version.

Certificates

If certificates do not appear in the Kibana dashboard after configuring the custom JSON configuration file, fix the common issues outlined in Common Issues before reviewing the following potential causes:

After fixing the common issues, If a certificate alert is triggered in Kibana, review the following potential causes:

Certificate is misconfigured.

Run the following PowerShell command based on the certificate store location and name. For LocalMachine and My, use:

Get-ChildItem Cert:\LocalMachine\My

Certificate configured correctly, but the certificate is expired.

• Verify whether the certificate is expired from the certificate dashboard.
• If it is expired, replace it with a valid certificate. For LocalMachine and My, use:

Get-ChildItem Cert:\LocalMachine\My

Ensure the host name is correct for monitoring by Host or Installed Product.

• Run the following PowerShell command:

hostname

• Ensure the hostName property in your configuration matches the output. Example:

{
  "environmentWatchConfiguration": {
    "monitoring": {
      "instance": {
        "sources": {
          "certificates": {},
          "windowsServices": {
            "enabled": false,
            "include": []
          }
        },
        "otelCollectorYamlFiles": []
      },
      "installedProducts": [
        {
          "productName": "Agent",
          "sources": {
            "certificates": {
              "enabled": true,
              "include": []
            },
            "windowsServices": {
              "enabled": true,
              "include": []
            }
          },
          "otelCollectorYamlFiles": []
        }
      ],
      "hosts": [
        {
          "hostName": "SQL01",
          "sources": {
            "certificates": {
              "enabled": true,
              "include": []
            },
            "sqlServers": {
              "enabled": true,
              "include": [
                {
                  "clusterVirtualName": "SQLCLUSTER",
                  "instanceName": "SQL_INSTANCE"
                }
              ]
            },
            "windowsServices": {
              "enabled": true,
              "include": []
            }
          },
          "otelCollectorYamlFiles": []
        }
      ]
    },
    "alertNotificationHandlers": {}
  }
}

Avoid configuring certificate thumbprints in Monitoring by instance.

Example configuration for instance monitoring:

{
  "environmentWatchConfiguration": {
    "monitoring": {
      "instance": {
        "sources": {
          "certificates": {
            "enabled": false,
            "include": []
          },
          "windowsServices": {
            "enabled": true,
            "include": [
              "WindowsAzureGuestAgent",
              "mpssvc"
            ]
          }
        }
      },
      "installedProducts": [
        {
          "productName": "Agent",
          "sources": {
            "certificates": {
              "enabled": false,
              "include": []
            },
            "windowsServices": {
              "enabled": false,
              "include": []
            }
          },
          "otelCollectorYamlFiles": []
        }
      ],
      "hosts": [
        {
          "hostName": "SQL01",
          "sources": {
            "certificates": {
              "enabled": false,
              "include": []
            },
            "sqlServers": {
              "enabled": false,
              "include": []
            },
            "windowsServices": {
              "enabled": false,
              "include": []
            }
          },
          "otelCollectorYamlFiles": []
        }
      ]
    },
    "alertNotificationHandlers": {}
  }
}

Windows Services

If Windows services do not appear in the Kibana dashboard after configuring the custom JSON configuration file, fix the common issues outlined in Common Issues before reviewing the following potential causes:

After fixing the common issues, if Windows services still do not appear in the Kibana dashboard, review the following potential causes:

Verify the Windows services in the custom JSON configuration file.

• Ensure the custom JSON configuration file contains the correct configuration for the Windows services to monitor.
• Place the configuration under the windowsServices section for the relevant hosts, instance, or installedProducts entry.

Example:

{
  "environmentWatchConfiguration": {
    "monitoring": {
      "instance": {
        "sources": {
          "certificates": {
            "enabled": false,
            "include": []
          },
          "windowsServices": {
            "enabled": true,
            "include": [
              "WindowsAzureGuestAgent",
              "mpssvc"
            ]
          }
        }
      },
      "installedProducts": [
        {
          "productName": "Agent",
          "sources": {
            "certificates": {
              "enabled": false,
              "include": []
            },
            "windowsServices": {
              "enabled": true,
              "include": []
            }
          },
          "otelCollectorYamlFiles": []
        }
      ],
      "hosts": [
        {
          "hostName": "SQL01",
          "sources": {
            "certificates": {
              "enabled": false,
              "include": []
            },
            "sqlServers": {
              "enabled": false,
              "include": []
            },
            "windowsServices": {
              "enabled": true,
              "include": ["MSSQLSERVER", "AnotherService"]
            }
          },
          "otelCollectorYamlFiles": []
        }
      ]
    },
    "alertNotificationHandlers": {}
  }
}

Verify the Windows services are running and exist on the host.

• Verify the Windows services to be monitored are running and exist on the host machine.
• Use the Services management console (services.msc) or the following PowerShell command:

Get-Service -Name MSSQLSERVER, AnotherService

Always include the Windows service name in the custom JSON configuration file.

• Ensure that the correct service names (not display names) are used in the include section of the custom JSON configuration file.

SQL Server Instances

When monitoring SQL server instances using the custom JSON configuration file, ensure the following:

• All nodes of the SQL environment are properly configured.
• Update the correct SQL server virtual name and instance name.
• The Environment Watch Windows service picks up the configuration changes.
• The following log message appears in Kibana to confirm that SQL server instance details are picked up from the custom JSON configuration file:

Ensure all instances/nodes of the SQL environment are monitored.

• Include each node of the SQL server in the hosts section of the custom JSON configuration file with the correct hostName, clusterVirtualName, and instanceName.
• Set sqlServers -> enabled to true for each host entry.

{
  "environmentWatchConfiguration": {
    "monitoring": {
      "instance": {
        "sources": {
          "certificates": {},
          "windowsServices": {}
        },
        "otelCollectorYamlFiles": []
      },
      "installedProducts": [
        {
          "productName": "Agent",
          "sources": {
            "certificates": {},
            "windowsServices": {}
          },
          "otelCollectorYamlFiles": []
        }
      ],
      "hosts": [
        {
          "hostName": "SQLNode1",
          "sources": {
            "certificates": {},
            "sqlServers": {
              "enabled": true,
              "include": [
                {
                  "clusterVirtualName": "SQLCLUSTER",
                  "instanceName": "SQL_INSTANCE"
                }
              ]
            },
            "windowsServices": {}
          },
          "otelCollectorYamlFiles": []
        },
        {
          "hostName": "SQLNode2",
          "sources": {
            "certificates": {},
            "sqlServers": {
              "enabled": true,
              "include": [
                {
                  "clusterVirtualName": "SQLCLUSTER",
                  "instanceName": "SQL_INSTANCE"
                }
              ]
            },
            "windowsServices": {}
          },
          "otelCollectorYamlFiles": []
        }
      ]
    },
    "alertNotificationHandlers": {}
  }
}

Slack Notifications

When troubleshooting Slack notification issues, the first step is to verify whether the following log message appears in Kibana Discover for the triggered alert.

• Log message: Message successfully sent alert {AlertId} to Slack channel {SlackChannel}

When a Kibana alert is triggered, and unable to see: "Message successfully sent alert to Slack channel " log in Kibana Discover

• Pass the Kibana Alert ID ({AlertId}) and Slack Channel ID ({SlackChannel}) in Kibana Discover.
• If the message is not found, it may indicate that the custom JSON configuration file is not set up correctly.
• Refer to Common Issues section 1.1 to 1.4 to troubleshoot why the Environment Watch Windows service is not picking up the custom JSON configuration file changes.

The log message appears in Kibana Discover after fixing common issues, but Slack notifications are still not being sent

• Verify that the Slack handler is correctly configured in the alertNotificationHandlers section of the custom JSON configuration file.
• Ensure that the accessToken, channel, enabled, and messageIntervalSeconds properties are set correctly.
• This can also occur if the Slack API token is invalid or does not have the necessary permissions to post messages to the specified channel.
• Verify the token and permissions.
• Once the issues above are resolved, Slack notifications should be sent successfully.

If users need to adjust the frequency of Slack messages, follow these steps:

• Set the messageIntervalSeconds property in the Slack handler configuration to a value greater than or equal to 180 seconds to avoid rate limiting by Slack.
• Restart the Environment Watch Windows service to apply the changes.