Skip to main content

Observability Integrations

.

AppDynamics

The AI Incident Assistant AppDynamics integration is used for full-stack/APM monitoring.

To configure the AI Incident Assistant Datadog integration, populate the following fields:

Field

Description

AppD Controller URL

Enter the controller URL for your AppDynamics instance. (For example, https://xyz.saas.appdynamics.com)

Client ID

Your AppDynamics Client ID is a combination of the client name and account name in this format: [CLIENT_NAME]@[ACCT_NAME]

Client Secret

Auto-generated AppDynamics client secret.

Custom Headers (Optional)

Add custom HTTP headers to include with all API requests. For each header, include the Header Name and Header Value. To add additional headers, click the + sign. 

Deployment Type

Select whether your AppDynamics integration is Cloud or On-Prem.

Certain integrations have endpoints that work only with one deployment type. Selecting the correct option automatically applies the right guardrails to the AI Incident Assistant agents.

Note: On-prem integrations can connect via the Relay Client, enabling secure communication with infrastructure behind your firewall.Remote Connections

Agent Notes

Add notes to help AI Incident Assistant understand your AppDynamics environment.

We highly recommend configuring this field, as it allows you to enter organization-specific information that enables AI Incident Assistant to provide more accurate and consistent results. 

For example, if your logs contain a specific field that is used to identify the type of application or service, provide that information here.

Datadog

The AI Incident Assistant Datadog integration is used for full-stack monitoring.

Beta integration

The AI Incident Assistant Datadog integration is currently in Beta release status.

In Datadog, ensure the following permissions/scopes have been enabled:

  • apm_api_catalog_read

  • apm_read

  • apm_remote_configuration_read

  • apm_retention_filter_read

  • apm_service_catalog_read

  • apm_service_ingest_read

  • audience_management_read

  • continuous_profiler_pgo_read

  • continuous_profiler_read

  • debugger_read

  • error_tracking_read

  • events_read

  • incident_read

  • logs_read_data

  • logs_read_index_data

  • metrics_read

  • monitors_read

  • reference_tables_read

  • rum_apps_read

  • rum_retention_filters_read

  • rum_session_replay_read

  • synthetics_global_variable_read

  • synthetics_private_location_read

  • synthetics_read

  • timeseries_query

To configure the Datadog integration, populate the following fields:

Field

Description

Datadog Region

The region of your Datadog instace.

If you access Datadog via datadoghq.eu, select EU.

If not, choose your US region. Select from US, US3, or US5.

API Key

Your Datadog API key.

Application Key

Your Datadog application key with required read-only scopes.

Agent Notes

Add notes to help AI Incident Assistant understand your Datadog environment.

We highly recommend configuring this field, as it allows you to enter organization-specific information that enables AI Incident Assistant to provide more accurate and consistent results. 

For example, if your logs contain a specific field that is used to identify the type of application or service, provide that information here.

Dynatrace

The AI Incident Assistant Dynatrace integration is used for application performance monitoring.

Ensure the following Dynatrace scopes/permissions have been enabled:

API v1 Scopes 

  • DataExport

  • ReadConfig

  • ExternalSyntheticIntegration

  • ReadSyntheticData

API v2 Scopes 

  • entities.read

  • events.read

  • metrics.read

  • problems.read

  • settings.read

  • logs.read

  • syntheticLocations.read

  • syntheticExecutions.read

To configure the Dynatrace integration, populate the following fields:

Field

Description

Dynatrace URL

The URL of your Dynatrace instance.

API Key

Your Dynatrace API key.

Custom Headers (Optional)

Add custom HTTP headers to include with all API requests. For each header, include the Header Name and Header Value. To add additional headers, click the + sign. 

Deployment Type

Select whether your Dynatrace integration is Cloud or On-Prem.

Certain integrations have endpoints that work only with one deployment type. Selecting the correct option automatically applies the right guardrails to the AI Incident Assistant agents.

Note: On-prem integrations can connect via the Relay Client, enabling secure communication with infrastructure behind your firewall.Remote Connections

Agent Notes

Add notes to help AI Incident Assistant understand your Dynatrace environment.

We highly recommend configuring this field, as it allows you to enter organization-specific information that enables AI Incident Assistant to provide more accurate and consistent results. 

For example, if your logs contain a specific field that is used to identify the type of application or service, provide that information here.

Elasticsearch

The AI Incident Assistant Elasticsearch integration is used for metrics monitoring. 

Beta integration

This integration is currently in beta release status.

Beta integrations are still being tested, but can be enabled by any customer. Core connectivity is stable, but the integration may receive updates based on user feedback.

Populate the following fields to set up the integration:

Field

Description

Authentication Method

Choose how you'd like to authenticate with this integration. The following methods are available:

  • Basic Authentication - authenticate using an Elasticsearch username and password

  • API Key Authentication - authenticate using an Elasticsearch API key

  • Bearer Token Authentication - authenticate using an Elasticsearch bearer token

Elasticsearch Instance URL

Enter the base URL of the inbound Elasticsearch instance. Include the port if applicable. (For example, https://<inbound-ip>:9200)

Credentials

Enter the credentials that allow the integration to authenticate.

If you selected the Basic Authentication method, enter your Elasticsearch username and password.

If you selected API Key Authentication, enter an Elasticsearch API key.

If you selected Bearer Token Authentication, enter an Elasticsearch bearer token.

Custom Headers (Optional)

Add custom HTTP headers to include with all API requests. For each header, include the Header Name and Header Value. To add additional headers, click the + sign. 

Deployment

Select whether your Elasticsearch integration is Cloud or On-Prem.

Certain integrations have endpoints that work only with one deployment type. Selecting the correct option automatically applies the right guardrails to the AI Incident Assistant agents.

Note: On-prem integrations can connect via the Relay Client, enabling secure communication with infrastructure behind your firewall.Remote Connections

Elasticsearch Agent Notes

Add notes to help AI Incident Assistant understand your Elasticsearch environment.

We highly recommend configuring this field, as it allows you to enter organization-specific information that enables AI Incident Assistant to provide more accurate and consistent results.

For example, if your logs contain a specific field that is used to identify the type of application or service, provide that information here.

Grafana

Use the AI Incident Assistant Grafana integration for metrics monitoring. Use this integration for action plans, such as the Observability Agent.

Beta integration

This integration is currently in beta release status.

Beta integrations are still being tested, but can be enabled by any customer. Core connectivity is stable, but the integration may receive updates based on user feedback.

Populate the following fields to set up the integration:

Field

Description

Authentication Method

Choose how you'd like to authenticate with this integration. The following methods are available:

  • Basic Authentication - authenticate using a Grafana username and password

  • Bearer Token Authentication - authenticate using a Grafana bearer token

Grafana Instance URL

Enter the base URL of the inbound Grafana instance. Include the port if applicable. (For example, https://myinstance.grafana.net).

Credentials

Enter the credentials that allow the integration to authenticate.

If you selected the Basic Authentication method, enter your Grafana username and password.

If you selected Bearer Token Authentication, enter a Grafana bearer token.

Custom Headers (Optional)

Add custom HTTP headers to include with all API requests. For each header, include the Header Name and Header Value. To add additional headers, click the + sign.

Deployment Type

Select whether your Grafana integration is Cloud or On-Prem.

Certain integrations have endpoints that work only with one deployment type. Selecting the correct option automatically applies the right guardrails to the AI Incident Assistant agents.

Note: On-prem integrations can connect via the Relay Client, enabling secure communication with infrastructure behind your firewall.Remote Connections

Connectivity

After entering the authentication method, instance URL, and credentials, click Verify Connection to ensure that the connection to Grafana is established.

Click Update Data Schema to allow AI Incident Assistant to retrieve and store relevant schema information for this integration. This helps AI Incident Assistant understand how your data is structured, allowing for more accurate and efficient information retrieval.

Grafana Agent Notes

Add notes to help AI Incident Assistant understand your Grafana environment.

We highly recommend configuring this field, as it allows you to enter organization-specific information that enables AI Incident Assistant to provide more accurate and consistent results.

For example, if your logs contain a specific field that is used to identify the type of application or service, define the purpose and function of that field here.

New Relic

The New Relic - AI Incident Assistant integration is used to retrieve data from application performance monitoring (APM), metrics, logs, and infrastructure telemetry.

Beta integration

This integration is currently in beta release status.

Beta integrations are still being tested, but can be enabled by any customer. Core connectivity is stable, but the integration may receive updates based on user feedback.

Required Permissions

To ensure AI Incident Assistant has the access it needs to perform necessary functions, either the New Relic Read Only role must be granted, or a custom role containing the following permissions:

  • NRQL query execution 

  • Entity read (APM, Infrastructure, Browser, Mobile, Synthetics, Workloads, Dashboards) 

  • Alert policies read 

  • Alert issues and incidents read 

  • Synthetics monitor read 

  • Topology read 

To configure the New Relic integration:

  1. In New Relic, create or choose a dedicated integration user with access to the account that the AI Incident Assistant should query.

  2. In the New Relic API Keys page, generate a user API key for the user selected in step 1 and copy it.

    API key restrictions

    Do not use License, Browser, Mobile, or retired REST API keys

  3. In New Relic Account Settings, copy the target New Relic account ID. 

  4. Populate the following fields in the web app New Relic integration page:

    Field

    Description

    New Relic API Base URL

    Base URL for your New Relic region. (For example, https://api.newrelic.com for US or https://api.eu.newrelic.com for EU)

    New Relic Account ID

    New Relic account ID copied in step 3.

    New Relic User Key

    User API key copied in step 2.

    Custom Headers (optional)

    Add custom HTTP headers to include with all API requests. For each header, include the Header Name and Header Value. To add additional headers, click the + sign.

    Deployment Type

    Select whether your New Relic integration is Cloud or On-Prem.

    Certain integrations have endpoints that work only with one deployment type. Selecting the correct option automatically applies the right guardrails to the AI Incident Assistant agents.

    Note: On-prem integrations can connect via the Relay Client, enabling secure communication with infrastructure behind your firewall.Remote Connections

    New Relic Agent Notes

    Add notes to help AI Incident Assistant understand your New Relic environment.

    We highly recommend configuring this field, as it allows you to enter organization-specific information that enables AI Incident Assistant to provide more accurate and consistent results.

    For example, enter explanations of custom fields and when or how they should be used to handle user queries.

OpsRamp

The OpsRamp integration is used for hybrid IT infrastructure monitoring, alerting, and resource management.

Beta integration

This integration is currently in beta release status.

Beta integrations are still being tested, but can be enabled by any customer. Core connectivity is stable, but the integration may receive updates based on user feedback.

Required permissions

The following OpsRamp permissions must be enabled to use the integration:

  • Alerts: View, Search

  • Resources: View, Search

  • Metrics: View, Query

  • Incidents/Tickets: View, Search

  • Service Maps: View

  • Topology: View

To configure the integration:

  1. In OpsRamp, go to Setup > Integrations and install a custom integration. Be sure to copy your Key, Secret, and API Endpoint.

  2. Go to Setup > Accounts and copy your Tenant ID.

  3. In the web app, configure the following fields:

    Field

    Description

    OpsRamp API Endpoint

    The API endpoint from your custom integration, copied from the integration setup page in step 1. (i.e., https://api.opsramp.com)

    Integration Key

    OpsRamp integration key, copied from the integration setup page in step 1.

    Integration Secret

    OpsRamp integration secret, copied from the integration setup page in step 1.

    Tenant ID

    OpsRamp tenant ID, copied from the OpsRamp Accounts page in step 2.

    Custom Headers

    Add custom HTTP headers to include with all API requests. For each header, include the Header Name and Header Value. To add additional headers, click the + sign.

    Deployment Type

    Select whether your OpsRamp integration is Cloud or On-Prem.

    Certain integrations have endpoints that work only with one deployment type. Selecting the correct option automatically applies the right guardrails to the AI Incident Assistant agents.

    Note: On-prem integrations can connect via the Relay Client, enabling secure communication with infrastructure behind your firewall.Remote Connections

    OpsRamp System/Schema Nuances

    Add notes to help AI Incident Assistant understand your OpsRamp environment.

    We highly recommend configuring this field, as it allows you to enter organization-specific information that enables AI Incident Assistant to provide more accurate and consistent results.

    For example, enter explanations of custom fields and when or how they should be used to handle user queries.

Prometheus

The AI Incident Assistant Prometheus integration is used for metrics monitoring. 

Beta integration

This integration is currently in beta release status.

Beta integrations are still being tested, but can be enabled by any customer. Core connectivity is stable, but the integration may receive updates based on user feedback.

Configure the following fields to set up the integration:

Field

Description

Prometheus Instance URL

Enter the base URL of the inbound Prometheus instance. Include the port, if applicable.

(For example, https://<prometheus-dns-name>.io or https://<prometheus-ip>:9090).

Username and Password

Add credentials of the Prometheus account that AI Incident Assistant should use for API interactions.

Custom Headers (Optional)

Add custom HTTP headers to include with all API requests. For each header, include the Header Name and Header Value. To add additional headers, click the + sign. 

Deployment Type

Select whether your OpsLevel integration is Cloud or On-Prem.

Certain integrations have endpoints that work only with one deployment type. Selecting the correct option automatically applies the right guardrails to the AI Incident Assistant agents.

Note: On-prem integrations can connect via the Relay Client, enabling secure communication with infrastructure behind your firewall.Remote Connections

Connectivity

After entering the instance URL, credentials, and deployment type, click Verify Connection to ensure that the connection to Prometheus is established.

Click Update Data Schema to allow AI Incident Assistant to retrieve and store relevant schema information for this integration. This helps AI Incident Assistant understand how your data is structured, allowing for more accurate and efficient information retrieval.

Prometheus Agent Notes

Add notes to help AI Incident Assistant understand your Prometheus environment.

We highly recommend configuring this field, as it allows you to enter organization-specific information that enables AI Incident Assistant to provide more accurate and consistent results.

For example, if your logs contain a specific field that is used to identify the type of application or service, define the purpose and function of that field here.

Splunk

The AI Incident Assistant Splunk integration is used for log monitoring. 

Beta integration

This integration is currently in beta release status.

Beta integrations are still being tested, but can be enabled by any customer. Core connectivity is stable, but the integration may receive updates based on user feedback.

Configure the following fields to set up the integration:

Field

Description

Authentication Method

Choose how you'd like to authenticate with this integration. The following methods are available:

Basic Authentication - authenticate using a Splunk username and password.

Bearer Token Authentication - authenticate using a Splunk bearer token.

Splunk Instance URL

Enter the Base URL and Port of the inbound Splunk instance.

Note: When configuring Splunk Enterprise (on-prem), you must include port 8089 in the base URL. (For example, https://<inbound-ip>:8089)

Credentials

Enter the credentials that allow the integration to authenticate.

If you selected the Basic Authentication method, enter the Username and Password of the Splunk account that AI Incident Assistant should use for API interactions.

If you selected the Bearer Token Authentication method, enter a Splunk Bearer Token.

Custom Headers (Optional)

Add custom HTTP headers to include with all API requests. For each header, include the Header Name and Header Value. To add additional headers, click the + sign. 

Deployment Type

Select whether your Splunk integration is Cloud or On-Prem.

Certain integrations have endpoints that work only with one deployment type. Selecting the correct option automatically applies the right guardrails to the AI Incident Assistant agents.

Note: On-prem integrations can connect via the Relay Client, enabling secure communication with infrastructure behind your firewall.Remote Connections

Splunk Agent Notes

Add notes to help AI Incident Assistant understand your Splunk environment.

We highly recommend configuring this field, as it allows you to enter organization-specific information that enables AI Incident Assistant to provide more accurate and consistent results. 

For example, if your logs contain a specific field that is used to identify the type of application or service, provide that information here.

ThousandEyes

Configure the ThousandEyes integration for network monitoring, application availability, BGP analysis, and endpoint metrics.

Beta integration

This integration is currently in beta release status.

Beta integrations are still being tested, but can be enabled by any customer. Core connectivity is stable, but the integration may receive updates based on user feedback.

Required permissions

To ensure AI Incident Assistant has the access it needs to perform necessary functions, either the ThousandEyes Regular User role must be granted, or a custom role containing the following permissions:

  • API access

  • Internet Insights - Catalog settings

  • View agents in account group

  • View alert rules

  • View alert suppression windows

  • View BGP monitors

  • View connectors and operations

  • View dashboards

  • View endpoint agent data

  • View endpoint agent settings

  • View endpoint tests

  • View labels

  • View tests

ThousandEyes GenAI required

To use the integration, your organization must not have opted out of ThousandEyes GenAI features.

To check your organization's status, log in to the ThousandEyes dashboard and go to Manage > Account Settings > Organization Settings > AI Features.

See the ThousandEyes documentation for more information.

To configure the ThousandEyes integration:

  1. Log in to the ThousandEyes dashboard.

  2. Navigate to Account Settings > Users and Roles > Profile > User API Tokens.

  3. Create a new bearer token and copy it.

  4. In the web app ThousandEyes integration page, populate the following:

Field

Description

ThousandEyes MCP URL (optional override)

Optionally, enter a ThousandEyes MCP URL to use as an override.

ThousandEyes API Bearer Token

ThousandEyes bearer token copied in step 3.

Custom Headers (Optional)

Add custom HTTP headers to include with all API requests. For each header, include the Header Name and Header Value. To add additional headers, click the + sign.

Deployment Type

Select whether your ThousandEyes integration is Cloud or On-Prem.

Certain integrations have endpoints that work only with one deployment type. Selecting the correct option automatically applies the right guardrails to the AI Incident Assistant agents.

Note: On-prem integrations can connect via the Relay Client, enabling secure communication with infrastructure behind your firewall.Remote Connections

ThousandEyes Agent Notes

Add notes to help AI Incident Assistant understand your ThousandEyes environment.

We highly recommend configuring this field, as it allows you to enter organization-specific information that enables AI Incident Assistant to provide more accurate and consistent results. 

For example, if your logs contain a specific field that is used to identify the type of application or service, provide that information here.

In this section: