Grafana

Important Capabilities

Capability	Status	Notes
Column-level Lineage	✅	Enabled by default
Detect Deleted Entities	✅	Enabled by default
Extract Ownership	✅	Enabled by default
Extract Tags	✅	Enabled by default
Platform Instance	✅	Enabled by default
Table-Level Lineage	✅	Enabled by default

This plugin extracts metadata from Grafana and ingests it into DataHub. It connects to Grafana's API to extract metadata about dashboards, charts, and data sources. The following types of metadata are extracted:

• Container Entities:

  • Folders: Top-level organizational units in Grafana
  • Dashboards: Collections of panels and charts
  • The full container hierarchy is preserved (Folders -> Dashboards -> Charts/Datasets)

• Charts and Visualizations:

  • All panel types (graphs, tables, stat panels, etc.)
  • Chart configuration and properties
  • Links to the original Grafana UI
  • Custom properties including panel types and data source information
  • Input fields and schema information when available

• Data Sources and Datasets:

  • Physical datasets representing Grafana's data sources
  • Dataset schema information extracted from queries and panel configurations
  • Support for various data source types (SQL, Prometheus, etc.)
  • Custom properties including data source type and configuration

• Lineage Information:

  • Dataset-level lineage showing relationships between:
    • Source data systems and Grafana datasets
    • Grafana datasets and charts
  • Column-level lineage for SQL-based data sources
  • Support for external source systems through configurable platform mappings

• Tags and Ownership:

  • Dashboard and chart tags
  • Ownership information derived from:
    • Dashboard creators
    • Technical owners based on dashboard UIDs
    • Custom ownership assignments

The source supports the following capabilities:

• Platform instance support for multi-Grafana deployments
• Stateful ingestion with support for soft-deletes
• Fine-grained lineage at both dataset and column levels
• Automated tag extraction
• Support for both HTTP and HTTPS connections with optional SSL verification

Prerequisites:

1. A running Grafana instance
2. A service account token with permissions to:
   • Read dashboards and folders
   • Access data source configurations
   • View user information

A sample configuration file:

source:
    type: grafana
    config:
        # Coordinates
        platform_instance: production # optional
        env: PROD # optional
        url: https://grafana.company.com
        service_account_token: ${GRAFANA_SERVICE_ACCOUNT_TOKEN}

        # SSL verification for HTTPS connections
        verify_ssl: true # optional, default is true

        # Source type mapping for lineage
        connection_to_platform_map:
            postgres:
                platform: postgres
                database: grafana  # optional
                database_schema: grafana  # optional
                platform_instance: database_2  # optional
                env: PROD  # optional
            mysql_uid_1:  # Grafana datasource UID
                platform: mysql
                platform_instance: database_1  # optional
                database: my_database  # optional

CLI based Ingestion

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

Field	Description
service_account_token ✅ string(password)	Grafana API token
url ✅ string	URL of Grafana instance (e.g. https://grafana.company.com)
ingest_owners boolean	Whether to ingest owners from Grafana dashboards and charts Default: True
ingest_tags boolean	Whether to ingest tags from Grafana dashboards and charts Default: True
platform_instance string	Platform instance for DataHub
platform_instance_map map(str,string)
verify_ssl boolean	Verify SSL certificate for secure connections (https) Default: True
env string	The environment that all assets produced by this connector belong to Default: PROD
connection_to_platform_map map(str,PlatformConnectionConfig)
connection_to_platform_map.key.platform ❓ string	Platform to connect to
connection_to_platform_map.key.database string	Database name
connection_to_platform_map.key.database_schema string	Schema name
connection_to_platform_map.key.platform_instance string	Platform instance
connection_to_platform_map.key.env string	Environment Default: PROD
stateful_ingestion StatefulStaleMetadataRemovalConfig	Base specialized config for Stateful Ingestion with stale metadata removal capability.
stateful_ingestion.enabled boolean	Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False Default: False
stateful_ingestion.remove_stale_metadata boolean	Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled. Default: True

Schema

The JSONSchema for this configuration is inlined below.

{
  "title": "GrafanaSourceConfig",
  "description": "Base configuration class for stateful ingestion for source configs to inherit from.",
  "type": "object",
  "properties": {
    "env": {
      "title": "Env",
      "description": "The environment that all assets produced by this connector belong to",
      "default": "PROD",
      "type": "string"
    },
    "platform_instance_map": {
      "title": "Platform Instance Map",
      "description": "A holder for platform -> platform_instance mappings to generate correct dataset urns",
      "type": "object",
      "additionalProperties": {
        "type": "string"
      }
    },
    "stateful_ingestion": {
      "$ref": "#/definitions/StatefulStaleMetadataRemovalConfig"
    },
    "url": {
      "title": "Url",
      "description": "URL of Grafana instance (e.g. https://grafana.company.com)",
      "type": "string"
    },
    "service_account_token": {
      "title": "Service Account Token",
      "description": "Grafana API token",
      "type": "string",
      "writeOnly": true,
      "format": "password"
    },
    "verify_ssl": {
      "title": "Verify Ssl",
      "description": "Verify SSL certificate for secure connections (https)",
      "default": true,
      "type": "boolean"
    },
    "ingest_tags": {
      "title": "Ingest Tags",
      "description": "Whether to ingest tags from Grafana dashboards and charts",
      "default": true,
      "type": "boolean"
    },
    "ingest_owners": {
      "title": "Ingest Owners",
      "description": "Whether to ingest owners from Grafana dashboards and charts",
      "default": true,
      "type": "boolean"
    },
    "platform_instance": {
      "title": "Platform Instance",
      "description": "Platform instance for DataHub",
      "type": "string"
    },
    "connection_to_platform_map": {
      "title": "Connection To Platform Map",
      "description": "Map of Grafana connection names to their upstream platform details",
      "default": {},
      "type": "object",
      "additionalProperties": {
        "$ref": "#/definitions/PlatformConnectionConfig"
      }
    }
  },
  "required": [
    "url",
    "service_account_token"
  ],
  "additionalProperties": false,
  "definitions": {
    "DynamicTypedStateProviderConfig": {
      "title": "DynamicTypedStateProviderConfig",
      "type": "object",
      "properties": {
        "type": {
          "title": "Type",
          "description": "The type of the state provider to use. For DataHub use `datahub`",
          "type": "string"
        },
        "config": {
          "title": "Config",
          "description": "The configuration required for initializing the state provider. Default: The datahub_api config if set at pipeline level. Otherwise, the default DatahubClientConfig. See the defaults (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/graph/client.py#L19).",
          "default": {},
          "type": "object"
        }
      },
      "required": [
        "type"
      ],
      "additionalProperties": false
    },
    "StatefulStaleMetadataRemovalConfig": {
      "title": "StatefulStaleMetadataRemovalConfig",
      "description": "Base specialized config for Stateful Ingestion with stale metadata removal capability.",
      "type": "object",
      "properties": {
        "enabled": {
          "title": "Enabled",
          "description": "Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False",
          "default": false,
          "type": "boolean"
        },
        "remove_stale_metadata": {
          "title": "Remove Stale Metadata",
          "description": "Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.",
          "default": true,
          "type": "boolean"
        }
      },
      "additionalProperties": false
    },
    "PlatformConnectionConfig": {
      "title": "PlatformConnectionConfig",
      "type": "object",
      "properties": {
        "platform": {
          "title": "Platform",
          "description": "Platform to connect to",
          "type": "string"
        },
        "database": {
          "title": "Database",
          "description": "Database name",
          "type": "string"
        },
        "database_schema": {
          "title": "Database Schema",
          "description": "Schema name",
          "type": "string"
        },
        "platform_instance": {
          "title": "Platform Instance",
          "description": "Platform instance",
          "type": "string"
        },
        "env": {
          "title": "Env",
          "description": "Environment",
          "default": "PROD",
          "type": "string"
        }
      },
      "required": [
        "platform"
      ],
      "additionalProperties": false
    }
  }
}

Code Coordinates

• Class Name: datahub.ingestion.source.grafana.grafana_source.GrafanaSource
• Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Grafana, feel free to ping us on our Slack.