Datadog Experiments

Create a new Feature Management environment. *** FEATURE FLAG DETECTION *** If users mention: flags, toggles, feature switches, A/B tests, experiments, gradual rollouts, canary releases, or say they want to 'flag' something, this is feature flag work and should use feature flag tools. Use when reconciliation against list-environments shows a detected DD_ENV value has no covering environment ("will create" outcome), or when a user explicitly wants a new environment. WRITE OPERATION. Before calling, show the user the exact plan (name, queries, is_production, require_feature_flag_approval) and get explicit approval. Production environments have serious operational impact — confirm intent for is_production=true. queries are the DD_ENV values this environment covers (e.g. ["dev"], ["staging","stg"]). Wildcards are rejected by the service; DD_ENV values already used in another environment also cause a 409 Conflict.

Create a new experiment.

Create a new feature flag with a FEATURE_GATE allocation linked to a standard experiment. Use this tool instead of create-feature-flag when an allocation has experiment_id. The allocation schema and environment requirements are otherwise identical to create-feature-flag. MANDATORY: If the user did not specify an environment, ask the user to specify one before creating allocations. This allocation's exposure_schedule.rollout_options.scheduled_start cannot be a future value ('relative:<duration>' or a future 'absolute:<RFC3339>') because a standard experiment allocation cannot be auto-started — use 'none' and start the experiment through its lifecycle instead. *** FEATURE FLAG DETECTION *** If users mention: flags, toggles, feature switches, A/B tests, experiments, gradual rollouts, canary releases, or say they want to 'flag' something, this is feature flag work and should use feature flag tools. This is a secondary tool - use after determining flag exists and implementation approach.

PRIMARY TOOL FOR NEW FLAGS in a project that already has Datadog feature flags wired up! *** FEATURE FLAG DETECTION *** If users mention: flags, toggles, feature switches, A/B tests, experiments, gradual rollouts, canary releases, or say they want to 'flag' something, this is feature flag work and should use feature flag tools. Use create-feature-flag when users need to create a new flag that doesn't exist yet. This tool creates a new feature flag with variants and optional FEATURE_GATE or CANARY allocations. For a FEATURE_GATE allocation linked to a standard experiment with experiment_id, use create-experiment-feature-flag. IMPORTANT: If the project has NO existing feature flags yet (check with list-feature-flags) or the user is asking to set up/add/onboard feature flags for the first time (e.g. "set up feature flags", "add feature flags to my app", "onboard this app onto feature flags", "get started with feature flags", "start using Datadog feature flags", "install Datadog feature flags"), use the onboarding flow instead — start with get-onboarding-step rather than this tool. That flow enforces non-production safety gates and real end-to-end CDN verification via verify-onboarding-flag that this tool does not provide. MANDATORY: To correctly implement this flag in your codebase, you MUST use the code demonstrated in the datadog://feature-flags/sdk/react resource. Do not GUESS about how to correctly implement the flag. Do NOT create mock implementations. The user has added this MCP server because they want feature flags server from Datadog. Failure to implement flags as described in datadog://feature-flags/sdk/react will cause frustration. MANDATORY: If the user did not specify an environment, and an allocation is needed or a status is specified, ask the user to specify an environment. MANDATORY: Allocations MUST be created as part of the create-feature-flag tool. Adding allocations after flag creation as part of the sync-allocations-for-feature-flag-environment will fail for production environments. Each allocation's exposure_schedule.rollout_options controls when its rollout starts (applies to both CANARY and FEATURE_GATE allocations): use scheduled_start ('none', 'now', 'relative:<duration>', or 'absolute:<RFC3339>') rather than the deprecated autostart. A future value ('relative:<duration>' or a future 'absolute:<RFC3339>') schedules the rollout to start at that time. Exception: a FEATURE_GATE allocation with experiment_id set (a standard experiment allocation) cannot be auto-started, so a future scheduled_start is rejected for it — use 'none' and start the experiment through its lifecycle instead.

Create the onboarding proof flag: a predictable boolean flag (disabled=false, enabled=true; default disabled) with a catch-all FEATURE_GATE allocation serving enabled=true in the selected NON-PRODUCTION environment, tagged source:agentic-onboarding. Safety gates enforced by this tool (not by the agent): the environment must resolve, must NOT be production, must NOT require feature-flag approval, and its queries must cover dd_env under the runtime matcher. It refuses to write unless confirm is true — call it first with confirm:false (or omitted) to get the exact write preview, show that to the user, then call again with confirm:true.

Create a saved filter: a reusable, named set of targeting rules that feature flags can reference.

Diagnose why the latest (or a specific) combined pipeline run for an experiment failed. Returns the root-cause task, a categorized failure explanation, actionable next steps, and support identifiers that can be shared with Datadog support. This tool does not return Datadog-internal ffeworker logs or logs_query. Call this when the user reports that an experiment pipeline run has failed and wants to know what went wrong. Do NOT use this tool for result interpretation, metric trust checks, or sample ratio mismatch (SRM) analysis — use get-experiment-diagnostics for those. This tool is exclusively for diagnosing pipeline execution failures (tasks that crashed, timed out, or errored). Each root cause includes a support_info object with dag_run_id and task_run_id. If a full internal stack trace is needed, surface the support_info IDs so the user can share them with Datadog support. When the failure category is unknown, do not speculate about the root cause.

do-not-call

Segment computed experiment results by an assignment property or over time. Results use the experiment's configured analysis settings. Returns per-variant per-segment point estimates, confidence intervals, and p-values. Use after get-experiment-results when the caller asks 'how does this look for mobile users?' or 'how did this trend over time?'. Call get-experiment-diagnostics first; do not declare a segment winner if results are unreliable.

Get results from completed or in-progress canary experiments for a feature flag. This tool retrieves experimental data showing how each variant performed during canary testing, including metric measurements, statistical significance, and traffic distribution. Use this when you need to: - Review the performance of a canary deployment - Check if a canary rollout met its success criteria - Analyze metric differences between variants in a canary test - Understand why a canary was paused or aborted The tool returns data for ALL canary allocations associated with the feature flag. Each canary result includes: - Allocation metadata (name, key, environment, creation time) - Experiment status and state - Variant performance metrics (assignment counts, metric means, statistical lift, confidence intervals) - Guardrail metric information Canary results are available both during and after the canary run. Results during an active canary reflect data collected so far and will change as more traffic is processed. *** FEATURE FLAG DETECTION *** If users mention: flags, toggles, feature switches, A/B tests, experiments, gradual rollouts, canary releases, or say they want to 'flag' something, this is feature flag work and should use feature flag tools. This is a secondary tool - use after determining flag exists and implementation approach.

Get a single Datadog metric property's definition by ID (name, column, source type/subtype, description). Does NOT include discovered values — call list-datadog-metric-property-values for those. Obtain `metric_property_id` from list-datadog-metric-properties.

Get a specific experiment by ID.

Return the customer-warehouse assignment source SQL definition configured for an experiment, if one exists. Assignment sources define which subjects were assigned to which variant and when; a bug in this SQL (bad join, wrong column types, missing partition filter) is one of the most common causes of a broken analytics pipeline. Use this after get-experiment-pipeline-diagnostic surfaces a pipeline error, to inspect the assignment SQL as a possible root cause. If the experiment does not use a customer-warehouse assignment source (e.g. it uses a Datadog-default or reference-table source instead), this tool returns a message saying so rather than SQL — that is expected, not an error; look elsewhere (Datadog-default sources cannot fail this way).

Get a read-only health summary for a single experiment before interpreting experiment outcomes. Returns every diagnostic the analysis pipeline computed for the experiment — sample ratio mismatch (type=ASSIGNMENT_IMBALANCE), metric data/reliability checks (METRIC_HAS_DATA, METRIC_WINSORIZE_ZERO, PRE_EXPERIMENT_IMBALANCE), segment-level degradation (DIMENSIONAL_DEGRADATION), mixed assignments, dimensional assignment imbalance, implausible priors, and assignment/flag-evaluation presence checks — plus an experiment-level `status` rollup. Each entry's `description` is pre-rendered, human-readable copy (metric names and affected segments/variants already filled in) — but is only populated for non-passing checks; a PASS row has no `description`, only `title`/`status`. `meta` carries the raw numeric payload for every row regardless of status (e.g. ASSIGNMENT_IMBALANCE's `p_value`/`chi_square`/`observed_fractions`/`expected_fractions` keyed by variant_id, DIMENSIONAL_DEGRADATION's per-segment lift/CI/p-value) — its shape varies by `type`; use it when you need the actual numbers rather than just the pass/fail verdict. Use this before narrating results, comparing variants, or declaring a winner so the response can call out data-quality risks first. This tool does not return lift/effect-size results and does not decide the winning variant. PIPELINE_STATUS is not included here — use get-experiment-pipeline-diagnostic for that. When a diagnostic has status FAIL or WARN, do not silently ignore it: for `ASSIGNMENT_IMBALANCE`, the experiment's random assignment may be broken — do not interpret variant-level differences, do not declare a winner, and recommend investigating assignment integrity first. For `DIMENSIONAL_DEGRADATION`, call out the affected segment(s) rather than only reporting the aggregate lift. For any metric-scoped failure (`METRIC_HAS_DATA`, `METRIC_WINSORIZE_ZERO`, `PRE_EXPERIMENT_IMBALANCE`), surface the flag on that metric_id and avoid over-interpreting its magnitude in get-experiment-results.

Return the failed customer-warehouse queries executed during a specific pipeline run, to help pinpoint exactly which SQL definition (assignment source or a metric source) caused a pipeline failure. Each entry includes the query text, its data source type, status, and the dag_task_run_id it ran under. Use this after get-experiment-pipeline-diagnostic identifies a pipeline error and gives you a dag_run_id — pass that dag_run_id here to see the actual failing SQL, rather than guessing based on get-experiment-assignment-sql / list-experiment-metric-sql-definitions alone. This is scoped to customer-warehouse queries only; it does not return Datadog-internal ffeworker logs (use diagnose-experiment-run-failure for those).

Detect whether the treatment effect in one experiment changes across variants of another. Call get-experiment-diagnostics for both experiments first. Proceed only if both responses have overall status PASS; stop for every other status. Passing those checks does not establish cross-experiment assignment independence: treat assignment_mismatch_detected (joint goodness-of-fit p < 0.001) as authoritative and stop if true. The estimand defaults to relative lift. Select absolute lift only when the user explicitly asks for absolute lift or difference in means. Report exactly one result with the user-facing label Relative Lift or Absolute Lift; never expose backend estimand identifiers. For the selected estimand, inspect every stratum result. If any stratum has is_unreliable=true, surface its unreliable_reason and do not report interaction_detected. Otherwise, report cuped and interaction_detected as authoritative. Use the global Wald rule (p < 1-confidence_level) only to explain the verdict; differing stratum-level significance does not establish an interaction.

Return the PIPELINE_STATUS diagnostic for an experiment — the platform's verdict on whether the experiment's most recent SQL-driven analytics pipeline (assignments + metrics) ran successfully. If present, the diagnostic includes an `error` message, a `resolution_prompt` describing the likely fix, an optional `resolution_link`, and the `dag_run_id` of the pipeline run it was computed from. Call this FIRST whenever a user reports that an experiment's pipeline or analysis 'failed', 'has no data', or 'looks broken' — before reaching for any other diagnostic tool. Its `dag_run_id` output feeds directly into get-experiment-failed-queries to find the exact query that failed. This tool is specifically for SQL-DEFINITION-caused pipeline failures (bad assignment SQL, bad metric SQL, warehouse permission errors, etc.) — NOT for interpreting experiment results, sample ratio mismatch (SRM), or metric trust (use get-experiment-diagnostics for those), and NOT for diagnosing dag/task-level execution crashes, timeouts, or Datadog-internal orchestration errors (use diagnose-experiment-run-failure for those). If no PIPELINE_STATUS diagnostic exists yet, this tool returns a message saying so — that does not necessarily mean the pipeline is healthy, only that no diagnostic has been computed.

Get computed per-variant per-metric experiment results. Each metric carries an authoritative `verdict` field — one of `better`, `worse`, `inconclusive`, or `unreliable` — derived from the platform's significance classifier (CI excludes zero AND sign matches the metric's desired_change). TREAT `verdict` AS AUTHORITATIVE: do not re-derive significance from raw p-values or confidence intervals to overturn it. When verdict=unreliable, do not reason about magnitudes — the platform deems the result not trustworthy; surface the unreliable_reason and stop. Magnitudes (point estimates, lifts, confidence intervals, p-values, z-scores, sample sizes) are returned for narration only. Results are filtered to the experiment's configured CUPED setting; the alternate flavor is not exposed. Call get-experiment-diagnostics first so SRM warnings or zero-data metrics are surfaced before interpreting these numbers; never declare an overall experiment winner from this tool's output alone.

Return the concrete value set for a segmentation property (e.g. ["mobile", "desktop", "tablet"] for device_type). Use this to pick valid filter values BEFORE calling explore-experiment-results with a `split_by` filter — do not guess values, since filter strings are case- and spelling-sensitive. Always call list-experiment-segmentation-properties FIRST to obtain `property_id` and `source_type`. Both fields come directly from that tool's output and must be passed through verbatim. Obtain `experiment_id` from list-experiments or get-experiment.

Gets variants and status in environment for a specific feature flag by ID or key. Provide featureFlagID or featureFlagKey (or both). *** FEATURE FLAG DETECTION *** If users mention: flags, toggles, feature switches, A/B tests, experiments, gradual rollouts, canary releases, or say they want to 'flag' something, this is feature flag work and should use feature flag tools. This is a secondary tool - use after determining flag exists and implementation approach. MANDATORY: You MUST implement using the exact patterns from datadog://feature-flags/sdk/react resource. Do NOT create mock implementations.

Return the locations in the customer's source code where a feature flag is used, so you can investigate whether the flag and its metric tracking are wired up correctly. Use this when a flag-backed experiment records no flag evaluations or no metric data (FLAG_HAS_EVALUATIONS / METRIC_HAS_DATA diagnostics) — the root cause is usually in the application code, not the experiment config. Returns two lists: - `evaluations`: SDK call sites where the flag is evaluated at runtime (the spots that decide which variant a subject sees). Check that the SDK is initialized and that metric tracking events fire near these. - `references`: other mentions of the flag (constants, config). Each entry has `repository`, `file_path`, and `start_line`. An empty result is meaningful: the flag is likely not integrated into the code yet, or Datadog has not indexed the repository. Obtain `feature_flag_id` from get-experiment (`feature_flag_id` field). Org-scoped.

Get details for a specific guardrail metric by ID. Use this to inspect a metric's full details including description and aggregation configuration. *** FEATURE FLAG DETECTION *** If users mention: flags, toggles, feature switches, A/B tests, experiments, gradual rollouts, canary releases, or say they want to 'flag' something, this is feature flag work and should use feature flag tools. This is a secondary tool - use after determining flag exists and implementation approach.

Return the definition of a Product Analytics / experiment metric (also called PA metric, business metric, decision metric, guardrail metric, KPI, conversion metric) so the agent can investigate WHY the metric moved. Call this any time the user asks about the cause of a metric movement on an experiment, feature flag rollout, or business outcome — not just to display the metric. Not for Datadog infra/APM timeseries metrics (CPU, latency, request count); for those use the Datadog MCP `get_datadog_metric` / `search_datadog_metrics`. Response selects one of three branches via `bridge_status`: - `datadog`: response includes `recommended_tool_call` ({tool_name, dataset}) naming which DD MCP tool to call (aggregate_rum_events for RUM, aggregate_product_analytics_events for Product Analytics) plus the structured pieces you need to assemble its args yourself: `event_type`, `filter_query`, `aggregation` (the metric's operation label), `aggregated_field`, and `threshold_*` for threshold metrics. Use the per-tool aggregator-mapping rules in the response's guidance to build args. Empty result from the executed call = no instrumentation; stop, do not broaden. Inspect `translation_warnings` for filter pieces that were dropped. - `warehouse`: response includes raw `sql`; hand to the user's warehouse tooling. Experiment-scoped SQL is out of scope here. - `unknown`: data_source_type not bridged (EXPOSURES, future enums); surface the raw definition and admit uncertainty. Org-scoped — no experiment ID needed.

Use this FIRST — before create-feature-flag — whenever a user asks to set up, add, or onboard Datadog feature flags in a project that doesn't have them yet: e.g. "set up feature flags", "add feature flags to my app", "get started with feature flags", "onboard this app onto feature flags", "start using Datadog feature flags", "install Datadog feature flags". Prefer it for any first-time/from-scratch feature-flag integration, since it enforces non-production safety gates and provides real end-to-end CDN verification (via verify-onboarding-flag) that create-feature-flag does not. Plan the next step of Datadog feature-flag onboarding. Read-only: it makes no writes and no upstream calls. Pass the current onboarding 'state' (empty to start). Returns the next instructions and the expected next-state shape. All onboarding branching lives here — call it at the start of each step and follow the returned instructions. Never put a client token in state.

Get a saved filter by id, including its targeting rules.

Get a single warehouse metric property by ID, including its discovered values (with observed counts). Warehouse metric properties are breakdownable dimensions on a warehouse-backed metric's source; they are created via metric-sync, so there is no org-wide list endpoint — fetch one by its known ID. These are METRIC properties, NOT experiment segmentation/assignment properties.

do-not-call

Load a saved Product Analytics chart's full definition by UUID. Use this AFTER search_product_analytics_org_entities returns a saved chart you want to reproduce. The response includes the chart's name, type (funnel / retention_curve / sankey / ...), and the raw widget definition JSON containing the actual query parameters (cohort/return queries, filters, time interval, etc.). Parse the definition to feed the matching run_* tool (run_product_analytics_journey / run_product_analytics_retention / run_product_analytics_pathway) with the chart's real parameters. When to call: - The user wants to reproduce, inspect, or show data from a saved PA chart whose UUID you resolved via search_product_analytics_org_entities. - You need the chart's exact cohort/return queries, filters, or time interval to construct a follow-up analytics call. When NOT to call: - You only need the chart's existence / name / type — search_product_analytics_org_entities already returned that. - The user wants to enumerate charts — search the saved_charts source without a query. Returns: a JSON object with name, type, and the raw widget definition. The definition's shape depends on the chart type; common fields include "requests" (array of query specs), "time" (default time window), and chart-type-specific query parameters under requests[*].query.search.

Get an AI-generated play-by-play of what a user did during a session replay — what pages they visited, what actions they took, and what happened step by step. Use this tool when the user wants to understand the content of a specific session: "what did the user do", "summarize this session", "what happened in this replay", "give me a play-by-play". Typically called after search_replays to summarize a session from the results. Returns a natural language summary and time-based chapters, plus a direct link to the replay. Use "rum" datasource when working with RUM replays, or "product_analytics" (default) for Product Analytics replays. This should match the datasource used when you found the session via search_replays.

do-not-call

Link a feature flag to an experiment.

List allocations for a specific feature flag. *** FEATURE FLAG DETECTION *** If users mention: flags, toggles, feature switches, A/B tests, experiments, gradual rollouts, canary releases, or say they want to 'flag' something, this is feature flag work and should use feature flag tools. This is a secondary tool - use after determining flag exists and implementation approach. For context on using feature flags in applications, see the datadog://feature-flags/sdk/react resource.

List the Datadog metric properties an org has defined for a given Datadog data source — the breakdownable dimensions (e.g. device_type, country, plan_tier) a metric on that source can be split into sub-metrics by. These are METRIC properties (attached to a metric's Datadog source), NOT experiment segmentation/assignment properties — use list-experiment-segmentation-properties for splitting experiment results. Both `datadog_source_type` and `datadog_source_subtype` are required. To find them for a specific metric, call get-metric-definition first and read `datadog_bridge.source_type` / `datadog_bridge.source_subtype`. Each entry carries an `id`, `property_name`, `property_column`, and source fields. Pass `id` to list-datadog-metric-property-values to enumerate discovered values, or to get-datadog-metric-property for the full definition.

Return the discovered values for a Datadog metric property (e.g. ["mobile", "desktop", "tablet"] for device_type), each with an observed count, sorted by count descending. Use this to see which concrete values a metric can be broken out by — do not guess values, they are case- and spelling-sensitive. Obtain `metric_property_id` from list-datadog-metric-properties.

List all flagging environments for the organization. *** FEATURE FLAG DETECTION *** If users mention: flags, toggles, feature switches, A/B tests, experiments, gradual rollouts, canary releases, or say they want to 'flag' something, this is feature flag work and should use feature flag tools. This is a secondary tool - use after determining flag exists and implementation approach. For context on how environments work with feature flags, see the datadog://feature-flags/sdk/react resource.

List the SQL definitions of every warehouse metric source attached to an experiment's decision/guardrail metrics. Warehouse metric sources define how a customer's warehouse data is queried to compute a metric; besides the assignment source (see get-experiment-assignment-sql), these are the only customer-controlled SQL in the pipeline and a common cause of pipeline failures (bad column references, type mismatches, permission errors). Use this alongside get-experiment-assignment-sql when diagnosing a pipeline failure surfaced by get-experiment-pipeline-diagnostic. Metrics sourced from Datadog (RUM, Product Analytics) have no warehouse SQL and will not appear here — an empty result means no warehouse metric sources are attached, not an error.

List the assignment properties an experiment can be segmented / split by (e.g. device_type, country, plan_tier). Call this BEFORE explore-experiment-results whenever you need a `split_by_property_id` and don't already have one — do not invent or guess property UUIDs. Obtain `experiment_id` from list-experiments (browse) or get-experiment (lookup by name). Output unifies properties from both Datadog-default and warehouse assignment sources; each entry carries an `id`, `name`, `column_name`, and `source_type` (`datadog_default` or `warehouse`). Pass BOTH `id` and `source_type` to get-experiment-segmentation-property-values to enumerate concrete filter values, or pass `id` to explore-experiment-results as `split_by_property_id`. If an experiment has only one source configured, the other source's empty/4xx response is surfaced as a soft warning rather than an error.

List experiments for the organization.

List all feature flags for the organization. *** FEATURE FLAG DETECTION *** If users mention: flags, toggles, feature switches, A/B tests, experiments, gradual rollouts, canary releases, or say they want to 'flag' something, this is feature flag work and should use feature flag tools. This is a secondary tool - use after determining flag exists and implementation approach. If this returns an empty list and the user's goal is initial setup (e.g. "set up feature flags", "add feature flags to my app", "onboard this app onto feature flags", "start using Datadog feature flags", "install Datadog feature flags"), prefer the onboarding flow — start with get-onboarding-step — over creating a flag directly with create-feature-flag. The onboarding flow enforces non-production safety gates and provides real end-to-end CDN verification via verify-onboarding-flag that create-feature-flag does not.

List all available guardrail metrics for the organization. Use this tool to discover metric IDs needed for allocation guardrails. Results are paginated to prevent context window overflow. *** FEATURE FLAG DETECTION *** If users mention: flags, toggles, feature switches, A/B tests, experiments, gradual rollouts, canary releases, or say they want to 'flag' something, this is feature flag work and should use feature flag tools. This is a secondary tool - use after determining flag exists and implementation approach.