Alertmanager CLI

amtool cheatsheet — create silences, check alerts, view routing with amtool silence add, amtool alert query. Manage Prometheus Alertmanager from the CLI.

6 min read

What it is

amtool is the command-line interface for Alertmanager, used to interact with its configuration, silences, and alerts.

Installation

Linux

# Download the latest release from GitHub
wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar xvfz alertmanager-0.27.0.linux-amd64.tar.gz
cd alertmanager-0.27.0.linux-amd64
sudo mv amtool /usr/local/bin/

Mac

# Download the latest release from GitHub
curl -LO https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.darwin-amd64.tar.gz
tar xvfz alertmanager-0.27.0.darwin-amd64.tar.gz
cd alertmanager-0.27.0.darwin-amd64
sudo mv amtool /usr/local/bin/

Windows

Download the appropriate .tar.gz file from the Alertmanager releases page and extract amtool.exe. Place it in a directory that’s in your system’s PATH.

Core Concepts

Alertmanager handles alerts sent by client applications like the Prometheus server. It receives alerts, deduplicates them, groups them, and routes them to the correct receiver integration (like email, PagerDuty, Slack). amtool allows you to manage these aspects from the command line.

  • Alerts: Notifications sent by Prometheus (or other clients) indicating a problem.
  • Silences: Temporary suppression of alerts matching specific criteria, preventing them from triggering notifications.
  • Receivers: Integrations that Alertmanager uses to send notifications (e.g., email, Slack, PagerDuty).
  • Routes: Configuration that determines which alerts go to which receivers based on labels.

Commands / Usage

Configuration Management

Interact with Alertmanager’s configuration.

  • Check configuration syntax:

    amtool config check /etc/alertmanager/alertmanager.yml

    Validates the syntax of the Alertmanager configuration file.

  • Print configuration:

    amtool config print --config.file=/etc/alertmanager/alertmanager.yml

    Prints the loaded configuration in YAML format.

  • Amtool requires a running Alertmanager instance to interact with its API for most operations. The following commands assume you have an Alertmanager running and accessible.

Silence Management

Create, list, and delete silences.

  • Create a silence:

    amtool --alertmanager.url=http://localhost:9093 silence add --starts '2023-10-27T10:00:00Z' --ends '2023-10-27T12:00:00Z' --comment 'Testing silencing' --creator 'admin' 'alertname=InstanceDown' 'job=prometheus'

    Creates a new silence that will match alerts with alertname: InstanceDown and job: prometheus from the specified start to end times.

  • List all silences:

    amtool --alertmanager.url=http://localhost:9093 silence list

    Lists all active silences.

  • List silences matching a specific label:

    amtool --alertmanager.url=http://localhost:9093 silence list --label 'alertname=InstanceDown'

    Lists silences that have the label alertname: InstanceDown.

  • Get details of a specific silence:

    amtool --alertmanager.url=http://localhost:9093 silence get <silence_id>

    Retrieves detailed information about a silence using its unique ID. You can get the ID from the list command.

  • Delete a silence:

    amtool --alertmanager.url=http://localhost:9093 silence delete <silence_id>

    Deletes an existing silence using its unique ID.

Alert Management

View and interact with alerts.

  • List all active alerts:

    amtool --alertmanager.url=http://localhost:9093 alert list

    Lists all currently firing alerts.

  • List alerts matching a specific label:

    amtool --alertmanager.url=http://localhost:9093 alert list --label 'severity=critical'

    Lists alerts that have the label severity: critical.

  • Get details of a specific alert:

    amtool --alertmanager.url=http://localhost:9093 alert get --alertname 'HighCpuUsage' --label 'instance=webserver-01'

    Retrieves detailed information about a specific alert by its name and labels.

Receiver Management

Interact with Alertmanager receivers.

  • List receivers:
    amtool --alertmanager.url=http://localhost:9093 receiver list
    Lists all configured receivers.

Status

Get the overall status of the Alertmanager instance.

  • Get Alertmanager status:
    amtool --alertmanager.url=http://localhost:9093 status
    Shows the current status of the Alertmanager cluster, including configuration and version.

General Flags

These flags apply to most amtool commands.

  • --alertmanager.url <url>: The URL of the Alertmanager instance (e.g., http://localhost:9093). This is mandatory for most API interactions.
  • --config.file <path>: Path to the Alertmanager configuration file. Used for config check and config print.
  • --web.timeout <duration>: Timeout for HTTP requests to Alertmanager (e.g., 5s).

Common Patterns

  • Silencing all alerts from a specific host:

    amtool --alertmanager.url=http://localhost:9093 silence add --ends 1h --comment 'Maintenance on server X' 'instance=server-x.example.com'

    Silences all alerts originating from server-x.example.com for the next hour.

  • Checking configuration and then reloading (if Alertmanager supports it via API):

    amtool config check /etc/alertmanager/alertmanager.yml && curl -X POST http://localhost:9093/-/reload

    First, check the syntax of your Alertmanager configuration. If it’s valid, then trigger a configuration reload in Alertmanager.

  • Finding alerts that are NOT silenced:

    # First, get all active silences and extract their label matchers
amtool --alertmanager.url=http://localhost:9093 silence list | jq -r '.[] | "\(.matchers | map("\(.name)=\(.value)") | join(","))"' > silenced_alerts.txt

# Then, list all alerts and filter out those that match any silenced alert
amtool --alertmanager.url=http://localhost:9093 alert list | grep -v -f silenced_alerts.txt

    This is a more complex pattern to identify alerts that are currently firing but are not covered by any active silence. It involves using jq to parse the JSON output of amtool silence list.

  • Creating a silence for a specific alert name and instance:

    amtool --alertmanager.url=http://localhost:9093 silence add --starts $(date -u +"%Y-%m-%dT%H:%M:%SZ") --ends $(date -u -d '+2h' +"%Y-%m-%dT%H:%M:%SZ") --comment 'Investigating alert' 'alertname=HighLatency' 'instance=api-service-pod-xyz'

    Creates a silence for two hours for alerts named HighLatency on the specific instance api-service-pod-xyz.

Gotchas

  • Alertmanager URL is crucial: Most commands require the --alertmanager.url flag to specify which Alertmanager instance to communicate with. Forgetting this is a common mistake.
  • Time formats for silences: The --starts and --ends flags for silences expect RFC3339 format (e.g., 2006-01-02T15:04:05Z07:00). Using relative times like now+1h is not directly supported by amtool itself, you’ll need to use shell commands like date to generate them.
  • Label matching syntax: When specifying labels for silences or alerts, use the key=value format. For exact matches, this is straightforward. For more complex matching (regex, etc.), you’d typically configure that within Alertmanager’s alertmanager.yml, and amtool displays the results of those configurations.
  • amtool config print vs. actual running configuration: amtool config print shows the configuration file content. The actual running configuration of Alertmanager might differ if it hasn’t been reloaded after a file change, or if there are dynamic configurations applied.
  • Alertmanager API availability: amtool relies on the Alertmanager API. If the API is disabled or inaccessible, amtool commands that interact with it will fail.
  • amtool config check only checks syntax: It validates the YAML structure and basic Alertmanager configuration schema but doesn’t guarantee that the configuration will function as intended in a live environment (e.g., receiver webhook URLs might be incorrect).