HashiCorp Consul

Consul cheatsheet — service discovery, KV store, health checks, connect mesh. consul agent, consul kv put, consul catalog services. Full HashiCorp Consul reference.

10 min read

What it is

HashiCorp Consul is a tool for service discovery, configuration, and segmentation, enabling a distributed system to discover and communicate with each other. You reach for it when you need a robust, centralized way to manage services in dynamic, distributed environments.

Installation

Linux

# Download the latest version
wget https://releases.hashicorp.com/consul/1.15.0/consul_1.15.0_linux_amd64.zip

# Unzip the binary
unzip consul_1.15.0_linux_amd64.zip

# Move to a directory in your PATH
sudo mv consul /usr/local/bin/

# Verify installation
consul --version

macOS

# Download the latest version
curl -LO https://releases.hashicorp.com/consul/1.15.0/consul_1.15.0_darwin_amd64.zip

# Unzip the binary
unzip consul_1.15.0_darwin_amd64.zip

# Move to a directory in your PATH
sudo mv consul /usr/local/bin/

# Verify installation
consul --version

Windows

Download the appropriate .zip file from the Consul releases page. Extract the consul.exe file and place it in a directory included in your system’s PATH environment variable.

Core Concepts

  • Agent: Every Consul node runs an agent. Agents can operate in either client or server mode.
    • Client Agent: Lightweight, forwards requests to a server. Typically runs on every application node.
    • Server Agent: Stateful, holds the cluster state, processes requests, and communicates with other servers. A cluster needs at least 3 servers for fault tolerance.
  • Service: A logical unit of functionality (e.g., web-app, database). Services are registered with Consul.
  • Check: A health check associated with a service or a node. Consul uses checks to determine the health of services and nodes.
  • Service Discovery: Clients can query Consul to find the network locations (IP address and port) of healthy instances of a service.
  • Key/Value Store: Consul provides a distributed, highly available key/value store for dynamic configuration.
  • Consul Catalog: A registry of all services and their associated health checks.
  • Consul Connect: Provides service-to-service communication with automatic TLS encryption and identity.

Commands / Usage

Starting Consul Agents

Starting a Development Server (Single Node)

This is useful for local testing and development. It runs a single server agent and a client agent on the same machine.

consul agent -dev

Explanation: Starts Consul in development mode, suitable for a single machine.

Starting a Server Agent

This command starts a server agent. You’ll typically run this on dedicated nodes for your Consul cluster.

consul agent -server -ui -node=server-1 -bootstrap-expect=3 -data-dir=/opt/consul -client=0.0.0.0

Explanation:

  • -server: Starts the agent in server mode.
  • -ui: Enables the Consul web UI.
  • -node=server-1: Assigns a unique name to this node.
  • -bootstrap-expect=3: Specifies the expected number of server nodes in the cluster for initial bootstrapping.
  • -data-dir=/opt/consul: Specifies the directory where Consul stores its state.
  • -client=0.0.0.0: Binds the client interface to all network interfaces, allowing other machines to connect.

Starting a Client Agent

This command starts a client agent. It needs to join a Consul cluster.

consul agent -node=client-1 -join=<server_ip_address> -data-dir=/opt/consul -client=0.0.0.0

Explanation:

  • -node=client-1: Assigns a unique name to this node.
  • -join=<server_ip_address>: Joins an existing Consul cluster by connecting to a server agent at the given IP address.
  • -data-dir=/opt/consul: Specifies the directory where Consul stores its state.
  • -client=0.0.0.0: Binds the client interface to all network interfaces.

Service Registration

Registering a Service from a Configuration File

Create a JSON file (e.g., web-app.json):

{
  "service": {
    "name": "web-app",
    "tags": ["frontend", "production"],
    "port": 8080,
    "check": {
      "http": "http://localhost:8080/health",
      "interval": "10s"
    }
  }
}

Then register it:

consul services register web-app.json

Explanation: Registers the web-app service defined in web-app.json with Consul.

Registering a Service with Inline Flags

consul services register -name="api-service" -port=5000 -tag="backend" -check-http="http://localhost:5000/health" -check-interval="15s"

Explanation: Registers a service named api-service with specified port, tag, and health check directly via command-line flags.

Registering a Service with a Script Check

Create a script check file (e.g., redis-check.json):

{
  "service": {
    "name": "redis",
    "port": 6379
  },
  "check": {
    "script": "/usr/local/bin/check_redis.sh",
    "interval": "30s"
  }
}

Then register it:

consul services register redis-check.json

Explanation: Registers the redis service and associates it with a script-based health check.

Service Discovery

Discovering Services by Name

consul services -name=web-app

Explanation: Lists all registered instances of the web-app service.

Discovering Services with a Specific Tag

consul services -tag=production

Explanation: Lists all services that have the production tag.

Discovering Services and their Health Status

consul services --tag=production --healthy

Explanation: Lists only the healthy instances of services tagged with production.

Querying for Service Addresses

consul services -name=database -format=json | jq '.[].ServiceAddress'

Explanation: Retrieves the IP addresses of all healthy database service instances using jq for JSON parsing.

Health Checks

Listing All Checks

consul checks

Explanation: Lists all health checks registered in Consul.

Listing Checks for a Specific Node

consul checks -node=server-1

Explanation: Lists all health checks associated with the node named server-1.

Listing Checks for a Specific Service

consul checks -service=web-app

Explanation: Lists all health checks associated with the web-app service.

Deregistering a Check

consul checks deregister <check_id>

Explanation: Removes a specific health check from Consul using its ID.

Key/Value Store

Setting a Key

consul kv put config/database/url "postgres://user:pass@host:5432/db"

Explanation: Stores the value postgres://user:pass@host:5432/db under the key config/database/url.

Getting a Key

consul kv get config/database/url

Explanation: Retrieves the value associated with the key config/database/url.

Listing Keys

consul kv list config/

Explanation: Lists all keys under the config/ prefix.

Deleting a Key

consul kv delete config/database/url

Explanation: Removes the key config/database/url and its associated value from Consul.

Watching for Key Changes

consul kv watch config/api/timeout

Explanation: Continuously monitors the key config/api/timeout and prints its value whenever it changes.

ACLs (Access Control Lists)

Creating a Policy

Create a policy file (e.g., readonly.hcl):

policy "readonly" {
  description = "Read-only access to services and kv"
  rules = {
    "service" = {
      "*" = {
        "policy" = "read"
      }
    }
    "kv" = {
      "*" = {
        "policy" = "read"
      }
    }
  }
}

Then create the policy:

consul acl policy create -name="readonly" -description="Read-only access to services and kv" -rules-file="readonly.hcl"

Explanation: Creates a new ACL policy named "readonly" with defined read-only rules.

Creating a Token

consul acl token create -policy-name="readonly" -accessor-id="my-readonly-accessor" -name="web-app-token"

Explanation: Creates a new ACL token associated with the "readonly" policy, giving it a specific accessor ID and name.

Listing Tokens

consul acl token list

Explanation: Lists all ACL tokens in the Consul cluster.

Updating a Token

consul acl token update -id=<token_id> -name="updated-web-app-token"

Explanation: Updates the name of an existing ACL token identified by <token_id>.

Revoking a Token

consul acl token revoke -id=<token_id>

Explanation: Revokes (invalidates) an ACL token.

Consul Connect (Service Mesh)

Enabling Connect on a Service Definition

Modify your service registration file (e.g., api-service.json):

{
  "service": {
    "name": "api-service",
    "port": 5000,
    "connect": {
      "sidecar_service": {}
    }
  }
}

Register the service:

consul services register api-service.json

Explanation: Registers the api-service and automatically provisions a sidecar proxy for it, enabling Consul Connect.

Proxying Traffic Through a Service Mesh

consul connect proxy -service=web-app -upstream=api-service:5000

Explanation: Starts a proxy that listens locally, forwarding traffic to the api-service via the Consul Connect mesh.

Executing a Command with a Service Identity

consul connect call api-service ./my-client-app --target=localhost:5000

Explanation: Runs my-client-app with the identity of the web-app service, allowing it to securely connect to api-service.

Common Patterns

Running Consul in Docker

docker run -d -p 8500:8500 -p 8600:8600/udp consul agent -dev -client=0.0.0.0

Explanation: Starts a single-node Consul agent in a Docker container, exposing the HTTP and DNS ports.

Joining a Consul Cluster

On a new client or server node:

consul join <existing_server_ip_1> <existing_server_ip_2>

Explanation: Tells the current Consul agent to join the cluster by connecting to one or more existing server nodes.

Registering a Service and its Health Check

consul services register -name="my-app" -port=9000 -check-http="http://localhost:9000/health" -check-interval="5s"

Explanation: A common way to quickly register a service with its health check, ensuring Consul knows its status.

Getting all healthy instances of a service

consul services -name=user-service --healthy -format=json | jq -r '.[] | "\(.ServiceAddress):\(.ServicePort)"'

Explanation: Fetches all healthy instances of user-service and outputs them as IP:PORT strings, useful for dynamic load balancing.

Using Consul KV for application configuration

Ensure your application reads configuration from Consul:

# Example: Node.js app reading a config value
const consul = require('consul');
consul().kv.get('config/my-app/feature-flag', (err, data) => {
  if (!err && data) {
    console.log('Feature flag is:', data.Value);
  }
});

Explanation: Applications can dynamically fetch their configuration from Consul’s KV store, enabling zero-downtime configuration updates.

Setting up Consul Connect for secure inter-service communication

  1. Register services with connect.sidecar_service.
  2. Use consul connect proxy or consul connect call to interact.
    # Example: Proxying a request from a client to a backend service
consul connect proxy -register -service=my-client-proxy -upstream=backend-service:8080 -listen-addr=127.0.0.1:9090 &
curl http://127.0.0.1:9090

Explanation: Establishes secure, encrypted communication paths between services without application code changes.

Gotchas

  • agent -dev is not for production: The -dev mode runs a single-node cluster, lacks fault tolerance, and isn’t suitable for production environments.
  • Server Mode is Stateful: Server agents store the cluster state. Losing a majority of servers can lead to data loss or cluster unavailability. Ensure you have at least 3 servers for HA.
  • bootstrap-expect: This flag is crucial for bootstrapping a new HA cluster. Once the cluster is formed, it should not be changed without careful consideration.
  • Client Interface Binding (-client): By default, Consul agents bind the client interface to 127.0.0.1. If you need other machines to reach your Consul agent (e.g., for service discovery queries), you must explicitly set -client to 0.0.0.0 or a specific IP address.
  • DNS Interface Binding: Similar to the client interface, the DNS interface also defaults to 127.0.0.1. If you want other machines to use Consul’s DNS for service discovery, you need to configure this. The -dns flag can be used, or ensure the agent is bound to 0.0.0.0 and firewall rules allow access.
  • Service Registration vs. Deregistration: If a Consul agent process dies unexpectedly, its registered services and checks might remain in the catalog until their TTL expires or they are manually deregistered. Ensure proper shutdown procedures or use health checks with appropriate timeouts.
  • ACLs and Bootstrapping: If ACLs are enabled, the first token used to interact with Consul (often created during bootstrapping) must have sufficient permissions to manage ACLs.
  • Consul Connect TLS Certificates: Consul automatically manages TLS certificates for Connect services. Ensure the agents have write access to their data-dir for certificate storage.
  • HTTP API vs. CLI: While the CLI is convenient, the HTTP API is the underlying mechanism. Understanding the API can be helpful for complex integrations or troubleshooting.