Hashicorp Consul - How to do verified TLS from Pods in Kubernetes cluster

Solved with thanks to Hashicorp on their Consul discussion forum.

• Create a Kubernetes secret named consul with a key named CONSUL_GOSSIP_ENCRYPTION_KEY and an appropriate encryption key value.
  • Generate value using consul keygen
• Install the hashicorp/consul Helm chart with an values-override.yaml , such as below:

global:
  datacenter: sandbox

  gossipEncryption:
    secretName: "consul"
    secretKey: "CONSUL_GOSSIP_ENCRYPTION_KEY"

  tls:
    enabled: true
    httpsOnly: true
    enableAutoEncrypt: true
    serverAdditionalDNSSANs: ["'consul.service.consul'"]

server:
  replicas: 3
  bootstrapExpect: 3
  storage: 20Gi

dns:
  clusterIP: 172.20.53.53

ui:
  service:
    type: 'LoadBalancer'

syncCatalog:
  enabled: true

• Create an example Pod spec to represent our application.
  • Ensure it mounts the Consul server CA cert secret.
  • Ensure the Pod’s container has HOST_IP exposed as an environment variable.

apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: test-pod
spec:
  volumes:
  - name: consul-consul-ca-cert
    secret:
      secretName: consul-consul-ca-cert
  hostNetwork: false
  containers:
  - name: consul-test-pod
    image: alpine
    imagePullPolicy: IfNotPresent
    env:
    - name: HOST_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    command: ["/bin/sh"]
    args: ["-c", "while true; do sleep 24h; done"]
    volumeMounts:
    - name: consul-consul-ca-cert
      mountPath: /consul/tls/ca

• Upon creation of the Pod, kubectl exec into it, and ensure the ca-certificates and curl packages are installed (I’m using Alpine Linux in this example).
  • (curl is purely for testing purposes)

#> apk update
#> apk add ca-certificates curl

• Copy the mounted Consul server CA certificate into the /usr/local/share/ca-certificates/ and execute update-ca-certificates to add it to the system root CA store.

#> cp /consul/tls/ca/tls.crt /usr/local/share/ca-certificates/consul-server-ca.crt
#> update-ca-certificates  # might give a trivial warning - ignore it

• The Consul server is now accessible (and trusted) over HTTPS as below:

#> curl https://consul.service.consul:8501/v1/status/leader
## No TLS errors ##

• We also want to talk to the Consul client (instead of the server) over HTTPS, for performance reasons.
  • Since the Consul client has its own CA cert, we need to retrieve that from the server.
  • This requires the consul-k8s binary, so we need to get that.

#> cd /usr/local/bin
#> wget https://releases.hashicorp.com/consul-k8s/0.15.0/consul-k8s_0.15.0_linux_amd64.zip  # (or whatever latest version is)
#> unzip consul-k8s_0.15.0_linux_amd64.zip
#> rm consul-k8s_0.15.0_linux_amd64.zip

• Get the Consul client CA cert and install it via update-ca-certificates :

#> consul-k8s get-consul-client-ca -server-addr consul.service.consul -server-port 8501 -output-file /usr/local/share/ca-certificates/consul-client-ca.crt
#> update-ca-certificates  # might give a trivial warning - ignore it

• The Consul client is now accessible (and trusted) over HTTPS as below:

#> curl https://$HOST_IP:8501/v1/status/leader
## No TLS errors ##

• We can also access the Consul KV service from the client without issue:

#> curl https://$HOST_IP:8501/v1/kv/foo/bar/baz
## No TLS errors ##

Naturally, all of the above should be automated by the implementer. These manual steps are purely for demonstration purposes.