Envoy MTLS Configuration For Reverse Tunnels: A Comprehensive Guide

by Alex Johnson 68 views

Securing your services with mutual TLS (mTLS) is crucial in modern, distributed systems. When using Envoy as your proxy, configuring mTLS for reverse tunnels can be a bit tricky. This article dives deep into how to properly set up mTLS for reverse tunnels in Envoy, addressing common challenges and providing a step-by-step guide.

Understanding Reverse Tunnels and mTLS

Let's begin by clarifying what reverse tunnels and mTLS are, and why they are essential for secure communication.

What are Reverse Tunnels?

Reverse tunnels, in the context of Envoy, allow a downstream service to initiate a connection to an upstream service, even if the upstream service is behind a firewall or in a different network. This is particularly useful in scenarios where the downstream service cannot directly reach the upstream service. This can greatly enhance the security posture and network design of your microservices architecture.

What is mTLS?

Mutual TLS (mTLS) is a security mechanism where both the client and the server authenticate each other by exchanging digital certificates. This ensures that both parties are who they claim to be, preventing man-in-the-middle attacks and unauthorized access. Imagine it as a double handshake, where both sides present their IDs before engaging in conversation. Securing your services with mTLS adds a robust layer of authentication, especially crucial in zero-trust environments. Properly implementing mTLS helps in achieving compliance with security standards and regulations, reducing the risk of data breaches and ensuring data integrity.

The Challenge: Configuring mTLS for Reverse Tunnels in Envoy

Configuring TLS for upstream clusters in Envoy is well-documented, but applying this to reverse tunnels can be confusing. The primary challenge lies in ensuring that both the initiator (downstream) and the responder (upstream) Envoy instances are correctly configured to handle TLS and mTLS handshakes. Specifically, the responder needs to be configured to accept TLS connections initiated over the reverse tunnel, and both sides must trust each other's certificates.

In essence, mTLS ensures that the communication channel is not only encrypted but also that both ends of the tunnel are verified and trusted. This is achieved by each side presenting a certificate that the other side validates against a trusted Certificate Authority (CA). Without the correct configuration, the connection can fail with errors like WRONG_VERSION_NUMBER or PROTOCOL_IS_SHUTDOWN, as seen in the provided logs. Therefore, meticulous configuration of the TLS context on both the initiator and responder is critical for the successful establishment of secure reverse tunnels.

Step-by-Step Guide: Setting up mTLS for Envoy Reverse Tunnels

To effectively configure mTLS for reverse tunnels in Envoy, follow these steps meticulously. This guide will walk you through setting up the initiator and responder Envoy configurations, ensuring secure and authenticated communication.

1. Generate Certificates

First, you need to generate the necessary certificates for both the initiator and responder. This includes a Certificate Authority (CA) certificate, as well as certificates and private keys for each Envoy instance. Here’s a basic outline of the certificates you’ll need:

  • CA Certificate (ca-cert.pem): Used to sign the certificates for both the initiator and responder.
  • Initiator Certificate and Key (downstream-cert.pem, downstream-key.pem): Used by the initiator to identify itself to the responder.
  • Responder Certificate and Key (upstream-cert.pem, upstream-key.pem): Used by the responder to identify itself to the initiator.

Using a tool like OpenSSL, you can generate these certificates. For example:

openssl genrsa -out ca-key.pem 2048
openssl req -x509 -new -nodes -key ca-key.pem -subj "/CN=Example CA" -days 3650 -out ca-cert.pem

openssl genrsa -out upstream-key.pem 2048
openssl req -new -key upstream-key.pem -subj "/CN=upstream-envoy.local" -out upstream.csr
openssl x509 -req -in upstream.csr -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -days 365 -extfile <(printf "subjectAltName=DNS:upstream-envoy.local") -out upstream-cert.pem

openssl genrsa -out downstream-key.pem 2048
openssl req -new -key downstream-key.pem -subj "/CN=downstream-envoy.local" -out downstream.csr
openssl x509 -req -in downstream.csr -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -days 365 -extfile <(printf "subjectAltName=DNS:downstream-envoy.local") -out downstream-cert.pem

2. Configure the Initiator (Downstream) Envoy

The initiator Envoy instance initiates the reverse tunnel connection. Its configuration needs to include the TLS settings for the upstream cluster.

Here's an example initiator-envoy.yml configuration:

static_resources:
  clusters:
  - name: upstream-cluster
    type: STRICT_DNS
    connect_timeout: 30s
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        common_tls_context:
          tls_certificates:
          - certificate_chain:
              filename: "certs/downstream-cert.pem"
            private_key:
              filename: "certs/downstream-key.pem"
          validation_context:
            trusted_ca:
              filename: "certs/ca-cert.pem"
          alpn_protocols: ["h2", "http/1.1"]
        sni: upstream-envoy.local
    load_assignment:
      cluster_name: upstream-cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 192.168.122.1  # Address of responder-envoy service
                port_value: 9000  # Port for rev_conn_api_listener

Key points in the initiator configuration:

  • transport_socket: Specifies the TLS transport socket.
  • UpstreamTlsContext: Configures TLS settings for upstream connections.
  • tls_certificates: Includes the initiator’s certificate and private key (downstream-cert.pem, downstream-key.pem).
  • validation_context: Specifies the trusted CA certificate (ca-cert.pem) to validate the responder’s certificate.
  • sni: Sets the Server Name Indication (SNI) to upstream-envoy.local.

3. Configure the Responder (Upstream) Envoy

The responder Envoy instance listens for reverse tunnel connections and handles incoming requests. The responder configuration must include TLS settings for the downstream connection and the reverse tunnel filter.

Here's an example responder-envoy.yml configuration:

static_resources:
  listeners:
  # Accepts reverse tunnel requests
  - name: rev_conn_api_listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 9000
    filter_chains:
    - filters:
      - name: envoy.filters.network.reverse_tunnel
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.reverse_tunnel.v3.ReverseTunnel
          ping_interval: 2s
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          require_client_certificate: true
          common_tls_context:
            tls_certificates:
            - certificate_chain: {filename: "certs/upstream-cert.pem"}
              private_key: {filename: "certs/upstream-key.pem"}
            validation_context:
              trusted_ca:
                filename: certs/ca-cert.pem
            alpn_protocols: ["h2", "http/1.1"]

Key points in the responder configuration:

  • listeners: Defines the listener for reverse tunnel requests.
  • filter_chains: Specifies the filters to be applied to the incoming connections.
  • envoy.filters.network.reverse_tunnel: The reverse tunnel filter.
  • DownstreamTlsContext: Configures TLS settings for downstream connections.
  • require_client_certificate: true: This crucial setting enforces mTLS by requiring clients to present a certificate.
  • tls_certificates: Includes the responder’s certificate and private key (upstream-cert.pem, upstream-key.pem).
  • validation_context: Specifies the trusted CA certificate (ca-cert.pem) to validate the initiator’s certificate.
  • alpn_protocols: Includes supported ALPN protocols.

4. Verify Certificate Paths

Ensure that the file paths to your certificates in both the initiator and responder configurations are correct. Incorrect paths are a common cause of TLS setup failures. Double-check that the files exist at the specified locations and that Envoy has the necessary permissions to access them. This step is crucial for avoiding configuration errors and ensuring that the TLS handshake process can proceed without issues.

5. Test the Configuration

After configuring both Envoy instances, it’s time to test the setup. Start both Envoy instances and attempt to establish a connection through the reverse tunnel. Monitor the logs on both sides for any errors.

For example, you can use curl with the --cacert option to specify the CA certificate and verify the connection:

curl --cacert certs/ca-cert.pem https://upstream-envoy.local:9000

If the configuration is correct, you should see a successful connection. If there are issues, the logs will provide valuable information for troubleshooting. Check for TLS handshake failures, certificate validation errors, or protocol mismatches.

Troubleshooting Common Issues

Even with careful configuration, issues can arise. Here are some common problems and how to troubleshoot them:

1. WRONG_VERSION_NUMBER Error

This error typically indicates that the client is sending unencrypted traffic to a port expecting TLS. Ensure that the initiator is configured to use TLS and that the responder is listening for TLS connections.

  • Check Initiator Configuration: Verify that the transport_socket and UpstreamTlsContext are correctly configured in the initiator’s cluster settings.
  • Check Responder Configuration: Ensure that the DownstreamTlsContext is properly set up in the responder’s listener configuration.
  • Verify Port Configuration: Confirm that the port used for the reverse tunnel listener on the responder is configured to handle TLS traffic.

2. Certificate Validation Errors

If you encounter certificate validation errors, it means that the certificates are not being trusted. This could be due to an incorrect CA certificate or issues with the certificate chain.

  • Verify CA Certificate: Ensure that the correct ca-cert.pem is used in both the initiator and responder configurations.
  • Check Certificate Paths: Double-check the file paths to the certificates in the configurations.
  • Inspect Certificate Contents: Use OpenSSL to inspect the certificates and ensure that they are valid and correctly signed by the CA.
openssl x509 -in certs/downstream-cert.pem -text -noout

3. Protocol Mismatches

Mismatched ALPN protocols can also cause connection issues. Ensure that the ALPN protocols specified in the initiator and responder configurations are compatible.

  • Check ALPN Protocols: Verify that the alpn_protocols list in both the UpstreamTlsContext and DownstreamTlsContext includes compatible protocols (e.g., h2, http/1.1).
  • Ensure Protocol Support: Confirm that both Envoy instances and any intermediate proxies support the specified ALPN protocols.

4. mTLS Not Enforced

If mTLS is not being enforced, clients may be able to connect without presenting a certificate. This defeats the purpose of mTLS.

  • Verify require_client_certificate: Ensure that the require_client_certificate: true option is set in the responder’s DownstreamTlsContext.
  • Check Logs: Monitor the Envoy logs for messages indicating whether client certificates are being requested and validated.

Example Scenario: Securing Microservice Communication

Consider a scenario where you have two microservices, Service A (the initiator) and Service B (the responder), communicating through an Envoy reverse tunnel. Service B is behind a firewall and cannot be directly accessed by Service A. To secure this communication, you can configure mTLS on the reverse tunnel.

  1. Certificate Generation: Generate certificates for both services, ensuring each has its own certificate and key, signed by a common CA.
  2. Initiator Configuration (Service A): Configure Envoy in front of Service A to initiate the reverse tunnel and present its certificate. The configuration will include the UpstreamTlsContext with the certificate, key, and trusted CA.
  3. Responder Configuration (Service B): Configure Envoy in front of Service B to listen for reverse tunnel connections, require client certificates, and validate them against the trusted CA. The configuration will include the DownstreamTlsContext with require_client_certificate: true.
  4. Deployment: Deploy the configured Envoy instances alongside their respective services.
  5. Testing: Verify that Service A can communicate with Service B through the secured reverse tunnel, and that connections without a valid certificate are rejected.

By implementing mTLS in this scenario, you ensure that only authenticated services can communicate, significantly enhancing the security posture of your microservices architecture. This setup not only encrypts the traffic but also verifies the identity of both communicating parties, preventing unauthorized access and man-in-the-middle attacks.

Conclusion

Configuring mTLS for reverse tunnels in Envoy requires careful attention to detail, but the added security is well worth the effort. By following this guide and understanding the common pitfalls, you can establish secure and authenticated communication channels for your services. Ensuring that your configurations are correct, certificates are valid, and TLS settings are properly aligned will lead to a robust and secure reverse tunnel setup. This not only enhances your security posture but also ensures compliance with various security standards and regulations.

For more information on Envoy and TLS configuration, check out the official Envoy documentation on envoyproxy.io. This comprehensive resource provides detailed explanations and examples to help you master Envoy's powerful features and secure your applications effectively.