DevOps

Security Benefits of Egress Gateways in Kubernetes Clusters

How to use Istio egress to protect yourself

Security Benefits of Egress Gateways in Kubernetes Clusters
Photo by Scott Webb / Unsplash

Introduction

Kubernetes has become the de facto standard for orchestrating containerized applications, but with great power comes great responsibility—especially when it comes to security. One often-overlooked aspect of Kubernetes security is controlling the outbound traffic from a cluster, commonly known as egress traffic. While ingress controllers are widely used for managing incoming traffic, egress gateways are not as commonly implemented but offer several security advantages. In this article, we'll explore the security benefits of egress gateways in Kubernetes clusters and walk through a guide on how to implement them.

Why Egress Gateways?

Before we dive into the benefits, let's understand what an egress gateway is. An egress gateway is a dedicated point in a Kubernetes cluster through which all external service calls pass. Essentially, it's a way to manage and secure traffic leaving your cluster.

Security Benefits

  1. Traffic Control: Egress gateways allow you to control which external services your pods can access, ensuring that unauthorized or harmful destinations are blocked.
  2. Logging and Monitoring: Centralizing the exit point for all outbound traffic makes it easier to log and monitor these connections, which is useful for auditing and identifying suspicious activity.
  3. Compliance: Regulatory frameworks often require detailed control and logging of network traffic; egress gateways can make it easier to meet these requirements.
  4. Enhanced Firewall Rules: Since all outbound traffic goes through a known point, you can apply firewall rules more effectively.
  5. Data Leakage Prevention: By inspecting the data that's leaving your network, egress gateways can identify and block potentially sensitive information from being transmitted.
  6. Network Policy Simplification: Egress gateways can simplify the task of writing network policies, as you only have to manage the rules for the gateway rather than for each individual pod.

How to Implement Egress Gateways in Kubernetes

Istio is my service mesh of choice for Kubernetes. Below is a step-by-step guide to implementing an egress gateway in a Kubernetes cluster.

Prerequisites

  • A running Kubernetes cluster
  • kubectl installed and configured
  • Istio service mesh installed

Step 1: Enable Egress Gateway

Enable Istio's egress gateway by editing the IstioOperator custom resource.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true

Step 2: Create a Service Entry

When you're using an egress gateway, the ServiceEntry specifies which external services the cluster is allowed to access through the egress gateway. Without a ServiceEntry, the egress gateway wouldn't know which external domains or IPs it should allow traffic to, and you wouldn't be able to apply Istio policies to that outbound traffic.

Define a ServiceEntry to specify the external services that your cluster can access.

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: external-svc
spec:
  hosts:
  - example.com
  ports:
  - number: 80
    name: http
    protocol: HTTP

This ServiceEntry defines that the external service with the hostname example.com is allowed to be accessed over HTTP on port 80. Traffic destined for this host will be allowed to pass through the egress gateway, and Istio will also apply any other configurations or policies that you've set up for this host.

Step 3: Configure the Egress Gateway

The Gateway resource in Istio serves as a load balancer that handles incoming and outgoing HTTP/TCP connections. It configures exposed ports, the protocol to use, and other options like TLS settings. In the context of egress traffic in a Kubernetes cluster managed by Istio, the Gateway resource specifically configures the egress gateway, essentially defining how outbound traffic should be handled at the edge of the service mesh before it leaves the cluster.

The key roles of the Gateway resource:

  1. Port Configuration: Specifies which ports are open on the egress gateway and how they should handle traffic. This includes setting the protocol (HTTP, HTTPS, TCP, etc.).
  2. Traffic Routing: While the Gateway itself doesn't define the traffic routing rules, it serves as a reference for VirtualService resources that do. The VirtualService specifies how traffic that enters a gateway should be routed within the mesh.
  3. Security: You can configure TLS settings for secure traffic handling.
  4. Selector: Defines which workloads (usually pods) in the cluster will act as the gateway. This is typically based on labels.
# Egress Gateway
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP

Here, this Gateway resource does the following:

  • Selects the Istio egress gateway for configuration (istio: egressgateway label).
  • Opens port 80 and sets the protocol to HTTP.

The Gateway resource often works in conjunction with a VirtualService resource to define the complete traffic routing logic. The VirtualService binds itself to a gateway using the gateways field and defines how the traffic that enters the gateway should be routed.

# Virtual Service
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-external-svc-through-egress-gateway
spec:
  hosts:
  - example.com
  gateways:
  - istio-egressgateway
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        port:
          number: 80

In this example, the VirtualService is saying that traffic aiming to reach example.com that comes through the Istio egress gateway (istio-egressgateway) should be routed to the egress gateway service (istio-egressgateway.istio-system.svc.cluster.local) on port 80.

By combining the Gateway and VirtualService resources, you have fine-grained control over how egress traffic leaves your Kubernetes cluster, thereby enabling better security and routing capabilities.

Step 4: Apply Network Policies

The NetworkPolicy resource in Kubernetes is used to define how pods are allowed to communicate with various network endpoints, including other pods, services, and external hosts. Network policies play a crucial role in controlling the networking behavior in a Kubernetes cluster and are vital for implementing security best practices.

The key roles of the NetworkPolicy resource:

  1. Ingress Control: You can define which inbound connections to a pod are allowed.
  2. Egress Control: You can specify which outbound connections from a pod are allowed.
  3. Pod Selector: Uses labels to select which pods the network policy applies to.
  4. Policy Types: Specifies if the policy applies to ingress, egress, or both.
  5. IP Blocks: Allows you to specify CIDR ranges to whitelist or blacklist, giving you control over traffic based on IP ranges.

In the context of egress gateways in a Kubernetes cluster managed with Istio, a NetworkPolicy can be used to ensure that egress traffic from pods in the cluster only goes through the egress gateway. This is an additional security measure to make sure that no pod can bypass the egress gateway and communicate directly with external services, thereby circumventing security policies, logging, or auditing features that you've configured.

For example, consider the following NetworkPolicy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: egress-gateway-policy
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          istio: egressgateway

This NetworkPolicy does the following:

  • It applies to all pods in the namespace (podSelector: {} implies that the policy matches all pods).
  • It specifies that this is an Egress policy (policyTypes: - Egress).
  • It allows egress traffic only to pods that have the label istio: egressgateway.

By applying this NetworkPolicy, you effectively force all egress traffic to go through the Istio egress gateway, thereby gaining all the security and auditing benefits that come with it.

Step 5: Data Leakage Protection

While NetworkPolicy is focused on Layer 3 and Layer 4 network communication control, it doesn't offer native capabilities to filter traffic based on the content of the packets, such as sensitive data. Here is where an EnvoyFilter can be used to block sensitive data from being transmitted out of your Kubernetes cluster.

To block HTTP requests containing a payload with a "sensitiveData" key, you can use an Envoy filter with Lua scripting in Istio. The Lua script will check each HTTP request to see if it contains the "sensitiveData" key in the body. If so, it will reject the request and return a 400 Bad Request status code.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: block-sensitive-data
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: egressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.lua
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
          inlineCode: |
            function envoy_on_request(request_handle)
              local headers, body = request_handle:request_headers(), request_handle:body()
              if headers:get(":method") == "POST" then
                if body ~= nil then
                  local body_str = body:getBytes(0, body:length())
                  local is_sensitive = string.match(body_str, "\"sensitiveData\"")
                  if is_sensitive then
                    request_handle:respond(
                      {[":status"] = "400", ["content-type"] = "text/plain"},
                      "Bad Request: Posting sensitive data is not allowed."
                    )
                  end
                end
              end
            end

This example has a workloadSelector that applies to the Istio egress gateway, but you can modify the selector to target any specific workloads where you want to enforce this rule.

Here's what this EnvoyFilter does:

  • It adds a Lua script that gets executed on each incoming HTTP request (envoy_on_request).
  • The Lua script checks if the incoming request is a POST request.
  • If it's a POST request, the script then checks if the request body contains the key "sensitiveData".
  • If such a key is found, the request is immediately rejected with a 400 Bad Request response, along with a message stating that posting sensitive data is not allowed.

This example provides a basic illustration and might need additional refinements to suit your specific needs. It's essential to test thoroughly before deploying in a production environment.

How to Test Your Egress Gateway

Once you have your egress gateway set up, it's crucial to verify that it's functioning as expected. Below are several methods to test your egress gateway to ensure it's routing traffic correctly, blocking unauthorized requests, and logging activities as configured.

1. Verify Routing with curl

You can exec into a pod within the cluster and use curl to make requests to an external service that you've configured in your ServiceEntry.

kubectl exec -it [YOUR_POD_NAME] -c [YOUR_CONTAINER_NAME] -- /bin/bash
curl http://example.com

The request should succeed if the egress gateway is routing traffic correctly.

2. Check Istio Metrics

Istio exposes metrics that can be viewed via Grafana or another monitoring solution. Check the metrics related to the egress gateway to confirm that requests are being processed.

3. Examine Logs

You should have centralized logging configured for your egress gateway. Check the logs to confirm that they contain entries for the requests that have passed through the gateway. This is essential for auditing and monitoring.

kubectl logs -n istio-system $(kubectl get pod -n istio-system -l istio=egressgateway -o jsonpath='{.items[0].metadata.name}') -c istio-proxy

4. Test Unauthorized Requests

Try to access an external service that's not in your ServiceEntry. The request should be blocked, proving that your security policies are effective.

kubectl exec -it [YOUR_POD_NAME] -c [YOUR_CONTAINER_NAME] -- /bin/bash
curl http://unauthorized-service.com

5. Network Policy Test

You can also test the Kubernetes NetworkPolicy to ensure it's only allowing traffic through the egress gateway. Try to exec into a pod and make an external request without going through the gateway; it should fail to reach the external service.

kubectl exec -it [ANOTHER_POD_NAME] -c [ANOTHER_CONTAINER_NAME] -- /bin/bash
curl http://example.com

If the request is blocked, it confirms that your network policy is correctly enforcing the restriction.

6. Check Data Leakage Prevention

If you have configured data loss prevention measures, you can test by trying to send data that should be blocked by your rules. You can use curl to send a POST request with sensitive data to confirm that the egress gateway blocks it.

kubectl exec -it [YOUR_POD_NAME] -c [YOUR_CONTAINER_NAME] -- /bin/bash
curl -X POST -d "sensitiveData=1234" http://example.com/resource

By running these tests, you can ensure that your egress gateway is functioning as expected, thereby adding a robust security layer to your Kubernetes cluster.

You are set!

Egress gateways offer a myriad of security benefits ranging from fine-grained traffic control to compliance aid. Implementing them may add some complexity to your Kubernetes cluster, but the security advantages often outweigh the challenges. Testing your egress gateway ensures that it functions as expected, making it a key component in a mature Kubernetes security model.

Read next