How to use AWS Application Load Balancer with Istio Gateway

If you have been using Kubernetes in AWS (EKS), you may have noticed that when you create a Kubernetes service of type LoadBalancer like this:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: MyApp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: LoadBalancer

AWS provisions a Classic Load Balancer to route traffic to this service from outside the cluster. You may want to Network Load Balancer instead. It's just a matter of adding a annotation:

apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  selector:
    app: MyApp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: LoadBalancer

Istio Ingressgateway

Istio offers two ways of traffic ingress from outside of cluster:

1. Ingress Gateway: Part of the full-featured Istio installation and their recommended way.
2. Kubernetes Ingress: The built-in Ingress feature in Kubernetes. One has to setup the Ingress controller separately.

When you install the istio-ingressgateway with Istio in your cluster, it also creates a LoadBalancer Kubernetes service that brings external traffic to your mesh. This is the default behaviour. Again if you want to set NLB as your layer 4 load balancer the you can modify the Istio operator as follows:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istiocontrolplane
spec:
  profile: demo
  hub: gcr.io/istio-release
  values:
    gateways:
      istio-ingressgateway:
        serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

You can more detailed steps of achieving this on the Istio blog.

Why Would I Need AWS Application Load Balancer?

While the default istio-ingressgateway works perfectly, especially when you use NLB, there are certain features you would like to have in the long term:

1. Ability to route to non-Kubernetes targets.
2. Associate WAF and Shield with your load balancer for security.
3. Ingress of gRPC traffic along with load balancing (NLB is not very good for this use-case)

There maybe more reasons for ALB and some against it, but I personally needed these features and was sold.

Let's Dive In

We have already seen that only layer 4 load balancers work with Kubernetes service that istio-ingressgateway creates and ALB can be used only with Kubernetes Ingress. Here we take a hybrid approach.

1. We create a Kubernetes Ingress utilising an ALB.
2. We change the istio-ingressgateway service type to NodePort and send traffic from the Ingress in step 1 to this NodePort service.

Before going to the first step, we need to install the Ingress Controller for ALB. Follow these steps religiously to install the controller. This is to ensure that no IAM policy is messed up otherwise the controller throws an AccessDenied error.

We will now have to change our Istio operator to change the ingressgateway from LoadBalancer to NodePort service.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istiocontrolplane
spec:
  profile: demo
  hub: gcr.io/istio-release
  values:
    gateways:
      istio-ingressgateway:
        type: NodePort
        serviceAnnotations:
          alb.ingress.kubernetes.io/healthcheck-port: '30621'
          alb.ingress.kubernetes.io/healthcheck-path: /healthz/ready
        ports:
        - name: status-port
          protocol: TCP
          port: 15021
          targetPort: 15021
          nodePort: 30621
        - name: http2
          protocol: TCP
          port: 80
          targetPort: 8080
          nodePort: 31565
        - name: https
          protocol: TCP
          port: 443
          targetPort: 8443
          nodePort: 31804

Important things to note here are the health check annotations and corresponding port under ports list named status-port. The nodePort of status-port and healthcheck-port needs to be same for the load balancer target group health check to be successful.

Let's create an Ingress resource to route to this service now:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: common-external-ingress
  namespace: istio-system
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '443'
    alb.ingress.kubernetes.io/load-balancer-name: common-external-ingress
    alb.ingress.kubernetes.io/target-type: instance
    alb.ingress.kubernetes.io/subnets: subnet-031c06XXXXXXXXXXX, subnet-03444dXXXXXXXXXXX, subnet-0360f0XXXXXXXXXXX
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-west-2:123456789011:certificate/26cec536-5d01-4140-890d-XXXXXXXXXXXX
    alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:us-west-2:123456789011:regional/webacl/prod-common-alb-acl/ae2e63f9-02f4-44b8-9d25-XXXXXXXXXXXX
  labels:
    app: common-external-ingress
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: istio-ingressgateway
                port:
                  number: 443

This is a complete example with SSL termination, HTTPS redirection and WAF. You may not use all the annotations or modify as required.

If you look at the AWS elastic load balancing dashboard, you will see a new ALB being provisioning/active. When it is in the active state, you can add a CNAME or A (Alias) record in your DNS settings with the ALB's DNS name and you're ready to go.

Parting Words

At the end we are able to leverage the robust load balancing and security infra that AWS provides without loosing the networking features of Istio mesh like traffic routing, fault injection, circuit breaking etc.

Tell me if you would like similar short solutions to such (un)common problems.