mKubernetes, Ingress controllers and Traefik

When running your application services on top of an orchestration tool like Kubernetes or Mesos with Marathon, there are some common necessities you’ll need to satisfy. Your application will usually contain two types of services, those that should be visible only from inside of the cluster, and those that you want to expose to the external world, outside your cluster and maybe to the (e.g frontends). internet This article will focus on how to approach this on Kubernetes. You can make use of the different services types that Kubernetes makes available for you when creating a new in order to achieve what you want. service ClusterIP: This is the default. Choosing this value means that you want this service to be reachable only from inside of the cluster. ExternalName: It serves as a way to return an alias to an external service residing outside the cluster. NodePort: Expose the service on a port on each node of the cluster. LoadBalancer: On top of having a cluster-internal IP and exposing service on a NodePort, also ask the cloud provider for a load balancer which forwards requests to the Service exposed as a :NodePort for each Node. If the cloud provider does not support the feature, the field will be ignored. So, if your cloud does not support “loadBalancer” (e.g you run an on-premise private cloud), and you need something more sophisticated than exposing a port on every node of the cluster, then it used to be that you’d need to build your own custom solution. Fortunately this is not true anymore. Since you can use which includes support for TLS and L7 http-based traffic routing. Kubernetes v1.2.0 Kubernetes ingress You can also ask Kuwit “How can I expose services to the external world?” whenever you need to remember this ;-) An Ingress is a collection of rules that allow inbound connections to reach the cluster services. It can be configured to give services externally-reachable urls, load balance traffic, terminate SSL, offer name based virtual hosting etc. Users request ingress by POSTing the Ingress resource to the API server. In order for the Ingress resource to work, the cluster must have an Ingress controller running. The Ingress controller is responsible for fulfilling the Ingress dynamically by watching the ApiServer’s /ingresses endpoint. This is handy! Now you could go even further, isolate at the infra level where your ingress controller runs and think of it as an “edge router” that enforces the firewall policy for your cluster. The picture for a HA Kubernetes Cluster would look something like this: We’ll show how to use for this purpose. Traefik is a modern HTTP reverse proxy and load balancer made to deploy microservices with ease. It supports several backends among Mesos/Marathon and Kubernetes to manage its configuration automatically and dynamically. Traefik We’ll deploy a Kubernetes cluster similar to the picture above and will run Traefik as . DaemonSet apiVersion: extensions/v1beta1kind: DaemonSetmetadata: name: traefik-ingress-controller-v1 namespace: kube-system labels: k8s-app: traefik-ingress-lb kubernetes.io/cluster-service: "true"spec: template: metadata: labels: k8s-app: traefik-ingress-lb name: traefik-ingress-lb spec: terminationGracePeriodSeconds: 60 containers: - image: containous/traefik name: traefik-ingress-lb imagePullPolicy: Always ports: - containerPort: 80 hostPort: 80 - containerPort: 443 hostPort: 443 - containerPort: 8080 hostPort: 8080 volumeMounts: - mountPath: /etc/traefik name: traefik-volume readOnly: false args: - --web - --kubernetes - --configFile=/etc/traefik/traefik.toml - --logLevel=DEBUG volumes: - hostPath: path: /etc/traefik name: traefik-volume nodeSelector: role: edge-router The source code is here . You can configure Traefik to use automatic TLS config for your services on demand Our edge-router will be just another with some restrictions. Kubernetes node We don’t want any other pod to be scheduled to this node so we set when running the as well as giving it a convenient label: . **--register-schedulable=false** kubelet **--node-labels=edge-router** Kubernetes will run DaemonSets on every node of the cluster even if they are non-schedulable. We only want this DaemonSet to run on the edge-router node so we use “nodeSelector” to match the label we previously added. nodeSelector: role: edge-router Notice that with this approach, if you want to add a new edge-router to the cluster, all you need to do is spin up a new node with that label and a new DaemonSet will be automatically scheduled to that machine. Nice! Here is a video demo of all this in action using two different clouds (DigitalOcean and AWS), deploying two Kubernetes clusters from scratch: Recently I’ve seen a lot of users on with issues communicating to the ingress controller. This is often due to a known problem. Kubernetes slack The ingress controller might want to use to expose itself. hostPort ports: - containerPort: 80 hostPort: 80 - containerPort: 443 hostPort: 443 If you are using a for your cluster networking, hostPort is not supported yet. CNI Network Plugin You can track the current status of this problem here: _I am not sure if what I am doing is supposed to work. But I have created the following pod: apiVersion: v1 kind: Pod…_github.com HostPort seemingly not working · Issue #23920 · kubernetes/kubernetes https://github.com/kubernetes/kubernetes/issues/31307 https://github.com/containernetworking/cni/issues/46 For now potential workarounds for this are to use “hostNetwork” or run a service using the previously mentioned “nodePort” to match your ingress controller running as a DaemonSet. Hopefully this post will give you some insight about how to expose services on Kubernetes and the benefits of Ingress controllers. This post has also been published on Capgemini Engineering Blog