AWS EKS supports native VPC networking with the Amazon VPC Container Network Interface (CNI) plugin for Kubernetes. Using this plugin allows Kubernetes Pods to have the same IP address inside the pod as they do on the VPC network.
This is a great feature but it introduces a limitation in the number of Pods per EC2 Node instance. Whenever you deploy a Pod in the EKS worker Node, EKS creates a new IP address from VPC subnet and attach to the instance.
The formula for defining the maximum number of pods per instance is as follows:
N * (M-1) + 2
Nis the number of Elastic Network Interfaces (ENI) of the instance type.
Mis the number of IP addresses of a single ENI.
t3.small, this calculation is
3 * (4-1) + 2 = 11. For a list of all the instance types and their limits see this document
New Kubernetes versions introduce significant changes, so it's recommended that you test the behavior of your applications against a new Kubernetes version before performing the update on your production clusters.
The update process consists of Amazon EKS launching new API server nodes with the updated Kubernetes version to replace the existing ones. Amazon EKS performs standard infrastructure and readiness health checks for network traffic on these new nodes to verify that they are working as expected. If any of these checks fail, Amazon EKS reverts the infrastructure deployment, and your cluster remains on the prior Kubernetes version. Running applications are not affected, and your cluster is never left in a non-deterministic or unrecoverable state. Amazon EKS regularly backs up all managed clusters, and mechanisms exist to recover clusters if necessary. We are constantly evaluating and improving our Kubernetes infrastructure management processes.
To upgrade a cluster follow these steps:
- Upgrade all your charts to the latest version with helmfile.
helmfile deps helmfile apply
- Check your current version and compare it with the one you want to upgrade.
kubectl version --short kubectl get nodes
- Check the docs to see if the version you want to upgrade requires some special steps.
- If your worker nodes aren't at the same version as the cluster control plane upgrade them to the control plane version (never higher).
cluster_versionattribute of the eks terraform module and apply the changes (reviewing them first).
This is a long step (approximately 40 minutes) * Upgrade your charts again.