Sometimes you need to monitor the network traffic that goes between pods to solve an issue. There are different ways to see it:
- Using Mizu
- Running tcpdump against a running container
- Using ksniff
- Using ephemeral debug containers
Of all the solutions, the cleaner and easier is to use Mizu.
If the pod you want to analyze has root permissions (bad idea) you'll be able to install
apt-get install tcpdump) and pipe it into
wireshark on your local machine.
kubectl exec my-app-pod -- tcpdump -i eth0 -w - | wireshark -k -i -
There's some issues with this, though:
- You have to
kubectl execand install arbitrary software from the internet on a running Pod. This is fine for internet-connected dev environments, but probably not something you'd want to do (or be able to do) in production.
- If this app had been using a minimal
distrolessbase image or was built with a
buildpackyou won't be able to install
Kubernetes 1.16 has a new Ephemeral Containers feature that is perfect for our use case. With Ephemeral Containers, we can ask for a new temporary container with the image of our choosing to run inside an existing Pod. This means we can keep the main images for our applications lightweight and then bolt on a heavy image with all of our favorite debug tools when necessary.
kubectl debug -it pod-to-debug-id --image=nicolaka/netshoot --target=pod-to-debug -- tcpdump -i eth0 -w - | wireshark -k -i
nicolaka/netshoot is an optimized network troubleshooting docker.
There's some issues with this too, for example, as of Kubernetes 1.21 Ephemeral containers are not enabled by default, so chances are you won't have access to them yet in your environment.