5 Kubernetes worst practices that are causing you pain (or will soon)

In our years of experience with Kubernetes internally and with hundreds of customers, we’ve seen several mistakes happen over and over again. 

Don’t beat yourself up about it: we’ve all been there! 

Here are the 5 K8s worst practices that will cause you pain down the line.

1. Mixing competing placement constructs (NodeAffinity and NodeSelectors)

Node selectors were part of K8s for a long time. They’re simple to read but limited.
Node affinity came later. It’s a bit harder to read but more flexible – and so, more powerful.

The two mechanisms achieve the same goal: they attract pods to nodes with labels. Yup, two tools for the same job.

Teams usually agree to use one or the other and stick with it. Mixing them opens the door to configuration errors that eventually cause workloads to fail to find suitable nodes.

2. Not having observability in place

In K8s, observability gives you the ability to troubleshoot issues quickly thanks to telemetry signals: logs, metrics, and traces. 

If you don’t have complete visibility into your cluster, you’re in trouble. Identifying the root cause of issues will take hours, if not days.

Not capturing stdout logs is just criminal, so make sure to write your logs to the standard output (stdout) and standard error (stderr) streams. And collect these logs at a central location so that you don’t waste time chasing terminated pods and disappearing nodes.

3. Having excessive cross-AZ traffic

Spreading an application across multiple availability zones (AZs) helps to increase availability and performance, achieve compliance, and optimize costs by choosing the most cost-effective compute within the region.

While data transfer within the same AZ doesn’t cost you anything, if you move it in or out across AZs in the same Region, you’ll be charged $0.01/GB in each direction.

Excessive cross-AZ traffic can bump up your cloud bill in ways you don’t expect. 

So, do your best to optimize cross-communications network traffic for an architecture that meets your requirements without breaking the bank. We recently identified high inter-zone traffic between two services and, after moving these services to the same zone, reduced data transfer costs by 70%. 

A good way to reduce inter-zone traffic is to use Kubernetes Topology Aware Routing (via Service annotations). This will try to route traffic between pods in the same AZ, delivering not only cost savings but also network latency improvements.

4. Not using namespaces or keeping everything in “default” namespace

When running a relatively small cluster, you might survive without a structure in place. 

But as your cluster scales to dozens of pods and hundreds of containers, you can’t afford this chaos anymore. Not using namespaces or keeping everything in a “default” namespace are both really bad ideas.

By keeping your cluster well-organized using namespaces, labels, and annotations, you’ll steer clear of all the potential performance, maintenance, and security issues that come from a lack of control over the deployed objects and services.

5. Not adding wildcardable tolerations to vital DaemonSets

Essential DaemonSets providing basic IO for pods like CNI and CSI should run on all nodes. No exceptions. 

Similarly, observability components are generally needed on all nodes, like the logging forwarder and APM.

Quite often, if you need to create an isolated node group – for example, VMs with special hardware like GPUs or short-lived special needs workloads like Spark jobs – these nodes will most likely get tainted. 

Only workloads that should be scheduled on these nodes will be scheduled there – and nothing else. You don’t want random pods to exhaust your precious resources. 

When this happens, some essential DaemonSets may not be able to start on these nodes due to taints (lack of toleration) and you’ll end up with a blind spot in logging coverage.

Having these two lines on all essential DaemonSets will ensure you have perfect coverage:

Want to look further than these worst offenders?

We looked into a grand total of 16(!) K8s things you shouldn’t do:

• Scaling workload replicas manually
• Using CPU as the metric for scaling workload replicas
• Running a self-hosted Kubernetes control plane

Digging into each of these doesn’t fit into an email, so we will cover the worst of the second batch in the next edition of Kubernetes IRL.

I hope this makes your Kubernetes experience smoother!

Cheers,

Allen

Found this email useful? Forward it to your friends and colleagues who need more Kubernetes best practices in their lives.