IT & Test Environment Management Anti Patterns #2

Anti-patterns often arise when developers find creative or outside-the-box uses for a technology – or when they find workarounds for limitations that were specifically put in place by the creators of the technology to prevent misuse.Application container platforms like Docker have been the hot new thing for several years now, but they’re still fairly new concepts to many IT environment managers. And since they’ve been around for a while, there are a number of anti-patterns that environment managers have fallen into.In this post, we follow on from our previous article on Test Environment Anti Patterns and talk about some of the common anti-patterns of containers and consider why they exist.

1. Confusing Application Containers with System Containers or VMs

Some developers make the mistake of comparing application containers with small VMs. This may seem inconsequential, but it actually lays a dangerous frame of reference.

Anti-Pattern

The comparison between containers and VMs leads to developers running multiple processes and services, or even operating systems within application containers. However, containers don’t work like VMs. They should run one process or service.System containers like LXD or OpenVZ explicitly allow developers to install operating systems inside the container. They operate much like virtual machines. However, VMs emulate hardware while system containers do not. What does this mean for us? If you’re looking to run Windows on your Linux, you’ll want a virtual machine. If you want to run multiple types of Linux-based operating systems on your server instead, then you can use system containers.Why is this an anti-pattern? Application containers isolate processes; they don’t virtualize operating systems. They encapsulate parts of your application ecosystem into standalone deployable parts. If I have a Docker container that includes my web services, database, and message bus, I’m unable to scale or deploy different components of my stack.And that results in blowing away one of the main features of containerization.

Pattern

An application container should have only one service or process running in it. And that’s it.

2. Creating Tightly Coupled Dependencies

So now you’re ready to use application containers to deploy and run your application. You’ve properly segregated your services and functions into separate containers. What anti-pattern might you fall into next?

Anti-Pattern

Coupling your containers to each other may seem innocent at first. It’s as simple as adding a depends-on or waits-for-it to your Docker configuration. However, this can result in long deployments, circular dependency, and reduced availability.

Pattern

To get the most benefits from containerization, your containers should be independently runnable. This lets you destroy and create containers whenever necessary and lets you scale horizontally.But what does independently runnable mean? Does it mean that my one process can’t be dependent on anything else? That would be fairly difficult to create meaningful apps this way. It means that your container and the process inside should not be dependent on anything else when starting up. Your container should never wait for other containers or dependencies to be up or accessible upon startup.And again, this doesn’t mean that your application can’t depend on anything else. In fact, it should still throw errors if someone is using the container’s app and a critical dependency is down. This can be achieved in different ways. For one, don’t try to connect to another dependency on startup. Alternatively, consider using message buses to keep containers independently of each other.

3. Modifying Running Containers

Our next anti-pattern scolds those of us that like changing things on the fly.

Anti-Pattern

Now that you have an independently running process in each of your containers, you may be tempted to update application configuration or even the application code inside.

Pattern

However, a much better practice involves deploying a brand new version of the container. You should consider the container and the process running inside to be immutable. If you need to change a configuration, then change it in your Docker file or Helm chart and deploy the new version. If you want to change the application running, then deploy a new container with the new version of your code.

4. Including Unnecessary Services

Even if you follow the good patterns above, you may be tempted to add just a small service here or there to your containers.

Anti-Pattern

Developers strive to make life easier for themselves. That includes adding tools and small services that are unrelated to the core functionality of a container. For example, a common anti-pattern here involves adding an SSH daemon to your container image. Not only does it encourage developers to change or tweak containers once they’re running, but they create another way for someone to try to gain access to your container. And if you’re not following the anti-patterns above, there should be no need for having an SSH daemon on your container anyway.The more services you include in your container, the more bloated it becomes. Also, those extra services provide additional attack vectors for hackers.

Pattern

Keep your containers light and simple. Don’t add anything that isn’t necessary to what functionality your service provides.

5. Persisting Data Inside a Container or Image Registry

It may seem easy and convenient to store data inside our containers. But once again, that’s a big no-no.

Anti-Pattern

As a reminder, containers are ephemeral. They spin up and come down fairly frequently. They should never be considered as stable or long-lasting. If you rely on data or logs that are available only in the container, then you’re going to have a bad time.Data doesn’t persist in containers by default. Additionally, you’re introducing another attack vector or security flaw. Anyone that gains access to the container or image registry may also gain access to your data.

Pattern

Data shouldn’t be stored in containers. Instead, use Docker volumes or an external dependency to store your data and logs.

6. Relying on Backups and Restores of Containers

Our next anti-pattern might seem harmless at first. After all, we’ve all been trained to make sure everything is always backed up in case of emergencies. What’s wrong with that?

Anti-Pattern

In and of themselves, backups are useful. However, some developers use them as a shortcut to deployments. They take backups of existing containers and restore them on new servers—circumventing their deployment pipeline.

Pattern

There are some legitimate reasons why you may want to make a backup of your container. For example, in the event of a security breach in a container, you may want to back it up before destroying it for forensic analysis. However, restoring backups should not be part of your normal CI/CD or disaster recovery process. Instead, build and deploy a new version of your container and related services.

7. Giving Little Thought to Backward Compatibility

Whether your containers are directly dependent on other containers or dependencies like Kafka or RabbitMQ, you should always consider how changes will affect other systems.

Anti-Patterns

The anti-pattern here involves relying on deployment order to coordinate upgrades and changes across containers and dependencies. One big example many of us are guilty of involves forgetting backward compatibility for database changes. If your web application relies on a database, it may be tempting to make changes in both and then coordinate deployments so that both are upgraded at the same time. Another step may involve a maintenance window. Here, services are taken down or made inaccessible. This goes against the high-availability promise of containerization.

Patterns

When making changes to parts of your system, consider ways in which upgrading one or the other independently will work just fine. This may involve extra future deployments to remove unused code or columns, but it ensures more availability for your services. Give some thought to ways of enhancing and modifying systems so that other older versions of dependencies can still work—even if only temporarily.

8. Assuming Containers Will Fix Everything

I’d like to close out this list with an anti-pattern that’s less technical in nature. As lovers of technology, we’re often looking for the next great thing that will solve the limitations of our current platforms and tools. And this forward-looking experimentation is one of our strengths.However, we also need to be realistic about the limitations of the tools we use, like application containers. They were built to solve a specific set of problems. And if we try to circumvent that by creating workarounds, we’ll end up creating tomorrow’s anti-pattern. 

Author: Sylvia Fronczak.  This post was written by Sylvia Fronczak. Sylvia is a software developer that has worked in various industries with various software methodologies. She’s currently focused on design practices that the whole team can own, understand, and evolve over time.