Let's Code Some Infrastructure with Terraform

Intro

Perhaps for some of you, IT infrastructure management sounds easy, but it can be excruciating and full of disappointments. That is why, I wanted to share my thoughts and experiences which cover this subject. I finally decided to divide the article into two sections. In the beginning, I’m going to discuss a little bit the background of the entire DevOps tools ecosystem and explain what the Infrastructure-as-Code is. In the second part of the article, we will get our hands dirty with code by implementing a real-life example of using the IaC with Terraform.

Modern DevOps tools ecosystem

Let’s look at the areas listed below. Nowadays, each of the following is fundamental and need to be managed properly to have a deep and broad insight into the big picture of the DevOps world and to always keep calm on the production.

• Configuration management We want to manage environments and application configuration in a centralized way. In the era of microservices, it is not possible to keep in check so many components. Especially when we want to do some runtime tuning and extra customization or we want to scale up some application resources in a maintenance-free manner. Tools like Ansible, Chef, or Puppet are designed for such operations.
• Containers and Container orchestrators It doesn’t matter which container engine you decide to use - seriously. However, you should be familiar with at least one container technology. It’s worth mentioning that if the container engine is compliant with Open Containers Initiative (OCI), you may feel more comfortable - it guarantees a higher compatibility level with the orchestration systems like Kubernetes or Nomad. Using orchestration is always kind of trade-off when we consider small environments, but on the other hand, it can significantly speed up scaling the environments in the future. Tools like Kubernetes, Nomad should be considered to implement container orchestration.
• Continuous Integration and Continuous Delivery Currently, it’s hard to imagine software development without these themes. They are already a part of the project DNA. I’m not going to go more in-depth at the moment, but I want to focus your attention on these pipelines. It doesn’t even matter which pipeline engine you choose, Jenkins and the Jenkinsfile, Travis CI and the .travis.yaml file or any other. Such tools are still actively developed (e.g. Jenkins X project: https://jenkins-x.io) and it’s good to keep an eye on it.
• Infrastructure as Code Ok, so this is something I wanted to go deeper in my article. I choose Terraform as an implementation of the IoC approach due to the maturity of the project and a wide range of potential plugins for cloud providers.

What is Infrastructure-as-code?

In few words, this is a common approach for provisioning the cloud infrastructure based on the coded definitions. Everything that could be done via the cloud web console can be done via the IaC defined snippets - from small services (e.g., AWS Lambda, Azure function) through DNS services, firewall rules to the large database installations. Let’s take a closer look at the main IaC concepts:

• Versioning, static analysis, review My favorite advantage of following IaC concept is version control. It’s a fundamental and pretty straightforward concept. Complete infrastructure design can be after all committed to the repo as a bucket of source code files. You can track the complete history of your infrastructure versions. Of course, when something is written, it can be shared, and collaborated on by teams. You can analyze it or apply some review processes. Everything is in place.
• Testing A regular software development process has a few pre-prod environments, and we can apply the same scenario with the IaC projects. It enables you to safely and predictably create, change, verify and improve infrastructure that can be passed to the next development environments. Let’s imagine doing such things via the web browser console. Can you even count how many mistakes can we make here? If something can be coded, it can be automated as well. So, let’s do it!
• Collaboration Easy and transparent collaboration - this is the consequence of getting the project in source files oriented repo. There are no contraindications for introducing the GitFlow workflow when it comes to the infrastructure development process.
• Separation of concerns The final concept is modularization. As you will see later in the article, multiple cloud service providers may be put together in one infrastructure. Each of the providers or group of services may be split into modules or recipes. It may be tested, deployed, and developed separately without an impact on the entire environment.

Real scenario

Let’s assume that we are building hosting for a web application from scratch. It’s a single-page application, and it has to be accessible worldwide - which means that CDN should be considered in this case. The web app needs to have a backend for storing the data and should be able to send emails as well. Nowadays, email service has to be protected by the SFP, DKIM mechanisms - let’s use them.

We want to use:

• VPS from Digital Ocean,
• CDN and DNS services from CloudFlare,
• MailGun services for mass email sending.

This set is just an example, and you may find plenty of alternatives or similar services. As an example, I wanted to use a set of services from different providers. Now I’m going to show you how the IaC may be implemented in this case.

Note: Terraform is written in Go, you can find the installation guide here: https://learn.hashicorp.com/terraform/getting-started/install.html

So, let’s code.

• Providers First, what we need to do is to create the provider configuration. Because Terraform is an abstraction layer, on the cloud services provider, you should find proper provider documentation. All available providers are listed here: terraform.io/docs/providers/. In this case, we need to write a couple of providers configs in .tf file:

provider "digitalocean" {
  token = "DO_TOKEN_GOES_HERE"
}

provider "mailgun" {
  api_key = "MAILGUN_API_KEY_GOES_HERE"
}

provider "cloudflare" {
  email = "CLOUDFLARE_ACCOUNT_EMAIL"
  token = "CLOUDFLARE_API_KEY"
}

• Droplets Our first resources are Droplets (virtual servers), configured as below:

  resource "digitalocean_droplet" "my_droplet" {
    name = "my_droplet_name"
    image = "debian-9-x64"
    region = "ams3"
    size = "s-1vcpu-1gb"
    private_networking = true
    monitoring = false
    backups = false
    resize_disk = true
  }

• DNS and CDN Here is the provisioning of DNS zone with the one DNS record. You can find here also some customization with the TLS settings. Our new services will be proxied through the CloudFlare CDN engine (option “proxied: true). Please note that we have the reference to the previously created droplet IP - we’re going to create a DNS entry for the brand new droplet (IP address is unknown at the creation time).

resource "cloudflare_zone" "my_domain_entry" {
  zone = "example.com"
}

resource "cloudflare_zone_settings_override" "my_domain_entry_config" {
  name = "${cloudflare_zone.my_domain_entry.zone}"
  settings {
    tls_1_3 = "on"
  }
}

resource "cloudflare_record" "app_example_com_host" {
  domain = "${cloudflare_zone.my_domain_entry.zone}"
  type   = "A"
  name   = "@"
  value  = "${digitalocean_droplet.my_droplet.ip_address}"
  proxied = true
}

• Mailgun - sending e-mails So finally, the configuration of Mailgun. As in the previous steps, note that TXT DNS entries required for secure mailing domain are unknown at the creation time - just after the creation of “example_mailing_domain” we can read and put necessary DNS records into the DNS CloudFlare zone.

resource "mailgun_domain" "example_mailing_domain" {
  name          = "m.example.com"
  spam_action   = "disabled"
  smtp_password = "strong_password"
}

resource "cloudflare_record" "mail-receiving-dns-entry" {
  count  = 2
  domain = "${cloudflare_zone.my_domain_entry.zone}"
  type   = "${lookup(mailgun_domain.example_mailing_domain.receiving_records\[count.index\], "record_type")}"
  name   = "${mailgun_domain.example_mailing_domain.name}."
  value  = "${lookup(mailgun_domain.example_mailing_domain.receiving_records\[count.index\], "value")}"
  priority = "${lookup(mailgun_domain.example_mailing_domain.receiving_records\[count.index\], "priority")}"
}

resource "cloudflare_record" "mail-sending-dns-entry" {
  count  = 3
  domain = "${cloudflare_zone.my_domain_entry.zone}"
  type   = "${lookup(mailgun_domain.example_mailing_domain.sending_records\[count.index\], "record_type")}"
  name   = "${lookup(mailgun_domain.example_mailing_domain.sending_records\[count.index\], "name")}."
  value  = "${lookup(mailgun_domain.example_mailing_domain.sending_records\[count.index\], "value")}"
}

Provider config should be placed into the provider.tf file. The rest of the configuration let’s put into the main.tf file. After that, it is possible to manage the infrastructure with “terraform” commands:

• “terraform init” - first, we have to initialize config, getting external plugins, create the local state file .tfstate, etc.
• “terraform plan” - it’s using for comparing the local state with the remote state and calculating the changes need to be performed on the remote providers,
• “terraform apply” - applied after all, when the changes have to be committed to the cloud providers, in this case to the DigitalOcean, CloudFlare and Mailgun clouds.

Conclusion

The way of creation and management services I described above is the way I prefer, and I believe in. Especially for complex systems with hundreds of services, resources, providers - honestly I don’t see other options to keep it manageable at all. The tool you may choose is the next issue for the next story. I gave you an example of how the config may look with the Terraform approach. You may take into consideration other amazing tools like Puppet, Chief, SaltStack. In the end, I wish you happy infrastructure coding! :)