Let's code some infrastructure with Terraform

Wojciech Gębiś - July 8, 2019

Intro

Perhaps for some of you, IT infrastructure management sounds easy, but it can be excruciating and full of disappointments. That is why, I wanted to share my thoughts and experiences which cover this subject. I finally decided to divide the article into two sections. In the beginning, I’m going to discuss a little bit the background of the entire DevOps tools ecosystem and explain what the Infrastructure-as-Code is. In the second part of the article, we will get our hands dirty with code by implementing a real-life example of using the IaC with Terraform.

Modern DevOps tools ecosystem

Let’s look at the areas listed below. Nowadays, each of the following is fundamental and need to be managed properly to have a deep and broad insight into the big picture of the DevOps world and to always keep calm on the production.

  • Configuration management
    We want to manage environments and application configuration in a centralized way. In the era of microservices, it is not possible to keep in check so many components. Especially when we want to do some runtime tuning and extra customization or we want to scale up some application resources in a maintenance-free manner. Tools like Ansible, Chef, or Puppet are designed for such operations.

  • Containers and Container orchestrators
    It doesn’t matter which container engine you decide to use - seriously. However, you should be familiar with at least one container technology. It’s worth mentioning that if the container engine is compliant with Open Containers Initiative (OCI), you may feel more comfortable - it guarantees a higher compatibility level with the orchestration systems like Kubernetes or Nomad. Using orchestration is always kind of trade-off when we consider small environments, but on the other hand, it can significantly speed up scaling the environments in the future. Tools like Kubernetes, Nomad should be considered to implement container orchestration.

  • Continuous Integration and Continuous Delivery
    Currently, it’s hard to imagine software development without these themes. They are already a part of the project DNA. I’m not going to go more in-depth at the moment, but I want to focus your attention on these pipelines. It doesn’t even matter which pipeline engine you choose, Jenkins and the Jenkinsfile, Travis CI and the .travis.yaml file or any other. Such tools are still actively developed (e.g. Jenkins X project: https://jenkins-x.io) and it’s good to keep an eye on it.

  • Infrastructure as Code
    Ok, so this is something I wanted to go deeper in my article. I choose Terraform as an implementation of the IoC approach due to the maturity of the project and a wide range of potential plugins for cloud providers.

What is Infrastructure-as-code?

In few words, this is a common approach for provisioning the cloud infrastructure based on the coded definitions. Everything that could be done via the cloud web console can be done via the IaC defined snippets - from small services (e.g., AWS Lambda, Azure function) through DNS services, firewall rules to the large database installations. Let’s take a closer look at the main IaC concepts:

  • Versioning, static analysis, review
    My favorite advantage of following IaC concept is version control. It’s a fundamental and pretty straightforward concept. Complete infrastructure design can be after all committed to the repo as a bucket of source code files. You can track the complete history of your infrastructure versions. Of course, when something is written, it can be shared, and collaborated on by teams. You can analyze it or apply some review processes. Everything is in place.

  • Testing
    A regular software development process has a few pre-prod environments, and we can apply the same scenario with the IaC projects. It enables you to safely and predictably create, change, verify and improve infrastructure that can be passed to the next development environments. Let’s imagine doing such things via the web browser console. Can you even count how many mistakes can we make here? If something can be coded, it can be automated as well. So, let’s do it!

  • Collaboration
    Easy and transparent collaboration - this is the consequence of getting the project in source files oriented repo. There are no contraindications for introducing the GitFlow workflow when it comes to the infrastructure development process.

  • Separation of concerns
    The final concept is modularization. As you will see later in the article, multiple cloud services providers may be put together in one infrastructure. Each of the providers or group of services may be split into modules or recipes. It may be tested, deployed, developed separately without the impact on the entire environment.

Real scenario

Let’s assume that we are building hosting for a web application from scratch. It’s a single page application, and it has to be accessible worldwide - it means that CDN should be considered in this case. The web app needs to have a backend for storing the data and should be able to send emails as well. Nowadays, email service has to be protected by the SFP, DKIM mechanisms - let’s use them.

We want to use:

  • VPS from Digital Ocean,
  • CDN and DNS services from CloudFlare,
  • MailGun services for mass emails sending.

This set is just an example, and you may find plenty of alternatives or similar services. As an example, I wanted to use a set of services from different providers. Now I’m going to show you how the IaC may be implemented in this case.

Note: Terraform is written in Go, you can find the installation guide here: https://learn.hashicorp.com/terraform/getting-started/install.html

So, let’s code.

  • Providers
    First, what we need to do is to create the provider configuration. Because Terraform is an abstraction layer, on the cloud services provider, you should find proper provider documentation. All available providers are listed here: terraform.io/docs/providers/. In this case, we need to write a couple of providers configs in .tf file:

  • Droplets
    Our first resources are Droplets (virtual servers), configured as below:

  • DNS and CDN
    Here is the provisioning of DNS zone with the one DNS record. You can find here also some customization with the TLS settings. Our new services will be proxied through the CloudFlare CDN engine (option “proxied = true). Please note that we have the reference to the previously created droplet IP - we’re going to create a DNS entry for the brand new droplet (IP address is unknown at the creation time).

  • Mailgun - sending e-mails
    So finally, the configuration of Mailgun. As in the previous steps, note that TXT DNS entries required for secure mailing domain are unknown at the creation time - just after the creation of “example_mailing_domain” we can read and put necessary DNS records into the DNS CloudFlare zone.

Provider config should be placed into the provider.tf file. The rest of the configuration let’s put into the main.tf file. After that, it is possible to manage the infrastructure with “terraform” commands:

  • “terraform init” - first, we have to initialize config, getting external plugins, create the local state file .tfstate, etc.
  • “terraform plan” - it’s using for comparing the local state with the remote state and calculating the changes need to be performed on the remote providers,
  • “terraform apply” - applied after all, when the changes have to be committed to the cloud providers, in this case to the DigitalOcean, CloudFlare and Mailgun clouds.

Conclusion

The way of creation and management services I described above is the way I prefer, and I believe in. Especially for complex systems with hundreds of services, resources, providers - honestly I don’t see other options to keep it manageable at all. The tool you may choose is the next issue for the next story. I gave you an example of how the config may look with the Terraform approach. You may take into consideration other amazing tools like Puppet, Chief, SaltStack. In the end, I wish you happy infrastructure coding! :)

About the author

Wojciech gęiś

Project Lead & DevOps Engineer

Wojciech is a seasoned engineer with experience in development and management. He has worked on many projects and in different industries, making him very knowledgeable about what it takes to succeed in the workplace by applying Agile methodologies. Wojciech has deep knowledge about DevOps principles and Machine Learning. His practices guarantee that you can reliably build and operate a scalable AI solution.
You can find Wojciech working on open source projects or reading up on new technologies that he may want to explore more deeply.

Tempted to work
on something
as creative?

That’s all we do.

join nexocode

Find us on

Need help with implementing AI in your business?

Let's talk blue circle

This site uses cookies for analytical purposes.

Accept Privacy Policy

In the interests of your safety and to implement the principle of lawful, reliable and transparent processing of your personal data when using our services, we developed this document called the Privacy Policy. This document regulates the processing and protection of Users’ personal data in connection with their use of the Website and has been prepared by Nexocode.

To ensure the protection of Users' personal data, Nexocode applies appropriate organizational and technical solutions to prevent privacy breaches. Nexocode implements measures to ensure security at the level which ensures compliance with applicable Polish and European laws such as:

  1. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (published in the Official Journal of the European Union L 119, p 1); Act of 10 May 2018 on personal data protection (published in the Journal of Laws of 2018, item 1000);
  2. Act of 18 July 2002 on providing services by electronic means;
  3. Telecommunications Law of 16 July 2004.

The Website is secured by the SSL protocol, which provides secure data transmission on the Internet.

1. Definitions

  1. User – a person that uses the Website, i.e. a natural person with full legal capacity, a legal person, or an organizational unit which is not a legal person to which specific provisions grant legal capacity.
  2. Nexocode – NEXOCODE sp. z o.o. with its registered office in Kraków, ul. Generała Henryka Kamieńskiego 51, 30-644 Kraków, entered into the Register of Entrepreneurs of the National Court Register kept by the District Court for Kraków-Śródmieście in Kraków, 11th Commercial Department of the National Court Register, under the KRS number: 0000686992, NIP: 6762533324.
  3. Website – website run by Nexocode, at the URL: nexocode.com whose content is available to authorized persons.
  4. Cookies – small files saved by the server on the User's computer, which the server can read when when the website is accessed from the computer.
  5. SSL protocol – a special standard for transmitting data on the Internet which unlike ordinary methods of data transmission encrypts data transmission.
  6. System log – the information that the User's computer transmits to the server which may contain various data (e.g. the user’s IP number), allowing to determine the approximate location where the connection came from.
  7. IP address – individual number which is usually assigned to every computer connected to the Internet. The IP number can be permanently associated with the computer (static) or assigned to a given connection (dynamic).
  8. GDPR – Regulation 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of individuals regarding the processing of personal data and onthe free transmission of such data, repealing Directive 95/46 / EC (General Data Protection Regulation).
  9. Personal data – information about an identified or identifiable natural person ("data subject"). An identifiable natural person is a person who can be directly or indirectly identified, in particular on the basis of identifiers such as name, identification number, location data, online identifiers or one or more specific factors determining the physical, physiological, genetic, mental, economic, cultural or social identity of a natural person.
  10. Processing – any operations performed on personal data, such as collecting, recording, storing, developing, modifying, sharing, and deleting, especially when performed in IT systems.

2. Cookies

The Website is secured by the SSL protocol, which provides secure data transmission on the Internet. The Website, in accordance with art. 173 of the Telecommunications Act of 16 July 2004 of the Republic of Poland, uses Cookies, i.e. data, in particular text files, stored on the User's end device.
Cookies are used to:

  1. improve user experience and facilitate navigation on the site;
  2. help to identify returning Users who access the website using the device on which Cookies were saved;
  3. creating statistics which help to understand how the Users use websites, which allows to improve their structure and content;
  4. adjusting the content of the Website pages to specific User’s preferences and optimizing the websites website experience to the each User's individual needs.

Cookies usually contain the name of the website from which they originate, their storage time on the end device and a unique number. On our Website, we use the following types of Cookies:

  • "Session" – cookie files stored on the User's end device until the Uses logs out, leaves the website or turns off the web browser;
  • "Persistent" – cookie files stored on the User's end device for the time specified in the Cookie file parameters or until they are deleted by the User;
  • "Performance" – cookies used specifically for gathering data on how visitors use a website to measure the performance of a website;
  • "Strictly necessary" – essential for browsing the website and using its features, such as accessing secure areas of the site;
  • "Functional" – cookies enabling remembering the settings selected by the User and personalizing the User interface;
  • "First-party" – cookies stored by the Website;
  • "Third-party" – cookies derived from a website other than the Website;
  • "Facebook cookies" – You should read Facebook cookies policy: https://www.facebook.com/policy/cookies
  • "Other Google cookies" – Refer to Google cookie policy: www.google.com/policies/technologies/types/

3. How System Logs work on the Website

User's activity on the Website, including the User’s Personal Data, is recorded in System Logs. The information collected in the Logs is processed primarily for purposes related to the provision of services, i.e. for the purposes of:

  • analytics – to improve the quality of services provided by us as part of the Website and adapt its functionalities to the needs of the Users. The legal basis for processing in this case is the legitimate interest of Nexocode consisting in analyzing Users' activities and their preferences;
  • fraud detection, identification and countering threats to stability and correct operation of the Website.

4. Cookie mechanism on the Website

Our site uses basic cookies that facilitate the use of its resources. Cookies contain useful information and are stored on the User's computer – our server can read them when connecting to this computer again. Most web browsers allow cookies to be stored on the User's end device by default. Each User can change their Cookie settings in the web browser settings menu: Google ChromeOpen the menu (click the three-dot icon in the upper right corner), Settings > Advanced. In the "Privacy and security" section, click the Content Settings button. In the "Cookies and site date" section you can change the following Cookie settings:

  • Deleting cookies,
  • Blocking cookies by default,
  • Default permission for cookies,
  • Saving Cookies and website data by default and clearing them when the browser is closed,
  • Specifying exceptions for Cookies for specific websites or domains

Internet Explorer 6.0 and 7.0
From the browser menu (upper right corner): Tools > Internet Options > Privacy, click the Sites button. Use the slider to set the desired level, confirm the change with the OK button.

Mozilla Firefox
browser menu: Tools > Options > Privacy and security. Activate the “Custom” field. From there, you can check a relevant field to decide whether or not to accept cookies.

Opera
Open the browser’s settings menu: Go to the Advanced section > Site Settings > Cookies and site data. From there, adjust the setting: Allow sites to save and read cookie data

Safari
In the Safari drop-down menu, select Preferences and click the Security icon.From there, select the desired security level in the "Accept cookies" area.

Disabling Cookies in your browser does not deprive you of access to the resources of the Website. Web browsers, by default, allow storing Cookies on the User's end device. Website Users can freely adjust cookie settings. The web browser allows you to delete cookies. It is also possible to automatically block cookies. Detailed information on this subject is provided in the help or documentation of the specific web browser used by the User. The User can decide not to receive Cookies by changing browser settings. However, disabling Cookies necessary for authentication, security or remembering User preferences may impact user experience, or even make the Website unusable.

5. Additional information

External links may be placed on the Website enabling Users to directly reach other website. Also, while using the Website, cookies may also be placed on the User’s device from other entities, in particular from third parties such as Google, in order to enable the use the functionalities of the Website integrated with these third parties. Each of such providers sets out the rules for the use of cookies in their privacy policy, so for security reasons we recommend that you read the privacy policy document before using these pages. We reserve the right to change this privacy policy at any time by publishing an updated version on our Website. After making the change, the privacy policy will be published on the page with a new date. For more information on the conditions of providing services, in particular the rules of using the Website, contracting, as well as the conditions of accessing content and using the Website, please refer to the the Website’s Terms and Conditions.

Nexocode Team