Terraform - Tips and Tricks while using with AWS

Having worked quite extentsively with AWS Cloudformation, I was quite apprehensive about using terraform. Mainly becuase I guess I had to move out of my comfort zone! Nonetheless, I went on this journey to start using Terraform and learn as I went ahead with it. Needless to say I did end up discovering the pros and the cons of the terraform provisioning environment.

Now after spending a few months using Terraform I think its a good time to share my learnings from the experience.

    A Precursor to pros and cons:
  • Do not assume anything!
  • If you've used any provisioning tool other than Terraform then make sure you don't assume anything based on the tools you have used, while moving forward with Terraform.
  • No major version released yet
  • Terraform still hasn't had a major version release, so it may not be as evolved as you'd expect an infrastructure provisioning tool to be.
    Lets start with the good things.
  • terraform plan gives the ability to see what changes are going to be applied and more importantly how the existing resources would be affected. This is almost like a pre-deployment QA on the live setup. Similar to Cloudformation Change Sets.
  • The create_before_destroy flag can be set for pretty much all resources and terraform honours it even if AWS doesn't provide such an an attribute natively for the resources.
  • Terraform gives the ability to take existing resources created by some other means and bring it under the Terraform plan.
  • Terraform syntax is very easy to use.
  • Terraform supports a large number of providers which makes it easy to use one tools across the board rather than a bunch.
    Moving on to outliers
  • Terraform stores the state in a file with the .tfstate extension. Basically if the file gets accidently deleted or the file gets corrupted, the stack state is lost! This can be regenerated by either running the terraform plan again or by import each and every resource. However, it should be noted that
    Each resource in Terraform must implement some basic logic to become importable. As a result, not all Terraform resources are currently importable.Source
  • State Locking is a big concern with Terraform since the state is stored physically on disk. Each terraform plan need to specify the backend (read state) that it needs to initialize before provisioning or updating managed infrastructure. If two terraform run access the same state file and are triggered at the same time then they'll end up accessing and altering the same state. Potentially corrupting the state file.
  • Specific to autoscaling groups and launch configurations, terraform doesn't associate an updated launch configuration with an existing autoscaling group.
  • If a terraform plan fails mid way through a run then it doesn't rollback to the earlier stable version. Rather the stack might end up in a state of limbo where certain resources that should have been replaced during the run end up getting deleted.
  • Unlike Cloudformation, the resources within a terraform plan are not automatically tagged with the stack id. All tags need to be configured in the terraform module.
    Workarounds
    There are always workarounds, maybe not the most ideal but workable. Some of the workarounds for the outliers above are as below:
  • There is nothing that can be done around state corruption sadly. But state storage can be fixed. Ideally the state file should be kept remote and version controlled. Github may not be the most ideal location. A secured S3 bucket would be a better option.
  • To address state locking, terraform website provides a decent solution using S3 and dynamodb. But the configuration of the backend key to be unique still is a manual step.
    terraform {
        backend "s3" {
            role_arn = "arn:aws:iam::XXXXXXXX:role/terraform-state"
            region = "ap-southeast-1"
            lock_table = "terraform"
            bucket = "bucket-name"
            key = "make-sure-this-is-unique-per-terraform-plan-per-environment"
        }
    }
    
        provider "aws" {
            region = "ap-southeast-1"
            allowed_account_ids = ["XXXXXXXX"]
    }
  • In order to fix the autoscaling group and launch configuration syncing issue. The ideal way would be to have the autoscaling group and the launch configuration named the same.
    resource "aws_launch_configuration" "LC" {
        name_prefix   = "${var.stack_name}-"
        ...
        ...
        ...
    }
    
    resource "aws_autoscaling_group" "ASG" {
        name                      = "${aws_launch_configuration.LC.name}"
        launch_configuration      = "${aws_launch_configuration.LC.name}"
        ...
        ...
        ...
    }
  • To avoid resources being killed in case of a failed terraform run, make sure the create_before_destroy flag is set to true. However, this flag is only available for most resources, not all.
These are a few learnings from the terraform ecosystem. Enough to get started with atleast!