Bluetab
an IBM Company
Introduction to HashiCorp products
Jorge de Diego
Cloud DevOps Engineer
At Cloud Practice we look to encourage the use of HashiCorp products and we are going to publish thematic articles on each of them to do this.
Due to the numerous possibilities their products offer, this article will provide an overview, and we will go into detail on each of them in later publications, showing unconventional use cases that show the potential of HashiCorp products.
Why HashiCorp?
HashiCorp has been developing a variety of Open Source products over recent years that offer cross-cutting infrastructure management in cloud and on-premises environments. These products have set standards in infrastructure automation.
Its products now provide robust solutions in provisioning, security, interconnection and workload coordination fields.
The source code for its products is released under MIT licence, which has been well received within the Open Source community (over 100 developers are continually providing improvements)
In addition to the products we present in this article, they also have attractive Enterprise solutions.
As regards the impact of its solutions, Terraform has become a standard on the market. This means HashiCorp is doing things right, so understanding and learning about its solutions is appropriate.
Although we are focusing on their use in the Cloud environment, their solutions are widely used in on-premises environments, but they reach their full potential when working in the cloud.
What are their products?
HashiCorp has the following products:
- Terraform: infrastructure as code.
- Vagrant: virtual machines for test environments.
- Packer: automated building of images.
- Nomad: workload “orchestrator”.
- Vault: management of secrets and data protection.
- Consul: management and discovery of services in distributed environments.
We summarise the main product features below. We will go into detail on each of them in later publications.
Terraform has positioned itself as the most widespread product in the field of provisioning of Infrastructure as Code.
It uses a specific language (HCL) to deploy infrastructure through the use of various Cloud providers. Terraform also lets you manage resources from other technologies or services, such as GitLab or Kubernetes.
Terraform makes use of a declarative language and is based on having deployed exactly what is present in the code.
The typical example shown to see the difference with the paradigm followed, for example, by Ansible (procedural) is:
Initially we want to deploy 5 EC2 instances on AWS:
Ansible:
- ec2:
count: 5
image: ami-id
instance_type: t2.micro Terraform:
resource "aws_instance" "ejemplo" {
count = 5
ami = "ami-id"
instance_type = "t2.micro"
} As you can see, there are virtually no differences between the coding. Now we need to deploy two more instances to cope with a heavier workload:
Ansible:
- ec2:
count: 2
image: ami-id
instance_type: t2.micro Terraform:
resource "aws_instance" "ejemplo" {
count = 7
ami = "ami-id"
instance_type = "t2.micro"
} At this point, we can see that while in Ansible we would set the number of instances to be deployed at 2 so that a total of 7 are deployed, in Terraform we would set the number directly to 7 and Terraform knows that it needs to deploy 2 more because there are already 5 deployed.
Another important aspect is that Terraform does not need a Master server such as Chef or Puppet. It is a distributed tool whose common, centralised element is the tfstate file (explained in the next paragraph). It is launched from any site with access to the tfstate (which can be remote, stored in common storage such as AWS S3) and with Terraform installed, which is distributed as a binary downloadable from the HashiCorp website.
The last point to note about Terraform is that it is based on a file called tfstate, in which the information relating to infrastructure status is progressively stored and updated, which it consults to see whether changes need to be made to the infrastructure. It is very important to note that this state is what Terraform knows. Terraform will not connect to AWS to see what is deployed. This means it is necessary to avoid making changes manually to the infrastructure deployed by Terraform (and never Manual changes, ever), as the tfstate file is not updated and inconsistencies will therefore be created.
Vagrant enables you to deploy test environments locally, quickly and very simply, based on code. By default it is used with VirtualBox, but it is compatible with other providers such as VMware or Hyper-V. Machines can be deployed on Cloud providers such as AWS by installing plug-ins. I personally cannot find advantages in using Vagrant for this function.
The machines it deploys are based on boxes, and you indicate which box you want to deploy from the code. Its mode of operating can be compared with Docker in this aspect. However, the basis is completely different. Vagrant creates virtual machines with a virtualisation tool below it (VirtualBox), while Docker deploys containers and its support is a containerisation technology, Vagrant versus Docker.
Typical use cases can be to test Ansible playbooks, recreate a laboratory environment locally in just a few minutes, making a demo, etc.
Vagrant is based on a Vagrantfile. Once located in the directory where this Vagrantfile is located, by running vagrant up, the tool begins to deploy what is indicated within that file.
The steps to launch a virtual machine with Vagrant are:
1. Install Vagrant. It is distributed as a binary file, like all other HashiCorp products.
2. With an example Vagrantfile:
Vagrant.configure("2") do |config|
config.vm.box = "gbailey/amzn2"
end PS: (the parameter 2 within configure refers to the version of Vagrant, which I had to check for the post)
3. Run within the directory where the Vagrantfile is located
vagrant up 4. Enter the machine using ssh
vagrant ssh 5. Destroy the machine
vagrant destroy Vagrant manages ssh access by creating a private key and storing it in the .vagrant directory, apart from other metadata. The machine can be accessed by running vagrant ssh. The machines deployed can also be displayed in the VirtualBox application.
With Packer you can build automated machine images. It can be used, for example, to build an image on a Cloud provider, such as AWS, already with an initial configuration, and to be able to deploy it an indefinite number of times. This means that you only need to provision it once, and when you deploy the instance with that image you already have the desired configuration without having to spend more time provisioning it.
A simple example would be:
1. Install Packer. Similarly, it is a binary that will need to be placed in a folder located on our path.
2. Create a file, e.g. builder.json. A small script is also created in bash (the link shown in builder.json is dummy):
{
"variables": {
"aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
"aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
"region": "eu-west-1"
},
"builders": [
{
"access_key": "{{user `aws_access_key`}}",
"ami_name": "my-custom-ami-{{timestamp}}",
"instance_type": "t2.micro",
"region": "{{user `region`}}",
"secret_key": "{{user `aws_secret_key`}}",
"source_ami_filter": {
"filters": {
"virtualization-type": "hvm",
"name": "amzn2-ami-hvm-2*",
"root-device-type": "ebs"
},
"owners": ["amazon"],
"most_recent": true
},
"ssh_username": "ec2-user",
"type": "amazon-ebs"
}
],
"provisioners": [
{
"type": "shell",
"inline": [
"curl -sL https://raw.githubusercontent.com/example/master/userdata/ec2-userdata.sh | sudo bash -xe"
]
}
]
} Provisioners use builtin and third-party software to install and configure the machine image after booting. Our example ec2-userdata.sh
yum install -y \
python3 \
python3-pip \
pip3 install cowsay 3. Run:
packer build builder.json And we would already have an AMI provisioned with cowsay installed. Now it will never be necessary for you, as a first step after launching an instance, to install cowsay because we will already have it as a base, as it should be.
As you would expect, Packer not only works with AWS, but also with any Cloud provider such as Azure or GCP. It also works with VirtualBox and VMware and a long list of builders. You can also create an image by nesting builders in the Packer configuration file that is the same for various Cloud providers. This is very interesting for multi-cloud environments, where different configurations need to be replicated.
Nomad is a workload orchestrator. Through its Task Drivers they execute a task in an isolated environment. The most common use case is to orchestrate Docker containers. It has two basic actors: Nomad Server and Nomad Client. The two actors are run with the same binary, but different configurations. Servers organise the cluster, while Agents run the tasks.
A little “Hello world” in Nomad could be:
1. Download and install Nomad and run it (in development mode; this mode must never be used in production):
nomad agent -dev Once this has been done, you can access the Nomad UI at localhost:4646
2. Run an example job that will launch a Grafana image
nomad job run grafana.nomad grafana.nomad:
job "grafana" {
datacenters = [
"dc1"
]
group "grafana" {
task "grafana"{
driver = "docker"
config {
image = "grafana/grafana"
}
resources {
cpu = 500
memory = 256
network {
mbits = 10
port "http" {
static = "3000"
}
}
}
}
}
} 3. Access localhost:3000 to check that you can access grafana and access the Nomad UI (localhost:4646) to view the job
4. Destroy the job
nomad stop grafana 5. Stop the Nomad agent. If it has been run as indicated here, pressing Control-C on the terminal where it is running will suffice
In short, Nomad is a very light orchestrator, unlike Kubernetes. Logically, it does not have the same features as Kubernetes, but it offers the ability to run workloads at high availability easily and without the need for a large amount of resources. We will deal with examples in greater detail in the post on Nomad to be published in the future.
Vault manages secrets. Users or applications can request secrets or identities through its API. These users are authenticated in Vault, which has a connection to a trusted identity provider such as an Active Directory, for example. Vault works with two types of actors, like the other tools mentioned, Server and Agent.
When Vault starts up, it starts in a sealed state (Seal) and various tokens are generated for unsealing (Unseal). An actual use case would be too long for this article and we will look at this in detail in the article on Vault alone. However, you can download and test it in dev mode as noted previously for Nomad. In this mode, Vault starts up in Unsealed mode, storing everything in memory without need for authentication, without using TLS, and with a single sealing key. You can play with its features and explore it quickly and easily in this way.
See this link for further information.
1. Install Vault. Like the rest of HashiCorp’s products, it is distributed as a binary through its website. Once done, launch (this mode must never be used in production):
vault server -dev 2. Vault will display the root token used to authenticate against Vault. It will also display the single sealing key it creates. Vault is now accessible via localhost:8200
3. In another terminal, export the following variables to be able to conduct tests:
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_DEV_ROOT_TOKEN_ID="token-showed-when-running-vault-dev" 4. Create a secret (in the terminal where the above variables were exported)
vault kv put secret/test supersecreto=iewdubwef293bd8dn0as90jdasd 5. Recover that secret
vault kv get secret/test These steps are the Hello world of Vault, to check its operation. We will look in detail at the operation and installation of Vault in the article on it, as well as at further features such as the policies, exploring the UI, etc.
Consul is used to create a services network. It operates in a distributed manner and a Consul client will run in a location where there are services that you want to register. Consul includes Health Checking of services, a key value database and features to secure network traffic. Like Nomad and Vault, Consul has two primary actors, Server and Agent.
Just as we did with Nomad and Vault, we are going to run Consul in dev mode to show a small example defining a service:
1. Install Consul. Create a working folder and within it create a json file with a service to be defined (example folder name: consul.d).
web.json file (HashiCorp website example):
{
"service": {
"name": "web",
"tags": [
"rails"
],
"port": 80
}
} 2. Run the Consul agent by pointing it to the configuration directory created before where web.json is located (this mode must never be used in production):
consul agent -dev -enable-script-checks -config-dir=./consul.d 3. From now, although there is nothing actually running on the indicated port (80), a service has been created with a DNS name that can be requested to access it through that DNS name. Its name will be NAME.service.consul. In our case NAME is web. Run to query:
dig @127.0.0.1 -p 8600 web.service.consul Note: Consul runs by default on port 8600.
Similarly, you can dig for a record of SRV type:
dig @127.0.0.1 -p 8600 web.service.consul SRV You can also query through the service tags (TAG.NAME.service.consul):
dig @127.0.0.1 -p 8600 rails.web.service.consul And you can query through the API:
curl http://localhost:8500/v1/catalog/service/web Final note: Consul is used as a complement to other products, as it is used to create services on other tools already deployed. That is why this example is not too illustrative. Apart from the specific article on Consul, it will be found as an example in other articles, such as the one on Nomad.
Future articles will also explain Consul Service Mesh for interconnection of services via Sidecar proxies, federation of Nomad Datacentres in Consul to perform deployments or intentions, which are used to define rules under which services can communicate with another service.
Conclusions
All these products have great potential in their respective areas and offer cross-cutting management that can help to avoid the famous vendor lock-in by Cloud providers. But the most important thing is their interoperability and compatibility.
¿Quieres saber más de lo que ofrecemos y ver otros casos de éxito?
Jorge de Diego
Cloud DevOps Engineer
My name is Jorge de Diego and I specialise in Cloud environments. I usually work with AWS, although I have knowledge of GCP and Azure. I joined Bluetab in September 2019 and have been working on Cloud DevOps and security tasks since then. I am interested in everything related to technology, especially security models and Infrastructure fields. You can spot me in the office by my shorts.
SOLUTIONS, WE ARE EXPERTS
You may be interested in
Footer
