How to Build a Private Terraform Provider Registry in S3

A Quick Intro to Private Terraform Provider Registry

If you work at an organization where many people and perhaps multiple teams are using Terraform, you may find yourself in a situation where you want a consistent set of data available to everyone. Or, you may have a private API that is only accessible internally, and you want team members to be able to configure that API using Terraform.

In either case, writing either a data source-only Terraform Provider or a full Terraform Provider that integrates with your private API may be useful. However, if you are not a consumer of Terraform Cloud, you need an alternative way of making your Terraform Provider available to team members across the organization.

In this post, we will walk through one method of making this Terraform Provider accessible. We will use a self-hosted solution in AWS S3, which limits access by IP address.

Creating a Terraform Provider

Hashicorp has an excellent tutorial on creating a custom Terraform provider. I recommend cloning the hashicups provider repository as a starting point.

After you have downloaded the example provided, it is fairly trivial to make adjustments to sort your use case. If you are creating a data source, you will create a new data_source_<source_name>.go file in the provider directory. You will define the schema , and you will add some logic to return the data from your real-life data source.

The hashicups provider also has a handy Makefile for building your Provider, and it will install the binary into your ~/.terraform.d directory so that you can use it locally for testing before publishing.

Check out the entire tutorial for in-depth explanations on how to develop your own provider. Feel free to reach out if you get stuck.

Preparing the Terraform Provider for Release

Now that you are finished developing your shiny new provider, you need to make the artifacts available. You’ll need to publish them to a location where the team member using Terraform has access using a URL. This could be a company-hosted artifact store, or you could use S3. The important thing is that you have a download URL for the three files we're about to generate.

Before we can generate the files, you will need a GPG key which is used to verify the Provider package. Make sure you follow these instructions to generate the key, because if you use the default GPG options, the key will not work with the openpgp module used by Terraform to verify the package.

Generating the Necessary Files for Release

The three files we need to generate are the download (zip file containing the provider), the shasums (a file listing shasum of the zip file), and a shasums signature file (a signature created with your gpg key that can be used to verify that the shasums file has not been tampered with).

To step through what is happening here, you have a zip file that contains only the Go binary for a particular OS and Architecture. You then generate a SHA256 hash of the zip file. Anyone else in the world can also run a SHA256 hash on the zip file and they will receive the same output, allowing them to be sure that the zip file they downloaded is exactly the same as the file you uploaded. Finally, you create a signature using your GPG private key of the SHASUM file, which anyone else (using your public key) can use to verify that, in fact, you are the one who published the SHA256 file.

There is this helper script called shasign.sh , which you can use to generate the SHA files. This script makes things a lot easier, but there is an important change you will need to make. Remove the --armor option on line 64 because the OpenPGP library used by Terraform requires a binary file, not an ASCII representation.

At this point, you should have your three files and the URLs to access them. Now we can get started on registering our provider so that it is available to team members to use in their Terraform projects.

The details on running your own Terraform Provider Registry are published in the Provider Registry Protocol . Essentially, what you need is an API that responds appropriately to the specified requests.

Overcoming S3 Limitations

You may be thinking that S3 alone does not allow you to publish an API, and you would be correct. However, we can fake the functionality of an API by simply returning JSON objects and setting the content type to application/json in S3.

Another issue is that using S3 as a website does not support TLS, and TLS is a requirement for Terraform provider registries. To work around this, we will need to put a CloudFront distribution in front of S3.

Lastly, we need a WAF in front of CloudFront so that we can limit access to the distribution. A common setting is to limit access to your VPN endpoints.

Automating Configuration with StratusGrid’s Module

Setting up this directory structure by hand would be quite tiresome. StratusGrid has created a Terraform Module that will automate the configuration for you. For the rest of this post, we will assume you are using the module.

For inputs to the module, you will need to provide the following: The three URLs that you generated above, which point to your provider zip file, your SHASUMS file, and your SHASUMS signature. Your SHASUM (which you can retrieve from your SHASUMS file), and your ascii-armored GPG public key.

To get your GPG Key ID run gpg --list-keys. To export the key, run gpg --armor --export <YOUR KEY ID>.

For each version of your provider, you will need to provide a version object to Terraform and a package object. At this point, you have all the necessary information to fill in the values for those objects.

Accessing the Private Registry

After you have deployed your private registry, you access it as follows:

Next Steps

Congratulations on having pushed your own privately hosted Terraform Provider! Going forward, a great next step would be to automate the registration of your Terraform Provider on every release of your provider code. You could do this by automatically generating the inputs that we walked through manually above.

Streamline Your Terraform Workflows with StratusGrid's Private Registry Solutions

Take control of your Terraform provider management with StratusGrid's innovative approach to building a private Terraform Provider Registry in AWS S3. Whether you're managing internal APIs or seeking a streamlined workflow for your Terraform resources, StratusGrid is here to transform your infrastructure as code practices.

Contact us today to discover how our tailored private registry solutions can optimize your Terraform deployments and drive your cloud infrastructure forward.