Running Backstage In Production On AWS

Scaling Engineering Organizations

When an engineering organization is small, it is easy to hold the mental model of the system in mind. Troubleshooting and developing new features is fairly straightforward, and context about the system can be shared with newcomers in a few hours of whiteboarding.

However, as the organization grows, new services are created, which often correspond to new software delivery teams. Fast forward a few years, and what you often find is hundreds of services supported by dozens of teams. At this point, the mental model of the system is too large to understand in its entirety and the difficulty in conveying context results in breakdowns in communication as described by Conway's law.

As an advanced AWS consulting partner, StratusGrid has encountered numerous situations where agility is impaired due to the engineering organization growing at an accelerated pace. We’ve found that the implementation of Backstage offers three highly effective solutions to address challenges faced by large engineering organizations.

First, Backstage provides a software catalog that tracks dependencies between projects. This enables all members within the organization to rapidly gain context across projects and to hold the entire system in mind.

Second, Backstage integrates with commonly used third party developer tools. When troubleshooting a particular service, a team member can view build status, recent pull requests, monitoring alerts, api specifications, architecture diagrams, and more from a single pane of glass.

Third, Backstage enables the creation of new software projects using scaffolded templates. Create new services, modules, or other types of software with the click of a button. After clicking the button, Backstage automates the creation of a new GitHub repository with the organization's preferred settings, created from a template that configures CI/CD and infrastructure as code.

This third feature of Backstage is of critical importance. Gartner predicts that by 2026, 80% of all software engineering organizations will have established platform teams that provide reusable components to engineering organizations in order to accelerate software delivery. Backstage consolidates your reusable components and automation and makes them available for self-service to your software delivery teams.

StratusGrid helps clients build and operate mission-critical platforms and applications in the cloud - the right way. Backstage increases the reliability of your platform by enabling context sharing between engineering teams, it increases agility in the creation and development of new services, and it enables team members in various roles to understand the complex relationships between the software in your catalog.

Although Backstage provides features essential to success, configuring and operating Backstage requires careful planning. StratusGrid created a walkthrough which enables you to spin up Backstage on ECS for evaluation purposes, and subsequent sections of this article will delve into key aspects you'll need to consider for production-level roll out of Backstage.

Backstage Architecture

Backstage consists primarily of two TypeScript packages, backend, and frontend, which are managed using yarn in a React application. We chose to build the backend and frontend packages into a single container running on ECS Fargate. The database is stored in Aurora PostgreSQL.

Implementation

Local Development

The first step is to run Backstage locally. Instructions for installing Backstage and running it locally are available on the Backstage Getting Started page.

When you have multiple configuration files, Backstage will merge the files upon startup, and the files that appear later in the argument list will take precedence. Therefore, put all configuration information that is shared between environments in app-config.yaml. Put your staging-specific configuration into app-config.staging.yaml and production-specific configuration into app-config.production.yaml.

Infrastructure Provisioning

Using the walkthrough, deploy the infrastructure listed in the above architecture diagram. The Terraform provided will provision clusters, ECS services, databases, ALBs, Route53 records, and certificates for each environment. A single ECR repository will be shared by both ECS clusters.

The Terraform provided in the walkthrough provides an interesting solution for managing the task definition for ECS services. When using Terraform, you will codify the task definition. However, when you deploy using CI/CD, you need to update the task definition outside of Terraform. How can you ensure that Terraform provisions changes to the task definition but doesn't clobber over changes that have been made outside of Terraform?

At StratusGrid, we have extensive experience managing infrastructure as code without impairing code deployments. The answer we have implemented is to define the task definition as a data source and to also create a resource to manage it. This solution is illustrated in the ECS Module.

Configuring Backstage

Identity Federation

To prepare Backstage for use in your organization you will need to integrate with an identity provider. By default, Backstage contains a guest account and provides no authentication. By choosing GitHub for identity management, users and teams in your GitHub org will be translated into Backstage Users and Groups. Users will log into Backstage using an Oauth login flow. Backstage will be aware of what Groups the user belongs to, and plugins can provide important information about the team, such as all open pull requests and incidents.

In order to configure GitHub authentication, follow the instructions to create a GitHub Oauth App.

Then add the secrets locations to the ECS Container Definition.

Database

The next step is to configure database access. StratusGrid recommends setting sqlite3 for local development in your app-config.local.yaml and specifying separate Aurora PostgreSQL databases for staging and production. You can configure the database connection once in app-config.yaml and pull the connection info from environment variables. Here is an example:

This configuration in the app-config.yaml will be shared between staging and production environments, but the actual connection information is loaded from AWS Secrets Manager into environment variables when the ECS Task is started.

3rd Party Integrations

Backstage is highly extensible. Several popular 3rd party plugins exist, and writing your own is straightforward. At StratusGrid, we commonly start with plugins for CircleCI, API Documentation, GitHub Pull Requests, OpsGenie, and Snyk.

Backstage makes it easy to view the most important information about a service, such as documentation, important links, and Snyk vulnerabilities without having to click through to external applications.

Plugins come from many sources, but for starting out, we recommend the Core Plugins provided by Backstage and choosing from the plugins made available by Roadie. Another plugin that enables automated infrastructure provisioning is the AWS Proton open-source Backstage plugin. Spotify also has paid plugins available.

Look And Feel

In order to customize the look and feel of Backstage, you can create a new theme in the packages/app/src/theme directory. This will control the layout for the home page and individual software catalog pages. You will then configure different options for each type of software entity. We normally choose to start with software entities of type service, website, and library, but you can add more entities if you create software that doesn't fall into these categories.

Separating the layouts for different software types enables you to show API Docs for services and enable Lighthouse statistics for websites.

See Customize the look-and-feel of your App for more details on creating custom themes.

CI/CD

When upgrading, developing plugins, or adding new integrations, it is recommended to run Backstage locally for fast feedback. Additionally, we typically operate two environments in AWS: staging and production. Each environment has its own cluster. When changes to Backstage are merged into the main branch, we deploy to staging and hold for manual approval. After the manual approval, the image is promoted to the production cluster.

We use a single ECR repository for Backstage images. Images are tagged with the environment name and with the git commit sha, which enables us to roll back if necessary. Our CI system assumes a role that has the minimum permissions required to publish to the ECR repository and deploy an ECS task in the Backstage environments.

This is a simple deployment workflow that works well because changes to Backstage can be tested locally, and unit tests run in CI before the PR is merged. This ensures high confidence before merging to the main branch, and the ability to evaluate changes in staging before deploying to production.

Using Backstage

Creating Services

Once Backstage is running in production and integrated with your GitHub organization, you can start creating templates to automate the creation of new software projects. Templates usually consist of two components: The template itself, which describes the form that users will fill out in order to create a new project, and a git repository that includes content you want to scaffold into each new project.

Backstage will use the input from your form to replace placeholder variables in your scaffolded content.

We use a single git repository to store our templates. For Backstage to load templates into the catalog, you need a rule in your configuration that allows access to the Template type:

A strong benefit of Backstage is ensuring consistency across your software projects. When using Backstage to create new GitHub repositories, you can configure Backstage to create the repositories with your preferred branch protection and feature settings. For example, we choose to require two approving reviews and to disable the Github Wiki feature.

When your template is finished and published to the Backstage catalog, you will be able to use it to create new services:

Importing Existing Services

While creating new software projects is an incredibly useful feature of Backstage, another strength is that it serves as a definitive software catalog which tracks dependencies between services. To take full advantage of this feature, you can import existing services into Backstage.

Backstage itself stores very little state. The state is provided by the integrated 3rd parties. In order to import existing services, you simply add a catalog-info.yaml file to the root of the project repository that includes the name of the component and a few metadata settings.

Backstage can submit pull requests on existing repositories. I recommend creating a template specifically for importing existing projects into Backstage. This way, you will only need to enter the name of the repository into the Backstage form, and your catalog-info.yaml file will be templated out. This is an example catalog-info.yaml file:

Day Two Operations

Backstage releases upgrades once per month. Since Backstage is a React application that you are managing, there will likely be code changes required with every upgrade. Start the upgrade process by running yarn backstage-cli versions:bump.

After running this command, you will need to update code for any APIs that have changed. Use the Upgrade Helper to see a visual diff of the changes and incorporate the changes into the Backstage packages and plugins.

Next Steps

Follow the Backstage on AWS Walkthrough which includes everything required to provision Backstage on AWS for evaluation purposes. Fork the repository and begin implementing the suggestions in this article.

Conclusion

The adoption of Backstage in large engineering organizations has proven to be a significant step towards improving end-user engagement, agility, and communication between teams. By providing a centralized platform that integrates with various third-party tools and automates the creation of new software projects, Backstage helps manage the complex ecosystems that inevitably arise as organizations grow.

With the instructions provided in this article, you are well on your way to running a production instance of Backstage in AWS. For help with Backstage implementation or other Cloud modernization efforts, reach out to StratusGrid.