How to Remediate Amazon EBS Over-Provisioned Volumes

Amazon Elastic Block Store (EBS) enables you to provision block storage volumes in your Amazon Web Services (AWS) account. These storage volumes can be configured with a variety of size and performance options, which affects their monthly cost.

What is an Over-Provisioned EBS volume?

The over-provisioned EBS volume check comes from the AWS Trusted Advisor service. This service uses a proprietary algorithm to determine when EBS volumes should be flagged as over-provisioned. At a high level, it uses the performance utilization metrics of the EBS volume to determine if the provisioned capacity is being used or not.

If an EBS volume is over-provisioned, it means that the utilized capacity of the volume is not being fully consumed by workloads running in AWS.

How do I remediate over-provisioned EBS volumes?

Amazon EBS volumes can be reconfigured once every six hours. You can modify the performance characteristics of volumes, such as provisioned IOPS and throughput (aka. bandwidth), and the underlying storage type.

The modifications you can make to an EBS volume depend on the volume type.

• gp3 volumes do allow modification of both IOPS and Throughput
• io1 volumes do allow modification of IOPS, but not Throughput
• io2 Block Express volumes do allow modification of IOPS, but not Throughput
• gp2, sc1, st1 volumes do not allow modification of IOPS and Throughput

You can use the AWS Management Console to modify EBS volumes individually. To modify many EBS volumes, you can use automation tools like the AWS CLI or AWS PowerShell module to perform volume modifications at scale.

Example: Let's assume you have an EBS volume using the io2 Block Express volume type, with 

Risks & Other Considerations

• Amazon EBS volumes cannot be resized to a smaller size. If you need a larger EBS volume to be smaller, then you must create a new, smaller volume and manually move data from the larger volume to the smaller one.
  
• Amazon EBS volumes can only be reconfigured once every six hours, at most.
• Performance of an EBS volume may be somewhat degraded during volume modifications.
• After requesting an EBS volume modification, it may take some time for the new configuration to take full effect.

Remediation Steps

In order to remediate an over-provisioned Amazon EBS volume, you can reconfigure the volume's provisioned performance characteristics, using EBS volume modifications. 

Determine EBS Volume Used Capacity

Before you modify an EBS volume, you'll want to see what your currently utilized IOPS and Throughput (Bandwidth) are, over a given look-back period (eg. 2 weeks). You can examine the metrics for your EBS volumes in the Amazon CloudWatch metrics service.

• Login to the AWS Management Console
• Navigate to the Amazon CloudWatch service console
• Select All Metrics under the Metrics heading
• Select the EBS option under metrics (see screenshot below)
• Click the Per-Volume Metrics option (see screenshot below)
• Enter your EBS Volume ID in the search box and hit ENTER
• Enable the checkboxes for these metrics:
  • VolumeReadBytes
  • VolumeWriteBytes
  • VolumeWriteOps
  • VolumeReadOps
• Select the Graph Metrics tab
  • Change the Period to 1 minute
• Examine the data in the above graph and find some of the peak values 📈
• Divide the average values in the 1-minute data points by 60 to get per-second IOPS and Throughput

Once you've identified the average per-second IOPS and Throughput, you can use this information to make an informed decision about how to reconfigure your EBS volume.

In the graph screenshot below, you can see we're maxing out at roughly 7.8 GB/minute of EBS volume read throughput. If we divide that by 60, that's about 133 MB/sec of data throughput. Since this EBS volume has a GP3 volume type, with 125 MB/sec of throughput, we are actually exceeding the provisioned capacity very slightly.

For volume IOPS, we're using 164,000 read IOPS per minute, which is around 2,700 IOPS per second. Our GP3 volume already has the minimum of 3,000 IOPS configured, so we can't reduce that any further. In your case, you'll want to examine the provisioned IOPS for your volume, and compare it to the IOPS from CloudWatch, and determine if a reduction is possible.

AWS Management Console

If you'd like to use the AWS Management Console to modify an Amazon EBS volume, you can follow these steps:

• Login to the AWS Management Console
• Navigate to the EC2 service
• Under the Elastic Block Store heading, click Volumes
• Search for the EBS volume you're looking for
• Right-click the volume and click the Modify Volume option
  • Select a new Volume Type
  • Enter a new Size (larger only) if required
  • Enter the desired IOPS (depends on Volume Type)
  • Enter the desired Throughput / Bandwidth (depends on Volume Type)

AWS CLI

The AWS CLI tool can be used to automate changes to Amazon EBS volumes as well. This is helpful if you want to perform changes across a large number of volumes, in different regions, and in different AWS accounts.

Here's a sample command you could use to modify a specific volume ID. Let's say we wanted to increase the IOPS from 3,000 to 3,500, for a GP3 volume type.

aws ebs modify-volume --volume-id vol-1234567890 --iops 3500

Here's a sample command that changes the IOPS and Throughtput for a GP3 volume. The --throughput parameter accepts an integer in megabytes per second (MBps). If you're using the default 125 MB/sec of throughput for GP3, this would increase it to 300 MB/sec.

aws ebs modify-volume --volume-id vol-1234567890 --iops 3500 --throughput 300

AWS PowerShell

The AWS PowerShell module also acts as a wrapper around the AWS APIs, using an object-oriented shell scripting interface. You can either install the AWS PowerShell module locally or use it inside of the AWS CloudShell environment, within the AWS Management Console. For now, we'll assume you're running PowerShell from CloudShell, where the AWS modules are pre-installed.

To modify an EBS volume IOPS, use the Edit-EC2Volume command, similar to the example command below.

Edit-EC2Volume -VolumeId vol-1234567890 -Iops 4200

To change both the IOPS and Throughput for an EBS volume, try this command.

Edit-EC2Volume -VolumeId vol-1234567890 -Iops 5300 -Throughput 275

If you'd like to modify multiple volumes in the same AWS region, you can iterate over an array of EBS volume IDs in PowerShell.

$VolumeList = @(
  'vol-12345'
  'vol-56789'
)

foreach ($Volume in $VolumeList) {
  Edit-EC2Volume -VolumeId $Volume -Iops 3200
}

If you want to modify EBS volumes within a single AWS account, but across multiple AWS regions, you may want to use a HashTable (aka. Dictionary) data structure to define the volume IDs that belong to each region.

$VolumeList = @{
  'us-west-1' = @(
    'vol-7833'
    'vol-1234'
    'vol-3456'
  )
  'eu-central-1' = @(
    'vol-8383'
    'vol-4844'
    'vol-8228'
  )
}

foreach ($Region in $VolumeList.Keys) {
  foreach ($Volume in $VolumeList.$Region) {
    Edit-EC2Volume -Region $Region -VolumeId $Volume -Throughput 340
  }
}

Conclusion

For more advanced implementations of cost optimization at your company, please reach out to us at StratusGrid and ask about our success-based cost optimization program! Our cloud engineers are equipped to perform in-depth cost analysis of your cloud environment, make recommendations, and implement the changes in accordance with your organization's change management policies.