--- title: "AWS Kinesis: How to Overcome Batch Requests & Poison Pills" description: Learn how to effectively manage batch requests and poison pills in Amazon Kinesis. StratusGrid provides expert advice for seamless data streaming. image: https://stratusgrid.com/hubfs/Amazon%20Kinesis%20How%20to%20Overcome%20Batch%20Requests%20%26%20Poison%20Pills%201.webp --- # Amazon Kinesis: How to Overcome Batch Requests & Poison Pills Learn how to effectively manage batch requests and poison pills in Amazon Kinesis. StratusGrid provides expert advice for seamless data streaming. [ Matt Barlow ](https://stratusgrid.com/blog/author/matt-barlow) Dec 13, 2023 #### Other blogs you may be interested in: [ How to Automate AWS with Backstage ](https://stratusgrid.com/blog/how-to-automate-aws-with-backstage) [ Multiaccount Pipeline Module on Terraform ](https://stratusgrid.com/blog/aws-multiaccount-pipeline-terraform-module) [ How AWS Stabilization Can Help Enhance Performance and Reduce Costs ](https://stratusgrid.com/blog/how-aws-stabilization-can-boost-performance-and-reduce-costs) ### Share this article [mailto:?subject=Check%20out%20https://stratusgrid.com/blog/aws-kinesis-how-to-overcome-batch-requests-poison-pills&utm_medium=social&utm_source=email%20&body=Check%20out%20https://stratusgrid.com/blog/aws-kinesis-how-to-overcome-batch-requests-poison-pills&utm_medium=social&utm_source=email](mailto:?subject=Check%20out%20https://stratusgrid.com/blog/aws-kinesis-how-to-overcome-batch-requests-poison-pills&utm_medium=social&utm_source=email%20&body=Check%20out%20https://stratusgrid.com/blog/aws-kinesis-how-to-overcome-batch-requests-poison-pills&utm_medium=social&utm_source=email) ### Subscribe ### Subscribe ![Amazon Kinesis: How to Overcome Batch Requests & Poison Pills](https://stratusgrid.com/hubfs/Amazon%20Kinesis%20How%20to%20Overcome%20Batch%20Requests%20%26%20Poison%20Pills%201.webp) AWS Kinesis: How to Overcome Batch Requests & Poison Pills 5:14 The talk ***Serverless data streaming: Amazon Kinesis Data Streams and AWS Lambda*** by Anahit Pogosova was the first session I attended at re:Invent 2023. The presentation was loaded with practical information and real-world experience that anyone who works with Kinesis will need to understand. In this blog, I aim to distill some of the key takeaways and highlights to help you navigate the complexities of **Amazon Kinesis Data Streams and how to Overcome batch requests and poison pills**. ## **Amazon Kinesis responses to batch requests** One of the first points that struck me was the behavior of AWS Kinesis when handling batch requests. AWS Kinesis supports batching of records to optimize data transfer. However, when you send a batch request using the **PutRecords** API call, Amazon Kinesis doesn't guarantee that all records will be successfully ingested. It may ingest some, all, or none of them due to reasons such as throttling, internal errors, or exceeding the maximum allowed size. Here’s the interesting part. When a single record in a PutRecords request fails, rather than failing the whole request, Kinesis returns a successful HTTP 200 response with a **PutRecordsResult** object. This object includes an array of **PutRecordsResultEntry** items for each record. Each **PutRecordsResultEntry** has a **SequenceNumber** and a **ShardId**. If the record is not successfully ingested, the **ErrorCode** and **ErrorMessage** fields are also included. ![Amazon Kinesis How to Overcome Batch Requests & Poison Pills 2](https://stratusgrid.com/hs-fs/hubfs/Amazon%20Kinesis%20How%20to%20Overcome%20Batch%20Requests%20%26%20Poison%20Pills%202.webp?width=660&height=627&name=Amazon%20Kinesis%20How%20to%20Overcome%20Batch%20Requests%20%26%20Poison%20Pills%202.webp) ### **The developer's dilemma** This approach can be counterintuitive as developers may expect an all-or-nothing behavior where either all records are processed successfully or the whole request fails. Not handling this appropriately can lead to data loss if failed records are not retried for ingestion. Therefore, it's crucial to inspect the **PutRecordsResult** response and handle any failed records appropriately. This usually includes implementing retry logic possibly with exponential backoff and jitter, especially in the case of throttling errors. ## **Tackling the Poison Pill challenge** A **Poison Pill** in AWS Kinesis refers to a record that cannot be processed successfully due to its size, data type, or other characteristics. Kinesis Data Streams has a record size limit of 1MB. If a record exceeds this size, a **ProvisionedThroughputExceededException** error is thrown and the record is not ingested into Amazon Kinesis. When consuming data from Kinesis using the Amazon Kinesis Client Library (KCL), if this poison pill record is encountered, it can halt the progress of the entire shard, as KCL processes records in order. This problem becomes exacerbated with default SDK settings. The default iterator used by KCL is **TRIM_HORIZON** which makes the application read all the data from the stream. So if there's a poison pill record, it'll keep trying to process it until the record expires, preventing the processing of subsequent records. ### [**Mitigation strategies**](https://stratusgrid.com/blog/aws-mitigations-against-apache-log4j-vulnerability) The poison pill failure mode can be mitigated by the use of **destinations**, **event source failures**, and **bisect batch-on error**. Destinations are alternate locations for your records. When AWS Kinesis fails to process a record, instead of attempting to reprocess indefinitely, the record can be moved to a destination. This can be another Kinesis Stream, an Amazon S3 bucket, or a Lambda function. Having a separate location for failed records ensures that your stream continues processing the next records. When AWS Kinesis feeds data into Lambda (or another event source) and the invocation fails, Lambda will attempt to process the batch until it succeeds or the data expires. Starting from the AWS SDK v1.10.0, you can also configure retries on individual records to handle event source failures. If a record results in an error, it can be retried before moving to the next record. Bisect batch on error is another method to handle poison pills. If a batch contains a bad record, the function will throw an error. If you enable bisect on error, Kinesis splits the payload in two and retries. ## **Transform Your Amazon Kinesis Projects with Expert Guidance** The insights from Anahit Pogosova's session at re:Invent 2023 are invaluable for anyone looking to integrate Kinesis Data Streams into their applications. Understanding how Amazon Kinesis responds to failed records in batch requests, how to mitigate poison pills, and the limitations of AWS Kinesis On-Demand is vital for those who plan to integrate Kinesis Data Streams with their applications. For more excellent information[watch the full video of the presentation](https://www.youtube.com/watch?v=HjoQllOWRmM). At StratusGrid, we specialize in providing solutions and expert guidance to help you maximize the potential of your Amazon Kinesis projects. Whether you're grappling with batch request nuances, poison pill challenges, or seeking to optimize your data streaming architecture, our team of AWS experts is ready to assist. [Contact StratusGrid today](https://stratusgrid.com/book-a-consultation) and let's work together to turn your data streaming challenges into successes. BONUS: Download Your FinOps Guide to Effective Cloud Cost Optimization Here ⤵️ [![FinOps-Guide-Downloadable (2)](https://stratusgrid.com/hs-fs/hubfs/FinOps-Guide-Downloadable%20(2).webp?width=1307&height=448&name=FinOps-Guide-Downloadable%20(2).webp)](https://stratusgrid.com/free-finops-101-guide) ## Similar posts Modernization ### [How to Automate AWS with Backstage](https://stratusgrid.com/blog/how-to-automate-aws-with-backstage) Streamline your AWS cloud operations and enhance efficiency with our practical, easy-to-follow guide. Ideal for DevOps and cloud professionals! Matt Barlow Aug 29, 2023 Modernization ### [Multiaccount Pipeline Module on Terraform](https://stratusgrid.com/blog/aws-multiaccount-pipeline-terraform-module) Optimize CI/CD in multi-account AWS setups with StratusGrid's Pipeline module, ensuring efficient, secure, and scalable workflows. StratusGrid Jan 15, 2024 Stabilization ### [How AWS Stabilization Can Help Enhance Performance and Reduce Costs](https://stratusgrid.com/blog/how-aws-stabilization-can-boost-performance-and-reduce-costs) Discover StratusGrid's AWS stabilization strategies to boost performance and slash costs. Learn expert tips for efficient cloud management now! StratusGrid Dec 12, 2023 ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Matt Barlow", "url" : "https://stratusgrid.com/blog/author/matt-barlow" }, "dateModified" : "2024-09-03T15:50:03.816Z", "datePublished" : "2023-12-13T21:46:44.000Z", "headline" : "AWS Kinesis: How to Overcome Batch Requests & Poison Pills", "image" : [ "https://stratusgrid.com/hubfs/Amazon%20Kinesis%20How%20to%20Overcome%20Batch%20Requests%20&%20Poison%20Pills%201.webp" ], "mainEntityOfPage" : { "@id" : "https://stratusgrid.com/blog/aws-kinesis-how-to-overcome-batch-requests-poison-pills", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://stratusgrid.com/hubfs/Vertical%20Black.png" }, "name" : "StratusGrid" } } ``` ```json { "@context" : "https://schema.org/", "@type" : "Product", "aggregateRating" : { "@type" : "AggregateRating", "bestRating" : "5", "ratingCount" : "1", "ratingValue" : "5", "reviewCount" : "1", "worstRating" : "0" }, "brand" : { "@type" : "Brand", "name" : "StratusGrid" }, "description" : "StratusGrid, a premier AWS Consulting Partner, specializes in AWS cloud migration, modernization, stabilization, and cloud cost optimization services.", "image" : "https://stratusgrid.com/", "name" : "StratusGrid", "review" : { "@type" : "Review", "author" : { "@type" : "Person", "name" : "G2" }, "datePublished" : "2024-01-21", "name" : "G2 User", "publisher" : { "@type" : "Organization", "name" : "G2" }, "reviewBody" : "Great Customer Service and Extremely Knowledgable Team", "reviewRating" : { "@type" : "Rating", "bestRating" : "5", "ratingValue" : "5", "worstRating" : "0" } } } ```