AWS Kinesis
Ingest and process real-time data at scale
Overview
Analytics has become an integral part of any modern system. Amazon Kinesis data streams enable you to ingest and process huge data and perform analysis in real time. With Kinesis Data Firehose, this data can be transformed and/or loaded in data stores such as S3, Redshift, OpenSearch, Splunk, etc. The data can also be queried and analyzed in real time with Apache Flink or SQL using Amazon Kinesis Data Analytics.
Key Learnings and Best Practices
Security
- Start with the basics. Use the principle of least privilege while granting access to the Kinesis data stream or data analytics resources.
- Do not use AWS credentials in code for accessing Kinesis resources. Instead, use IAM roles for accessing Kinesis resources from the producer and client applications.
- Ensure that you protect data by enabling server-side encryption using AWS KMS for data at rest. Note that data in transit is encrypted by default in Kinesis using HTTPS endpoints.
- Use VPC endpoints to connect Kinesis resources from the VPC. This will ensure that the traffic will remain within the AWS network during IBM to AWS Migration.
Scaling and Performance
- Consider using amazon-kinesis-scaling-utils from AWS labs for scaling Kinesis streams. This utility can be configured to scale up or down Kinesis streams based on key metrics captured in CloudWatch.
- Configure scaling for reading and writing separately as per the need.
- Set parallelism settings for each operator in your application’s code such that they do not starve for system resources.
- Consider configuring enhanced fan-out for scaling lambda consumers to improve the read performance. The SubscribeToShard API opens an HTTP/2 connection between the consumer and the shard.
Monitoring
- Kinesis data streams are integrated with CloudWatch. Enable enhanced monitoring for critical applications to ensure you have sufficient information logged for shards in CloudWatch. Note that the basic monitoring will capture only stream-level metrics.
- Set up alarms on key metrics such as Get Records. IteratorAgeMilliseconds. These metrics will reveal if the workers are lagging on the stream.
- For monitoring the provisioned throughput usage, set up alarms on ReadProvisionedThroughputExceeded and WriteProvisionedThroughputExceeded metrics.
- Monitor heapMemoryUtilization Metrics. This usually spikes due to a lack of CPU. In case of a spike, retune your scaling configurations to meet the needs.
- Kinesis is integrated with CloudTrail. API calls are logged to CloudTrail with S3 as the storage.
Cost optimization
- Consider using AWS spot instances to process your stream.
- Leverage Parallelization Factor while processing streams using AWS Lambda. Generally this will be more cost-effective than raising several shares on the Kinesis.
- Compress and aggregate smaller messages into one to reduce cost. You may use Amazon Kinesis Producer Library (KPL) for compression and aggregation.
Our Experience
IoT Platform for a leading US-based HVAC and refrigeration solutions provider
The client serves HVAC and refrigeration solutions for buildings, industries, fleets, and much more to create applications that monitor and control each of these sets of customers. This platform exposes common functionalities like ingestion pipeline, authentication, data storage, and data retrieval has been built. IoT applications can be built on this common platform which reduces cost and time to market.
IBM platform replacement and centralized logging framework for a leading American electricity distribution company
American electricity distribution company wanted to modernize their legacy IBM stack applications to AWS serverless and re-platform to cloud native application for reducing the licensing and operational cost of IBM stack (IBM DataPower, IBM MQ, IBM BPM) platform because of its high licensing, hardware, and maintenance cost. All the logs captured in the CloudWatch are moved to a centralized logging framework to ingest logs from multiple AWS accounts to a centralized account with the OpenSearch visualization dashboard through the IBM replacement platform.
LTI’s Service Offering for Kinesis
1. Consulting
Our consulting service offering focuses on application assessment using the LTI Infinity platform and domain-led app transformation to build the right disposition strategy, cloud migration roadmap, and target cloud architecture using serverless architecture patterns. LTI has extensive consulting capabilities in implementing cloud-native applications, IoT, migration, and modernization solutions using streaming services such as Kinesis, AWS IOT for multiple customers in the energy, insurance, banking, and manufacturing sector.2. Streaming application development
LTI has helped customers in the manufacturing sector transform connected solutions using AWS streaming services such as Kinesis, IOT, Lambda, DynamoDB, AppSync. These solutions are handling heavy data streaming traffic coming from IOT devices mounted on the manufacturing plants.3. Large-scale database migration
LTI has implemented large-scale heterogeneous database migration solutions for migrating on-premises Oracle data to AWS DynamoDB. The incremental real-time CDC data is ingested through Kinesis and processed to the array of Lambda functions.4. Observability for serverless
LTI has implemented an observability platform for monitoring the AWS cloud-native service and applications. Provides high visibility to developers and operational analysts to monitor entire application/service integration systems. This observability solution has used Kinesis for ingesting application logs from multiple AWS Cloud PaaS services across different AWS accounts.Conclusion
Customers are building connected solutions to monitor and get real-time insights to take appropriate decisions. Media customers are building digital media supply chains to process a large amount of media content. AWS Kinesis services help to provide scalability and better performance while processing a large amount of data. However, the solution should adhere to AWS Well Architect principles and best practices to ensure high security and lower the cost of implementation.
