Kinesis Best Practices: Cost, Security, and Performance

Understanding Amazon Kinesis: A Streaming Data Platform

Amazon Kinesis Data Streams is a fully managed, real-time streaming service designed to ingest, buffer, and process terabytes of data per second from thousands of sources. Data is distributed across shards, each providing a fixed capacity of 1 MB/second write throughput and 2 MB/second read throughput. Producers (applications, IoT devices, log forwarders) push records into the stream, while consumers (Lambda functions, KCL workers, analytics services) pull and process those records in near real time.

Building production-grade streaming pipelines on Kinesis demands attention to three critical pillars: cost efficiency, security, and performance. Misconfigured shard counts lead to overpaying or throttling. Weak access controls expose sensitive data. Suboptimal producer or consumer patterns create backpressure and stale data. This tutorial distills actionable best practices for each pillar, complete with code snippets you can immediately apply.

Cost Optimization Best Practices

Right-Sizing Shards

Shards are the fundamental throughput units and the primary driver of Kinesis cost. You pay per shard-hour, regardless of utilization. The goal is to provision just enough shards to handle your peak workload without waste. Each shard provides 1 MB/s write capacity and 2 MB/s read capacity (for standard consumers). Use the formula: Number of shards = MAX( (total write MB/s) / 1, (total read MB/s) / 2 ), rounding up.

Regularly monitor WriteProvisionedThroughputExceeded and ReadProvisionedThroughputExceeded CloudWatch metrics. If throttling occurs, scale up. If average utilization stays below 50% for extended periods, scale down.

# Update shard count using AWS CLI
aws kinesis update-shard-count \
  --stream-name my-data-stream \
  --target-shard-count 6 \
  --scaling-type UNIFORM_SCALING

# Check current shard count
aws kinesis describe-stream-summary \
  --stream-name my-data-stream \
  --query 'StreamDescriptionSummary.OpenShardCount'

Choosing Between Standard Consumers and Enhanced Fan-Out

Standard consumers share the per-shard 2 MB/s read capacity across all consumers. If you have multiple downstream applications reading from the same stream, contention can force you to provision extra shards just to satisfy read throughput. Enhanced Fan-Out (EFO) provides each registered consumer with its own 2 MB/s per shard, eliminating contention. However, EFO charges per GB of data retrieved, while standard consumers do not incur additional retrieval charges beyond the shard-hour cost.

• Use standard consumers for low-throughput, single-consumer scenarios or development environments.
• Use Enhanced Fan-Out when you have multiple high-throughput consumers that would otherwise force shard over-provisioning.

EFO is particularly cost-effective when read throughput demand exceeds write throughput demand. You avoid paying for extra shards that would sit idle on writes.

Implementing Auto-Scaling

Manual shard adjustments are reactive. Automate scaling using CloudWatch alarms and AWS Lambda or a scheduled process. The typical pattern: trigger a scale-up alarm when throttling exceeds a threshold for a sustained period (e.g., 5 minutes), and a scale-down alarm when both write and read utilization fall below a safe threshold.

# Example CloudFormation snippet for an auto-scaling Lambda function
  KinesisScalingFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Runtime: python3.9
      CodeUri: ./scaling/
      Environment:
        Variables:
          STREAM_NAME: my-data-stream
          MIN_SHARDS: 2
          MAX_SHARDS: 20

  ScaleUpAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: kinesis-write-throttling
      ComparisonOperator: GreaterThanThreshold
      EvaluationPeriods: 2
      MetricName: WriteProvisionedThroughputExceeded
      Namespace: AWS/Kinesis
      Period: 300
      Statistic: Sum
      Threshold: 10
      Dimensions:
        - Name: StreamName
          Value: my-data-stream

  ScaleDownAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: kinesis-low-utilization
      ComparisonOperator: LessThanThreshold
      EvaluationPeriods: 2
      MetricName: UserRecordsSum
      Namespace: AWS/Kinesis
      Period: 300
      Statistic: Sum
      Threshold: 5000
      Dimensions:
        - Name: StreamName
          Value: my-data-stream

The Lambda function calculates the required shard count based on recent throughput metrics and calls update-shard-count accordingly. Always set sensible MIN_SHARDS and MAX_SHARDS to prevent runaway costs.

Aggregating and Compressing Records

Kinesis charges per PUT operation (up to 1 KB per record ingested, rounded up) and per shard-hour. By aggregating many small records into a single larger payload, you reduce the number of PUT requests and thus the cost. The Kinesis Producer Library (KPL) handles aggregation and compression (up to Snappy or gzip) automatically, packing thousands of records into one Kinesis record unit.

// Java example: configure KPL for cost-efficient aggregation
KinesisProducerConfiguration config = new KinesisProducerConfiguration()
    .setRegion("us-east-1")
    .setAggregationEnabled(true)
    .setAggregationMaxCount(5000)      // max records per aggregated blob
    .setRecordMaxBufferedTime(2000)    // flush after 2 seconds
    .setCollectionMaxCount(1000)       // max items in collection queue
    .setThreadingModel(ThreadingModel.POOLED)
    .setThreadPoolSize(8);

KinesisProducer producer = new KinesisProducer(config);

// Add records – KPL will batch them automatically
for (int i = 0; i < 10000; i++) {
    producer.addUserRecord("my-stream",
        UUID.randomUUID().toString(),  // random partition key
        "{\"sensor\":\"temp\",\"value\":" + (20 + Math.random() * 10) + "}"
            .getBytes(StandardCharsets.UTF_8));
}

The KPL also provides built-in retries and exponential backoff, reducing the likelihood of throttling-related cost spikes from failed writes.

Setting Data Retention Wisely

Kinesis Data Streams retains records from 24 hours up to 365 days. Longer retention incurs an additional charge per GB-hour. Choose the shortest retention that satisfies your replay and auditing needs. For most real-time pipelines, 24 to 72 hours is sufficient. Use an extended retention only if you need to reprocess historical data.

# Set retention period (in hours)
aws kinesis increase-stream-retention-period \
  --stream-name my-data-stream \
  --retention-period-hours 48

Security Best Practices

Enabling Encryption at Rest

Kinesis Data Streams supports server-side encryption using AWS KMS customer master keys (CMKs). When enabled, data at rest on shard storage is encrypted, protecting against unauthorized disk access. You can use the AWS managed key (aws/kinesis) or a customer-managed CMK for finer control.

# Enable encryption on an existing stream
aws kinesis start-stream-encryption \
  --stream-name my-data-stream \
  --encryption-type KMS \
  --key-id alias/my-kinesis-key

# Verify encryption status
aws kinesis describe-stream \
  --stream-name my-data-stream \
  --query 'StreamDescription.EncryptionType'

Encrypting with a customer-managed key allows you to audit usage via CloudTrail and enforce key rotation policies. Ensure that producer and consumer IAM roles have the necessary kms:Decrypt and kms:GenerateDataKey permissions on the chosen key.

Encrypting Data In Transit

All AWS SDKs and the KCL/KPL communicate with Kinesis endpoints over HTTPS/TLS by default. Always verify that your client configurations enforce TLS 1.2 or higher. If you’re using custom HTTP clients, explicitly specify the secure endpoint.

// Ensure HTTPS endpoint in Java SDK (default is secure)
AmazonKinesisClientBuilder builder = AmazonKinesisClientBuilder.standard()
    .withRegion(Regions.US_EAST_1)
    .withCredentials(DefaultAWSCredentialsProviderChain.getInstance());
// No explicit endpoint override needed – HTTPS is the default.

Fine-Grained IAM Policies

Restrict access to streams using IAM policies that specify individual stream ARNs and allowed actions. Apply the principle of least privilege: producers should only be allowed PutRecord and PutRecords; consumers only GetRecords, GetShardIterator, etc. Use conditions to further restrict by source VPC, IP address, or tags.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:PutRecord",
        "kinesis:PutRecords"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:123456789012:stream/producer-stream",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": "10.0.0.0/8"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:GetRecords",
        "kinesis:GetShardIterator",
        "kinesis:DescribeStream",
        "kinesis:ListShards"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:123456789012:stream/consumer-stream"
    }
  ]
}

For Enhanced Fan-Out consumers, include kinesis:SubscribeToShard in the consumer policy.

Network Isolation with VPC Endpoints

To keep Kinesis traffic off the public internet, create an interface VPC endpoint for Kinesis Data Streams. This routes requests from your VPC directly to Kinesis via AWS PrivateLink, enhancing security and reducing latency. Attach a VPC endpoint policy to control which streams can be accessed through the endpoint.

# CloudFormation snippet for a Kinesis VPC endpoint
  KinesisEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcId: !Ref MyVPC
      ServiceName: com.amazonaws.us-east-1.kinesis-streams
      VpcEndpointType: Interface
      SubnetIds:
        - !Ref PrivateSubnet1
        - !Ref PrivateSubnet2
      SecurityGroupIds:
        - !Ref EndpointSecurityGroup
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal: "*"
            Action: "*"
            Resource: "arn:aws:kinesis:*:*:stream/*"

Once the endpoint is created, configure your Kinesis client to use the endpoint-specific DNS name or enable private DNS for the endpoint, so standard SDK calls resolve to the VPC endpoint automatically.

Audit Logging and Monitoring

Enable AWS CloudTrail to log all Kinesis management and data plane API calls (including PutRecord, GetRecords, UpdateShardCount). Stream these logs to CloudWatch Logs or S3, then set up alerts for anomalous actions such as bulk data retrieval, unauthorized access attempts, or configuration changes.

# CloudTrail data event logging for Kinesis (CloudFormation snippet)
  KinesisDataEventTrail:
    Type: AWS::CloudTrail::Trail
    Properties:
      TrailName: kinesis-data-events
      IsMultiRegionTrail: true
      IncludeGlobalServiceEvents: true
      EnableLogFileValidation: true
      EventSelectors:
        - IncludeManagementEvents: true
          ReadWriteType: All
          DataResources:
            - Type: AWS::Kinesis::Stream
              Values:
                - arn:aws:kinesis:*:*:stream/*
  KinesisLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /aws/cloudtrail/kinesis-data
      RetentionInDays: 90

Performance Best Practices

Optimizing Producer Throughput

The Kinesis Producer Library (KPL) is the go-to for high-throughput ingestion. It batches records, compresses payloads, and handles retries with backoff. Tune the following parameters based on your latency and throughput trade-offs:

• RecordMaxBufferedTime: Maximum time (ms) to buffer a record before flushing. Increase for better aggregation, decrease for lower end-to-end latency.
• AggregationMaxCount: Max records per aggregated blob. Higher values improve compression and reduce PUT count.
• ThreadPoolSize: Number of threads sending data to Kinesis. Match to your expected shard capacity (e.g., 1 thread per 2 shards).

// High-throughput KPL configuration for a stream with 10 shards
KinesisProducerConfiguration config = new KinesisProducerConfiguration()
    .setRegion("us-east-1")
    .setAggregationEnabled(true)
    .setAggregationMaxCount(10000)
    .setRecordMaxBufferedTime(3000)      // 3 seconds buffer
    .setCollectionMaxCount(5000)
    .setThreadingModel(ThreadingModel.POOLED)
    .setThreadPoolSize(20)
    .setMaxConnections(20)
    .setRateLimitingEnabled(false);      // disable rate limiting when confident about capacity

KinesisProducer producer = new KinesisProducer(config);

For extremely high throughput (hundreds of MB/s), distribute producer instances across multiple EC2 instances or containers, and ensure they target different partition key ranges to avoid hot shards.

Choosing the Right Consumer Pattern

Kinesis offers several consumer options, each with distinct performance characteristics:

• Lambda with Enhanced Fan-Out: Lowest latency, scales automatically per shard. Ideal for event-driven architectures.
• Kinesis Client Library (KCL): Java/Python/Node.js library that manages shard leases and checkpoints. Good for custom stream processing with state.
• Spark Streaming / Flink: For heavy analytical processing with exactly-once semantics.

When using KCL, configure record fetching and processing carefully to avoid IteratorAge growth (lag).

// KCL worker configuration for multi-threaded processing
KinesisClientLibConfiguration config = new KinesisClientLibConfiguration(
    "order-processing-app",
    "order-stream",
    new DefaultAWSCredentialsProviderChain(),
    UUID.randomUUID().toString())
    .withMaxRecords(200)                     // fetch up to 200 records per call
    .withCallProcessRecordsEvenForEmptyRecordList(false)
    .withIdleTimeBetweenReadsInMillis(250)   // short idle to keep pace
    .withFailoverTimeMillis(30000)           // quick failover for lease reassignment
    .withShardConsumerDispatchThreadCount(4); // parallel processing within a shard

Worker worker = new Worker.Builder()
    .recordProcessorFactory(new OrderRecordProcessorFactory())
    .config(config)
    .build();

// In your record processor, checkpoint frequently after processing
public void processRecords(ProcessRecordsInput input) {
    input.getRecords().forEach(record -> {
        // process record
    });
    // checkpoint at intervals
    input.getCheckpointer().checkpoint();
}

Partition Key Design to Avoid Hot Shards

Kinesis maps partition keys (specified on each record) to shards via an MD5 hash. If many records share the same partition key, they all go to the same shard, creating a hot shard that throttles writes even when other shards are idle. To distribute records evenly:

• Use high-cardinality partition keys (e.g., device ID, user ID, order number).
• For aggregated data, combine multiple fields: deviceId + ":" + timestampBucket.
• Avoid timestamp-only keys unless you introduce randomness.

// Example: creating a well-distributed partition key
String partitionKey = sensorId + ":" + (System.currentTimeMillis() / 60000); // per-minute bucket
producer.addUserRecord("metrics-stream", partitionKey, jsonPayload.getBytes());

Monitor per-shard metrics (IncomingRecords, OutgoingRecords) in CloudWatch to detect imbalances and adjust your key strategy.

Error Handling and Retry Logic

Throttling exceptions (ProvisionedThroughputExceededException) are inevitable when traffic spikes. The KPL handles retries with exponential backoff and jitter automatically. For producers using the raw PutRecords API directly, implement your own retry with backoff and respect the ThrottledRecords response.

// Custom retry logic for PutRecords (Java SDK)
int retries = 0;
int maxRetries = 5;
while (retries < maxRetries) {
    try {
        PutRecordsResult result = kinesisClient.putRecords(request);
        if (result.getFailedRecordCount() == 0) break;
        // Resend failed records
        request = buildRequestFromFailedRecords(result);
        retries++;
        Thread.sleep((long) Math.pow(2, retries) * 100 + new Random().nextInt(50));
    } catch (ProvisionedThroughputExceededException e) {
        retries++;
        Thread.sleep((long) Math.pow(2, retries) * 100 + new Random().nextInt(50));
    }
}

Consumers must handle empty records gracefully and checkpoint only after successful processing to avoid replay storms.

Monitoring and Alerting for Performance

Key CloudWatch metrics to monitor:

• WriteProvisionedThroughputExceeded / ReadProvisionedThroughputExceeded: Indicates capacity overflow.
• IteratorAgeMilliseconds: The age of the last processed record. Rising values signal consumer lag.
• GetRecords.Success and GetRecords.Latency: Consumer health.
• PutRecord.Success and PutRecords.Latency: Producer health.

Set alarms on IteratorAge exceeding 60 seconds (for near-real-time) and on sustained throttling events.

# CloudWatch alarm for consumer lag
aws cloudwatch put-metric-alarm \
  --alarm-name kinesis-consumer-lag \
  --alarm-description "Alert when iterator age exceeds 60 seconds" \
  --namespace AWS/Kinesis \
  --metric-name IteratorAgeMilliseconds \
  --statistic Maximum \
  --period 60 \
  --evaluation-periods 3 \
  --threshold 60000 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=StreamName,Value=order-stream \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alert

Conclusion

Running Kinesis Data Streams efficiently in production requires a holistic approach. By right-sizing shards and leveraging aggregation, you keep costs proportional to actual throughput. Encryption, IAM, and VPC endpoints build a hardened security boundary without sacrificing ease of use. Tuning producers and consumers—whether via KPL, KCL, or Lambda—ensures your pipeline stays performant and resilient under load. Continuously monitor the metrics discussed above, and automate shard scaling to adapt as your data volumes evolve. Applying these best practices transforms Kinesis from a simple streaming service into a robust, cost-effective backbone for real-time data architectures.