Kinesis Security: IAM Policies and Network Security

Kinesis Security: IAM Policies and Network Security

Amazon Kinesis is a powerful suite of real-time data streaming services, but with great power comes great responsibility—especially when handling sensitive data streams. Securing your Kinesis streams involves two critical layers: IAM policies that govern who can access your streams and what they can do, and network security controls that restrict how traffic reaches your streams. This tutorial covers both domains in depth, providing practical guidance and code examples you can apply immediately.

What Is Kinesis Security?

Kinesis security is a multi-layered approach that combines AWS Identity and Access Management (IAM) policies, network-level controls, encryption, and monitoring to protect your streaming data infrastructure. At its core, it ensures that only authorized principals—whether they are IAM users, roles, applications running on EC2, or Lambda functions—can interact with your Kinesis streams, and that data in transit and at rest remains protected.

The security model for Kinesis operates on two fundamental axes:

• Identity-based policies – Attached to IAM users, groups, or roles, defining what actions those principals can perform on specific Kinesis resources.
• Resource-based policies – Though less common in Kinesis compared to services like SQS or S3, some Kinesis features (like cross-account access via stream-level policies) allow you to define permissions directly on the stream resource itself.
• Network-based controls – VPC endpoints, security groups, and network ACLs that restrict the network path through which Kinesis API calls travel.

Why Kinesis Security Matters

Streaming data often carries sensitive, real-time information—financial transactions, user activity logs, IoT sensor readings, or application telemetry. A misconfigured Kinesis stream could lead to:

• Data exfiltration – An unauthorized party reading your stream and siphoning sensitive data.
• Data injection attacks – Malicious actors writing fabricated records into your stream, poisoning downstream analytics or triggering false alerts.
• Resource abuse – Unauthorized users calling APIs like IncreaseStreamRetentionPeriod or SplitShard, driving up costs or degrading performance.
• Compliance violations – Failing audits under GDPR, HIPAA, PCI DSS, or SOC 2 due to insufficient access controls and network exposure.

Properly implemented IAM policies and network security controls mitigate these risks by enforcing the principle of least privilege and keeping traffic within trusted boundaries.

IAM Policies for Kinesis

Understanding the Kinesis IAM Policy Structure

An IAM policy for Kinesis follows the standard AWS policy document format: a JSON document containing an Effect, Action, Resource, and optional Condition block. The key elements specific to Kinesis are the action names and the resource ARN format.

The ARN for a Kinesis Data Stream follows this pattern:

arn:aws:kinesis:<region>:<account-id>:stream/<stream-name>

For Kinesis Data Firehose delivery streams, the ARN is:

arn:aws:firehose:<region>:<account-id>:deliverystream/<delivery-stream-name>

For Kinesis Data Analytics applications:

arn:aws:kinesisanalytics:<region>:<account-id>:application/<application-id>

Common IAM Policy Examples

1. Granting Read-Only Access to a Specific Stream

This policy allows a principal to read records, describe the stream, and list shards—but not modify, delete, or write to the stream.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:DescribeStreamSummary",
        "kinesis:GetRecords",
        "kinesis:GetShardIterator",
        "kinesis:ListShards",
        "kinesis:SubscribeToShard"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:123456789012:stream/my-financial-data"
    }
  ]
}

2. Granting Producer (Write) Access

Producers need the ability to put records onto the stream. This policy grants only the necessary write actions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:PutRecord",
        "kinesis:PutRecords",
        "kinesis:DescribeStream",
        "kinesis:DescribeStreamSummary"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:123456789012:stream/my-financial-data"
    }
  ]
}

Note that DescribeStream is included because most Kinesis client libraries call it to determine shard topology before writing. Without it, many SDKs will fail during initialization.

3. Full Administrative Access to a Stream

For stream administrators who need to manage shard splitting, retention periods, and deletion, use a broader action set.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:*"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:123456789012:stream/my-financial-data"
    }
  ]
}

While using kinesis:* is convenient, it's better practice to explicitly list the admin actions you actually need, such as:

"Action": [
  "kinesis:CreateStream",
  "kinesis:DeleteStream",
  "kinesis:UpdateShardCount",
  "kinesis:IncreaseStreamRetentionPeriod",
  "kinesis:DecreaseStreamRetentionPeriod",
  "kinesis:DescribeStream",
  "kinesis:DescribeStreamSummary",
  "kinesis:ListShards",
  "kinesis:GetRecords",
  "kinesis:PutRecord",
  "kinesis:PutRecords",
  "kinesis:StartStreamEncryption",
  "kinesis:StopStreamEncryption"
]

4. Cross-Account Access Using Stream-Level Policies

Kinesis supports resource-based policies (also called stream-level policies) that allow you to grant access to IAM principals from other AWS accounts. This is powerful for sharing streams across organizational boundaries without creating IAM roles in the owning account.

Here's how to add a resource-based policy to a Kinesis stream via the AWS CLI, granting write access to a role in account 987654321098:

aws kinesis put-resource-policy \
  --resource-arn arn:aws:kinesis:us-east-1:123456789012:stream/my-financial-data \
  --policy '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "AWS": "arn:aws:iam::987654321098:role/cross-account-producer"
        },
        "Action": [
          "kinesis:PutRecord",
          "kinesis:PutRecords",
          "kinesis:DescribeStream"
        ],
        "Resource": "arn:aws:kinesis:us-east-1:123456789012:stream/my-financial-data"
      }
    ]
  }'

The external account also needs an identity-based policy that allows the same actions on this stream ARN, creating a two-way trust relationship.

Using Condition Keys to Tighten Security

IAM condition keys add an extra layer of contextual security. Kinesis supports several useful condition keys:

• aws:SourceIp – Restrict access to specific IP ranges (e.g., your corporate VPN CIDR).
• aws:SourceVpc or aws:SourceVpce – Restrict access to requests originating from a specific VPC or VPC endpoint.
• aws:RequestedRegion – Ensure API calls come from a specific AWS region.
• kinesis:RequestedStreamName – Restrict the stream name pattern that can be used in a CreateStream call.
• aws:PrincipalTag – Enforce attribute-based access control based on tags on the principal.

Example: IP-Restricted Producer Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:PutRecord",
        "kinesis:PutRecords",
        "kinesis:DescribeStream"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:123456789012:stream/my-financial-data",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "10.0.0.0/8",
            "172.16.0.0/12"
          ]
        }
      }
    }
  ]
}

Example: VPC Endpoint Restriction

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": [
        "kinesis:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:SourceVpce": "vpce-0a1b2c3d4e5f6g7h8"
        }
      }
    }
  ]
}

This deny policy, when attached to a role or user, blocks any Kinesis API call that does not originate through the specified VPC endpoint, effectively forcing all Kinesis traffic through your private network path.

Best Practices for Kinesis IAM Policies

• Apply least privilege rigorously – Never use kinesis:* in production unless absolutely necessary. Split policies by role: producers get write actions, consumers get read actions, administrators get management actions.
• Use separate streams for separate sensitivity levels – If you handle PII and non-PII data, put them on different streams and apply different IAM policies to each.
• Combine identity and resource policies judiciously – For cross-account access, ensure both sides have matching policies to avoid one-sided trusts that silently break.
• Audit with IAM Access Analyzer – Periodically run IAM Access Analyzer on your Kinesis streams to detect unintended external access.
• Tag your streams and use aws:ResourceTag conditions – This enables scalable, tag-based policy management across many streams.

Network Security for Kinesis

Why Network-Level Controls Are Essential

IAM policies control who can access your streams. Network security controls how that access reaches the Kinesis service. Even with perfect IAM policies, if your applications are making API calls over the public internet, those calls traverse potentially untrusted networks. Network controls help you:

• Eliminate internet exposure – Keep Kinesis traffic entirely within the AWS backbone.
• Meet compliance requirements – Many regulatory frameworks mandate that data not traverse the public internet.
• Reduce latency – VPC endpoints often provide lower, more predictable latency than internet-routed traffic.
• Simplify firewall rules – Instead of managing outbound internet proxy rules, you route traffic through a VPC endpoint.

VPC Endpoints for Kinesis (PrivateLink)

An interface VPC endpoint for Kinesis (powered by AWS PrivateLink) creates an elastic network interface (ENI) in your subnet with a private IP address. Applications in your VPC communicate with Kinesis through this ENI, and traffic never leaves the AWS network.

Step-by-Step: Creating a VPC Endpoint for Kinesis Data Streams

1. Identify your VPC, subnets, and security groups. You'll need at least one subnet in each Availability Zone where your consumers or producers run, and a security group that allows outbound traffic to the Kinesis service.

2. Create the endpoint using the AWS CLI:

aws ec2 create-vpc-endpoint \
  --vpc-id vpc-0a1b2c3d4e5f6g7h8 \
  --service-name com.amazonaws.us-east-1.kinesis-streams \
  --vpc-endpoint-type Interface \
  --subnet-ids subnet-0x1a2b3c subnet-0x4d5e6f \
  --security-group-ids sg-0a1b2c3d4e5f6g7h8 \
  --tag-specifications 'ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=kinesis-vpce}]'

The service name for Kinesis Data Streams is com.amazonaws.<region>.kinesis-streams. For Kinesis Data Firehose, use com.amazonaws.<region>.kinesis-firehose. For Kinesis Data Analytics, use com.amazonaws.<region>.kinesis-analytics.

3. Enable private DNS for the endpoint. This is crucial—it allows your applications to resolve the public Kinesis API endpoint (kinesis.us-east-1.amazonaws.com) to the private IPs of the VPC endpoint automatically, requiring zero code changes.

aws ec2 modify-vpc-endpoint \
  --vpc-endpoint-id vpce-0a1b2c3d4e5f6g7h8 \
  --private-dns-enabled

4. Verify connectivity. From an EC2 instance in the same VPC, test that DNS resolution points to the VPC endpoint IPs:

nslookup kinesis.us-east-1.amazonaws.com
# Should return private IPs in your VPC CIDR, not public IPs

aws kinesis list-streams --region us-east-1 --endpoint-url https://kinesis.us-east-1.amazonaws.com
# Should succeed without traversing the internet

Security Groups for VPC Endpoints

The security group attached to the VPC endpoint ENIs controls inbound traffic to the endpoint. However, since interface endpoints are initiated by your applications (outbound from the VPC perspective), the security group primarily needs rules that allow return traffic. A typical configuration:

• Inbound rules: Allow TCP on port 443 (HTTPS) from the CIDR of your VPC, or from the specific security groups of your producer/consumer instances.
• Outbound rules: Typically left open (all traffic allowed) since the endpoint is making outbound calls to the Kinesis service backbone.

# Create a security group for the VPC endpoint
aws ec2 create-security-group \
  --group-name kinesis-vpce-sg \
  --description "Security group for Kinesis VPC endpoint" \
  --vpc-id vpc-0a1b2c3d4e5f6g7h8

# Add inbound rule allowing HTTPS from the VPC CIDR
aws ec2 authorize-security-group-ingress \
  --group-id sg-0x9a8b7c \
  --protocol tcp \
  --port 443 \
  --cidr 10.0.0.0/16

End-to-End Architecture: VPC Endpoint + IAM Policy

The most robust security posture combines both layers. Here's a complete example that ensures:

• Only traffic coming through the VPC endpoint can call Kinesis APIs.
• Only specific IAM roles have the necessary permissions.

Step 1: Attach this IAM policy to your producer role (the role assumed by your application):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:PutRecord",
        "kinesis:PutRecords",
        "kinesis:DescribeStream",
        "kinesis:DescribeStreamSummary"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:123456789012:stream/app-logs-stream"
    },
    {
      "Effect": "Deny",
      "Action": [
        "kinesis:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEqualsIfExists": {
          "aws:SourceVpce": "vpce-0a1b2c3d4e5f6g7h8"
        }
      }
    }
  ]
}

The Deny statement with StringNotEqualsIfExists ensures that if a request comes through a VPC endpoint, it must be the specific one you created. If the request doesn't use any VPC endpoint (e.g., comes over the internet), the condition evaluates to true and the action is denied.

Step 2: Ensure your application code uses the regional endpoint. With private DNS enabled, no code changes are needed—the AWS SDK automatically resolves to the VPC endpoint IPs.

Here's a Python producer example using boto3 that works seamlessly with the VPC endpoint:

import boto3
import json
import time

# No special endpoint configuration needed—private DNS handles routing
kinesis_client = boto3.client('kinesis', region_name='us-east-1')

stream_name = 'app-logs-stream'

def put_log_record(log_entry):
    response = kinesis_client.put_record(
        StreamName=stream_name,
        Data=json.dumps(log_entry).encode('utf-8'),
        PartitionKey=log_entry.get('user_id', 'default')
    )
    return response['ShardId'], response['SequenceNumber']

# Example usage
log = {
    'user_id': 'user123',
    'action': 'login',
    'timestamp': int(time.time() * 1000),
    'ip_address': '192.168.1.100'
}

shard_id, seq_num = put_log_record(log)
print(f"Record sent to shard {shard_id}, sequence number: {seq_num}")

Network ACLs and Additional Layers

While security groups operate at the ENI level, Network ACLs (NACLs) provide subnet-level stateless filtering. For defense-in-depth, you can configure NACLs on the subnets hosting your VPC endpoint to further restrict traffic. However, NACLs are stateless—you must explicitly allow both inbound and outbound ephemeral ports. For most Kinesis deployments, security groups provide sufficient control, and NACLs add operational complexity without proportional security benefit.

Monitoring Network Traffic to Kinesis

To verify that your Kinesis traffic is indeed routing through the VPC endpoint, use VPC Flow Logs and CloudTrail.

• VPC Flow Logs – Capture traffic metadata at the ENI or subnet level. Look for flows to the Kinesis private IPs rather than public IPs.
• CloudTrail – Kinesis data plane operations (PutRecord, GetRecords) are logged in CloudTrail if you enable data event logging. The sourceIPAddress field will show the private IP of the VPC endpoint ENI when traffic routes correctly.

Enable CloudTrail data events for Kinesis:

aws cloudtrail create-trail \
  --name kinesis-data-trail \
  --s3-bucket-name my-cloudtrail-bucket

aws cloudtrail put-event-selectors \
  --trail-name kinesis-data-trail \
  --event-selectors '[{
    "ReadWriteType": "All",
    "IncludeManagementEvents": true,
    "DataResources": [{
      "Type": "AWS::Kinesis::Stream",
      "Values": ["arn:aws:kinesis:us-east-1:123456789012:stream/app-logs-stream"]
    }]
  }]'

Best Practices for Network Security

• Always use VPC endpoints in production – For any workload handling sensitive data, route Kinesis traffic through interface endpoints. It's a straightforward setup with enormous security benefits.
• Enable private DNS – This eliminates the need for custom endpoint URLs in your code, reducing the risk of misconfiguration.
• Deploy endpoints across multiple Availability Zones – For high availability, create endpoint ENIs in at least two AZs within your VPC.
• Restrict security group ingress to known CIDRs or security groups – Don't leave your VPC endpoint security group open to 0.0.0.0/0. Only allow traffic from the specific subnets or security groups that host your legitimate producers and consumers.
• Combine with IAM aws:SourceVpce conditions – As shown above, this creates a dual enforcement point: the network path is physically constrained, and the IAM policy logically enforces the constraint.
• Monitor and alert on unexpected traffic patterns – Use CloudTrail and VPC Flow Logs to detect anomalies, such as Kinesis API calls originating from unexpected IP addresses.

Conclusion

Securing Amazon Kinesis requires a thoughtful combination of IAM policies and network controls. IAM policies define the who and what—which principals can perform which actions on which streams. Network controls, centered around VPC endpoints and security groups, define the how—ensuring traffic stays within the AWS backbone and only originates from trusted network segments.

By applying least-privilege IAM policies, leveraging condition keys like aws:SourceVpce and aws:SourceIp, deploying interface VPC endpoints with private DNS, and layering on CloudTrail monitoring, you build a robust defense-in-depth posture for your streaming data infrastructure. This approach not only protects sensitive data but also aligns with compliance requirements and reduces the attack surface of your real-time data pipelines. Start with the code examples in this tutorial, adapt them to your environment, and make Kinesis security an integral part of your streaming architecture from day one.