Amazon Web Services#

AWS is a set of APIs in front of a very large amount of hardware. That sentence is not a simplification — it is the most useful frame you can hold, because it explains almost every behaviour that otherwise seems arbitrary. The console is a web application that calls those APIs. The CLI is a program that calls those APIs. Terraform, CDK, the mobile app and your own code are all clients of the same interface, authenticated the same way, logged the same way, rate-limited the same way. There is no privileged path.

A second frame is nearly as useful: AWS services come in two halves, and they fail independently. The control plane is the part that creates, modifies and describes resources — RunInstances, CreateBucket, UpdateService. The data plane is the part that does the work — the instance running, the object being read, the packet being forwarded. Data planes are engineered to be far more available than control planes, and they are designed to keep working when the control plane is impaired. This is why AWS advice so often says "do not depend on a control plane call in your recovery path": an Auto Scaling group that must call RunInstances during a large regional event may be waiting behind everyone else's calls, while the instances you already have carry on serving traffic perfectly.

1.1 What an API call actually looks like#

Every AWS request is HTTPS to a regional endpoint, signed with Signature Version 4. The signature is an HMAC over a canonical form of the request — method, path, query string, a selected set of headers, a hash of the body, and a timestamp — keyed by a value derived from your secret access key, the date, the Region and the service. Three consequences follow, and all three show up as real bugs:

• Your secret key is never transmitted. It is the HMAC key, not a password. This is why a network observer cannot replay your credentials, and why AWS can log every request without logging anything sensitive.
• The signature covers a timestamp, and requests more than fifteen minutes out of skew are rejected with SignatureDoesNotMatch or RequestTimeTooSkewed. A container with a broken clock cannot talk to AWS at all, and the error message does not mention time.
• The signature is scoped to a Region and service. Sending a request signed for us-east-1 to the eu-west-1 endpoint fails. Cross-Region access means creating a client for that Region, not adding a parameter.

POST / HTTP/1.1
Host: dynamodb.us-east-1.amazonaws.com
X-Amz-Date: 20260210T142233Z
X-Amz-Target: DynamoDB_20120810.GetItem
Authorization: AWS4-HMAC-SHA256
    Credential=ASIA.../20260210/us-east-1/dynamodb/aws4_request,
    SignedHeaders=host;x-amz-date;x-amz-target,
    Signature=8f2c...        ← HMAC over the canonical request
X-Amz-Security-Token: IQoJb3JpZ2luX2Vj...   ← present only for role sessions

{"TableName":"orders","Key":{"pk":{"S":"USER#42"}}}

1.2 ARNs, the universal address#

Every resource on AWS has an Amazon Resource Name, and you will read and write thousands of them. The format is regular, and knowing the fields by position makes IAM policies far less mysterious:

1.3 Quotas, throttling and eventual consistency in the control plane#

Every AWS API has a rate limit and every account has service quotas. Most quotas are adjustable through Service Quotas or a support ticket; some are hard. The two you will meet earliest are the number of VPCs per Region (5 by default), and Lambda concurrency (1,000 per Region by default, shared across every function in the account — which means one runaway function can starve all the others unless you set reserved concurrency).

Control planes are also frequently eventually consistent, even where data planes are not. Create an IAM role and immediately assume it and you may get a failure; create a security group and immediately reference it and you may get "does not exist". This is not a bug and it is not going to be fixed — it is a consequence of a globally distributed control plane. Infrastructure tools handle it with retries; your own automation must too.

AWS's physical layout is a three-level hierarchy plus an edge network, and every availability and latency decision you make is really a decision about which level to operate at.

2.1 Regions#

A Region is a named geographic area (us-east-1, eu-central-1, ap-southeast-2) containing at least three Availability Zones. Regions are isolated by design: separate API endpoints, separate control planes, separate capacity pools, separate physical infrastructure. Data does not leave a Region unless you configure a mechanism that moves it — cross-Region replication, a global table, a backup copy.

Two Regions have properties worth knowing. us-east-1 (N. Virginia) is the oldest and largest; it hosts the control planes of several global services, which is why an IAM change, a CloudFront distribution update, or a Route 53 record change involves us-east-1 regardless of where you live. It is also where new services land first and where capacity constraints bite last. us-west-2 (Oregon) is the usual second choice, being similarly complete and slightly cheaper for some services.

Choosing a Region comes down to four questions, in this order: where are the users (latency), where must the data live (regulation, data residency), is the service available there (new services roll out over months), and what does it cost (prices differ by 10–40% between Regions for the same instance).

2.2 Availability Zones#

An Availability Zone is one or more discrete datacentres with independent power, cooling and physical security, connected to the other AZs in the Region by dedicated, redundant, low-latency fibre. The design target is that a single event — a flood, a fire, a power failure, a transformer explosion — affects one AZ and no others. Inter-AZ latency is typically under two milliseconds, which is the number that makes synchronous replication practical.

The mapping between an AZ name and a physical AZ is randomised per account. Your us-east-1a and another account's us-east-1a are usually different datacentres, and AWS does this deliberately so that "the first one in the list" does not become everyone's default. The stable identifier is the AZ ID (use1-az1…), which is consistent across accounts and is what you must use when coordinating placement between accounts.

2.3 The edge network, Local Zones and Outposts#

Outside the Region hierarchy sits a much larger network of edge locations — over 600 points of presence, in far more cities than there are Regions. You do not run code on them in the general sense; they run CloudFront, Route 53, AWS Global Accelerator, AWS Shield, and the small edge compute runtimes (CloudFront Functions and Lambda@Edge). Their purpose is to terminate the user's connection close to them and carry the traffic onward over AWS's own backbone, which is faster and far more predictable than the public internet.

• Local Zones place compute, storage and networking in a metro area far from the parent Region, for single-digit-millisecond latency to that city — used for real-time gaming, media production and interactive workloads.
• Wavelength Zones do the same inside telecom providers' 5G networks.
• Outposts is AWS hardware installed in your datacentre, running the same APIs, for workloads that genuinely cannot leave the building.
• Global Accelerator gives you two static anycast IPs at the edge that route traffic over the AWS backbone to regional endpoints — the answer when you need a static IP and global routing and fast failover, which CloudFront and DNS cannot provide together.

An AWS account is the fundamental container: a namespace for resources, a boundary for IAM, a unit of billing, and — most importantly — the only truly hard isolation boundary AWS offers. Everything inside an account can potentially reach everything else inside it, subject to IAM. Nothing crosses an account boundary without an explicit grant on both sides. Multi-account is therefore not an enterprise nicety; it is the primary security control.

3.1 AWS Organizations#

Organizations groups accounts into a tree: a management account at the root, organizational units (OUs) below it, and member accounts in the OUs. It gives you three things: consolidated billing (volume discounts and Savings Plans apply across the whole organisation), centralised account creation, and service control policies.

3.2 Service control policies#

An SCP looks like an IAM policy and behaves completely differently in one respect: it never grants anything. It sets the maximum set of permissions available to principals in an account. The effective permission is the intersection of the SCP and the identity policy, so an account with AdministratorAccess and an SCP denying us-east-1 simply cannot use us-east-1. Two more properties matter: SCPs do not apply to the management account (a strong argument for keeping no workloads in it), and they do not apply to service-linked roles.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RegionFence",
      "Effect": "Deny",
      "NotAction": [
        "iam:*", "organizations:*", "route53:*", "cloudfront:*",
        "support:*", "sts:*", "budgets:*", "waf:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": { "aws:RequestedRegion": ["eu-west-1", "eu-central-1"] }
      }
    },
    {
      "Sid": "ProtectSecurityControls",
      "Effect": "Deny",
      "Action": [
        "cloudtrail:StopLogging", "cloudtrail:DeleteTrail",
        "guardduty:DeleteDetector", "config:DeleteConfigurationRecorder"
      ],
      "Resource": "*"
    },
    {
      "Sid": "RequireImdsV2",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "StringNotEquals": { "ec2:MetadataHttpTokens": "required" }
      }
    }
  ]
}

3.3 Landing zones and Control Tower#

A landing zone is the pre-built account structure a team starts from: OUs, SCPs, centralised CloudTrail and Config into a dedicated log-archive account, GuardDuty enabled organisation-wide, IAM Identity Center for human access, a network account owning the Transit Gateway, and a repeatable process for vending new accounts. AWS Control Tower builds and maintains one of these with guardrails you can extend; Account Factory for Terraform or a similar pipeline does the same if you prefer to own the code.

Three services make multi-account tolerable rather than tedious:

• IAM Identity Center — one login for humans, federated from your identity provider, granting time-limited role sessions in whichever accounts a user is entitled to. This is what replaces IAM users entirely.
• Resource Access Manager (RAM) — shares specific resources across accounts: subnets from a central network account, Transit Gateways, Route 53 Resolver rules, License Manager configurations. Sharing subnets means workload accounts get networking without each one owning a VPC.
• Delegated administration — lets a member account (not the management account) administer GuardDuty, Security Hub, Config and others organisation-wide, so the management account can stay locked down and unused.

IAM answers one question for every API call: is this principal permitted to perform this action on this resource under these conditions? Before the evaluation logic in the next section, you need the vocabulary of who can be asking.

4.1 The kinds of principal#

The root user is the account's owner and cannot be restricted by IAM policies at all (it can be restricted by SCPs in a member account). The correct handling is: enable MFA, remove any access keys, set a long random password, store it in a physical safe or a break-glass vault, and use it only for the operations that genuinely require it — closing the account, changing the account's root email or support plan, restoring an IAM configuration that has locked everyone out, and a small number of others.

4.2 Why IAM users are the wrong default#

An IAM user with an access key is a permanent credential sitting in a file. It does not expire. It does not know where it is being used from. When it leaks — into a git repository, a container image layer, a CI log, a screenshot, a Stack Overflow question — it stays valid until a human notices and revokes it. Public repositories are scanned continuously by attackers precisely because this is such a productive attack.

Every context in which you might reach for a user has a role-based alternative:

4.3 Where credentials physically come from on an instance#

An EC2 instance with an instance profile receives credentials from the Instance Metadata Service at the link-local address 169.254.169.254. The SDKs query it automatically, cache the result, and refresh before expiry. You should never need to call it yourself, but knowing it exists explains a class of security incidents: a server-side request forgery vulnerability in your application can be used to fetch that URL and steal the role's credentials.

IMDSv2 is the mitigation and should be mandatory. It requires a PUT to obtain a session token before any read, sets a low IP TTL so the response cannot be forwarded, and thereby defeats the simple SSRF-to-credentials pivot. Enforce it with an SCP as shown in section 3.

# IMDSv2: get a token first, then use it. This is what the SDKs do.
TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/

# Verify enforcement across the fleet - anything "optional" is a finding
aws ec2 describe-instances \
  --query 'Reservations[].Instances[?MetadataOptions.HttpTokens!=`required`].InstanceId' \
  --output text

This section is the single highest-value piece of AWS knowledge there is, because every access problem on the platform — and every access control you design — reduces to it. Run all four variants of the animation before reading on.

5.1 The seven policy types#

5.2 The evaluation order, precisely#

1. Deny sweep. Any explicit Deny in any applicable policy ends the evaluation. Final.
2. SCPs and RCPs. If an Organizations policy does not allow the action, deny.
3. Resource-based policy. An Allow here can be sufficient on its own for same-account access to most services, and is required for cross-account access.
4. Identity-based policy. An Allow matching action, resource and conditions.
5. Permissions boundary. If present, it must also allow the action.
6. Session policy. If present, it must also allow the action.
7. Otherwise, implicit deny.

The rule to carry: a request is allowed only if every applicable gate allows it and no gate denies it. Permissions are an intersection. This is why an admin-level identity policy is powerless against an SCP, and why granting more permissions never fixes an explicit deny.

Cross-account access requires two sides. The resource's policy must allow the external principal, and the external principal's identity policy must allow the action. One without the other fails. This symmetry is deliberate: it means no other account can grant itself access to your resources, and you cannot accidentally grant access that the other account's administrator has not also approved.

5.3 Conditions, where policies get interesting#

Without conditions a policy is a blunt on/off switch. With them you can express real security requirements. The condition keys worth memorising:

Attribute-based access control deserves a moment, because it is how you avoid a policy per team. Tag the role with team=analytics, tag the resources with team=analytics, and write one policy that says "allow if the tags match". Adding a new team then requires no policy change at all.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AbacOnlyYourTeamsResources",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::acme-data/*",
      "Condition": {
        "StringEquals": {
          "aws:PrincipalTag/team": "${aws:ResourceTag/team}"
        }
      }
    },
    {
      "Sid": "DenyAnythingNotOnTheOrgNetwork",
      "Effect": "Deny",
      "Action": "s3:*",
      "Resource": ["arn:aws:s3:::acme-data", "arn:aws:s3:::acme-data/*"],
      "Condition": {
        "StringNotEquals": { "aws:SourceVpce": "vpce-0abc123def456" },
        "Bool": { "aws:ViaAWSService": "false" }
      }
    }
  ]
}

5.4 A procedure for debugging AccessDenied#

1. aws sts get-caller-identity — confirm which principal is actually making the call. A surprising share of these end here.
2. Read the error carefully. AWS increasingly names the cause: "with an explicit deny in a service control policy", "with an explicit deny in a resource-based policy", "because no identity-based policy allows".
3. Run iam simulate-principal-policy for the exact action and resource ARN. It reports the decision and the matching statements.
4. If the simulator says allow but reality says deny, the cause is outside identity policies: SCP, resource policy, permissions boundary, VPC endpoint policy, or a KMS key policy.
5. Look in CloudTrail for the event. The errorCode and errorMessage are recorded, along with the exact principal ARN and the request parameters.
6. Check the ARN granularity — bucket versus object, table versus index, function versus alias.
7. Check whether the target is encrypted with a KMS key whose key policy does not include your principal. An AccessDenied from KMS reads exactly like one from S3.

A role is a permission set with a trust policy instead of credentials. The trust policy is a resource-based policy on the role that answers "who may assume me?" — and it is evaluated independently of the assumer's own permissions. Both sides must agree.

6.1 Trust policies by scenario#

// An AWS service assuming the role (Lambda execution role)
{ "Effect": "Allow",
  "Principal": { "Service": "lambda.amazonaws.com" },
  "Action": "sts:AssumeRole" }

// A third-party vendor. The ExternalId prevents the confused-deputy problem:
// without it, the vendor could be tricked into using its own role to access
// YOUR account on behalf of a different customer.
{ "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::999999999999:root" },
  "Action": "sts:AssumeRole",
  "Condition": { "StringEquals": { "sts:ExternalId": "acme-9f3c1a77-unique" } } }

// GitHub Actions via OIDC - no stored credentials at all.
// The sub condition is the security boundary: scope it to repo AND ref,
// or any workflow in any branch of that repo can deploy to production.
{ "Effect": "Allow",
  "Principal": { "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com" },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub": "repo:acme/api:ref:refs/heads/main"
    } } }

// Cross-account, restricted to one role and requiring MFA
{ "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111111111111:role/deployer" },
  "Action": "sts:AssumeRole",
  "Condition": { "Bool": { "aws:MultiFactorAuthPresent": "true" } } }

6.2 The STS operations#

Role chaining — assuming role A, then using A to assume role B — works, but caps the session at one hour regardless of the roles' configured maximum. This catches long CI pipelines that chain roles and then fail an hour in with an expired token, and the failure is usually blamed on the wrong thing.

6.3 IAM Identity Center#

IAM Identity Center (formerly AWS SSO) is the answer for human access across a multi-account organisation. You connect an identity source (its own directory, Active Directory, or an external IdP like Okta or Entra ID), define permission sets — which are templates that become IAM roles in each account — and assign groups to (account, permission set) pairs. Users get a portal listing the accounts and roles available to them, and aws sso login gives the CLI a browser-based login with short-lived credentials.

# One-time configuration, then a browser login whenever the session expires
aws configure sso
# SSO start URL: https://d-1234567890.awsapps.com/start
# SSO region: eu-west-1
# then pick account + role, and name the profile

aws sso login --profile prod-admin
aws s3 ls --profile prod-admin

# ~/.aws/config - note there is no secret anywhere on this machine
# [profile prod-admin]
# sso_session = acme
# sso_account_id = 123456789012
# sso_role_name = AdministratorAccess
# region = eu-west-1

Every SDK resolves credentials through the same ordered default credential provider chain. Knowing the order explains most "it works on my machine" problems, because the first source that yields credentials wins and the later ones are never consulted.

1. Explicit parameters passed to the client constructor in code
2. Environment variables — AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN
3. The shared config and credentials files (~/.aws/config, ~/.aws/credentials), selected by AWS_PROFILE
4. SSO cached credentials, or a credential_process command
5. Web identity token (AWS_WEB_IDENTITY_TOKEN_FILE) — this is how EKS IRSA and GitHub Actions work
6. ECS or EKS container credentials, via AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
7. EC2 instance metadata (IMDSv2)

7.1 SDK behaviour worth configuring#

import boto3
from botocore.config import Config

# One client, module scope, reused for the process lifetime.
# Creating a client per request is the most common boto3 performance bug:
# it re-resolves credentials and re-parses the service model each time.
cfg = Config(
    region_name="eu-west-1",
    retries={"max_attempts": 5, "mode": "adaptive"},  # backoff + client-side rate limiting
    connect_timeout=3,
    read_timeout=10,
    max_pool_connections=50,      # raise for concurrent workloads
    user_agent_extra="acme-api/1.4.2",
)
s3 = boto3.client("s3", config=cfg)

# Paginators, not manual NextToken loops. Every List* API is paginated,
# and code that ignores that silently processes only the first page.
for page in s3.get_paginator("list_objects_v2").paginate(
    Bucket="acme-reports", Prefix="2026/"
):
    for obj in page.get("Contents", []):
        print(obj["Key"], obj["Size"])

# Waiters, not sleep loops
s3.get_waiter("object_exists").wait(Bucket="acme-reports", Key="2026/q1.csv")

import { S3Client, ListObjectsV2Command } from "@aws-sdk/client-s3";
import { paginateListObjectsV2 } from "@aws-sdk/client-s3";

// v3 is modular: import only the commands you use, so the bundle stays small.
const s3 = new S3Client({
  region: "eu-west-1",
  maxAttempts: 5,
  requestHandler: { connectionTimeout: 3000, requestTimeout: 10000 },
});

for await (const page of paginateListObjectsV2(
  { client: s3 },
  { Bucket: "acme-reports", Prefix: "2026/" }
)) {
  for (const obj of page.Contents ?? []) console.log(obj.Key, obj.Size);
}

# --query is JMESPath and runs client-side; --filters runs server-side.
# For large result sets, filter on the server or you pay to transfer it all.
aws ec2 describe-instances \
  --filters Name=tag:env,Values=prod Name=instance-state-name,Values=running \
  --query 'Reservations[].Instances[].{Id:InstanceId,Type:InstanceType,AZ:Placement.AvailabilityZone}' \
  --output table

# assume a role for a single command without touching your profile
eval $(aws sts assume-role --role-arn arn:aws:iam::123456789012:role/read-only \
  --role-session-name debug --query 'Credentials' --output json | \
  jq -r '"export AWS_ACCESS_KEY_ID=\(.AccessKeyId)
          export AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey)
          export AWS_SESSION_TOKEN=\(.SessionToken)"')

# or, far better, define it once in ~/.aws/config and let the CLI do it:
# [profile prod-ro]
# role_arn = arn:aws:iam::123456789012:role/read-only
# source_profile = default
# duration_seconds = 3600

7.2 Retries, and the storm they can cause#

Every SDK retries automatically, and the algorithm it uses is worth understanding because you will need to reimplement it for calls between your own services, where nobody has configured it for you. Run all three variants below.

boto3 offers three retry modes. legacy is the old default with few attempts; standard uses exponential backoff with full jitter and a consistent set of retryable error codes across services; adaptive adds a client-side rate limiter that slows the whole client down when it observes throttling, which is the right choice for bulk workloads that would otherwise keep a service saturated. The JavaScript SDK v3 uses the standard strategy by default and exposes maxAttempts and a retryStrategy.

Two rules cover the rest. Classify before retrying: throttling, 5xx, connection resets and timeouts are retryable; AccessDenied, ValidationException, 404 and conditional-check failures never are, and retrying them just burns your budget. Retry at one layer only, with a total time budget — three retries at each of five layers is 243 requests arriving at the service that is already struggling.

A Virtual Private Cloud is a software-defined network inside a Region. It is not a virtual switch and there is no real router — the whole thing is implemented in the hypervisor and the network fabric, which is why a route table change takes effect instantly across thousands of instances and why there is no device to run out of capacity. What you configure is a set of declarations that the fabric enforces.

8.1 Addressing#

A VPC has a primary IPv4 CIDR block between /16 (65,536 addresses) and /28 (16). You may add secondary CIDRs later, but you cannot change or shrink the primary one, and you cannot renumber a subnet. Use RFC 1918 space — 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 — and plan the whole organisation's allocation before you create the second VPC, because two VPCs with overlapping ranges can never be peered or attached to the same Transit Gateway. IP Address Manager (IPAM) exists to do this allocation for you and is worth adopting early.

Every subnet loses five addresses: the network address, the VPC router, the DNS resolver (base + 2), one reserved for future use, and the broadcast address. A /28 subnet therefore holds 11 usable addresses, not 16 — which matters when you are sizing subnets for Fargate tasks or Lambda ENIs, each of which consumes one.

8.2 Subnets and what makes one public#

A subnet lives in exactly one Availability Zone — that is the mechanism by which a regional VPC becomes tied to physical datacentres. There is no attribute called "public". A subnet is public if and only if its associated route table contains a route to an Internet Gateway. Change that route and the subnet's nature changes with it, silently and immediately.

An instance in a public subnet also needs a public IP to be reachable from the internet, because the IGW performs a one-to-one NAT between the private address and the public one. The instance itself never sees its public address; ip addr shows only the private one, which is why binding to a public IP inside the OS fails.

8.3 Route tables#

Each subnet is associated with exactly one route table (the VPC's main table by default). Routes are matched most specific first, so a /24 beats a /16 beats 0.0.0.0/0. Every table contains an immutable local route for the VPC's CIDR, which is why anything in the VPC can address anything else in it without configuration.

8.4 DNS inside a VPC#

Two VPC attributes control DNS and both must be on for most things to work. enableDnsSupport makes the Route 53 Resolver available at VPC base + 2 and at 169.254.169.253. enableDnsHostnames makes AWS assign DNS names to instances with public IPs. Interface VPC endpoints and private hosted zones both depend on these, and their absence produces failures that look like connectivity problems rather than name-resolution problems.

9.1 Internet and egress-only gateways#

The Internet Gateway is attached to the VPC, is horizontally scaled and redundant by construction, has no bandwidth limit of its own and costs nothing. It performs one-to-one NAT between private and public IPv4 addresses. For IPv6, an egress-only Internet Gateway provides the outbound-only equivalent, since IPv6 addresses are globally routable and need no address translation.

9.2 NAT Gateways, and why they dominate network bills#

A NAT Gateway is a managed, AZ-scoped device that lets private subnets initiate outbound connections. It scales to 100 Gbps, supports 55,000 simultaneous connections per unique destination, and is zonal — meaning that for real availability you deploy one per AZ and route each AZ's private subnets to their local one. Doing otherwise both creates a single point of failure and generates cross-AZ transfer charges on every byte.

The cost has two components: roughly $0.045 per hour per gateway, and roughly $0.045 per gigabyte processed. Three gateways running all month is about $100 before a single byte moves; a workload pulling 20 TB a month from S3 through NAT adds roughly $900 that a free gateway endpoint would remove entirely.

9.3 VPC endpoints and PrivateLink#

PrivateLink generalises the interface endpoint: a service provider puts a Network Load Balancer behind an endpoint service, and consumers create interface endpoints to it in their own VPCs. Traffic flows over the AWS network, unidirectionally initiated by the consumer, with no VPC peering, no route exchange, and no requirement that the two VPCs have non-overlapping CIDRs. For exposing an internal API to another team, another account, or a customer, it is a much smaller security surface than peering.

# Free, and removes most NAT charges for S3-heavy workloads
S3GatewayEndpoint:
  Type: AWS::EC2::VPCEndpoint
  Properties:
    VpcId: !Ref Vpc
    ServiceName: !Sub com.amazonaws.${AWS::Region}.s3
    VpcEndpointType: Gateway
    RouteTableIds: [!Ref PrivateRtA, !Ref PrivateRtB, !Ref PrivateRtC]
    PolicyDocument:                       # endpoint policies are a real control
      Statement:
        - Effect: Allow
          Principal: '*'
          Action: ['s3:GetObject', 's3:PutObject']
          Resource: 'arn:aws:s3:::acme-*/*'
          Condition:
            StringEquals:
              aws:PrincipalOrgID: o-abc123xyz   # nothing leaves the org

# Needed for private subnets with no NAT at all: SSM, ECR, logs, secrets
SsmEndpoint:
  Type: AWS::EC2::VPCEndpoint
  Properties:
    VpcId: !Ref Vpc
    ServiceName: !Sub com.amazonaws.${AWS::Region}.ssm
    VpcEndpointType: Interface
    PrivateDnsEnabled: true
    SubnetIds: [!Ref PrivateA, !Ref PrivateB, !Ref PrivateC]
    SecurityGroupIds: [!Ref EndpointSg]   # must allow 443 from the workload SG

10.1 Security groups#

A security group attaches to an elastic network interface, not to a subnet or an instance — which is why two instances on the same subnet can have entirely different rules, and why an instance with two ENIs can have two different postures. Its properties:

• Allow rules only. There is no deny. Anything not explicitly allowed is dropped silently, which is why a missing rule presents as a hang rather than a refusal.
• Stateful. A permitted inbound connection's response is automatically permitted outbound, and vice versa. You almost never need to write outbound rules.
• All rules are evaluated, in no particular order, and the union applies. There is no first-match precedence.
• Sources can be other security groups. This is the idiom that makes AWS networking maintainable.
• Multiple groups can attach to one ENI; their rules are unioned.

# The tiered pattern: each layer names the layer above it, never a CIDR.
# Instances can be replaced, scaled and re-IP'd and the rules keep working.
AlbSg:
  Type: AWS::EC2::SecurityGroup
  Properties:
    GroupDescription: public entry point
    VpcId: !Ref Vpc
    SecurityGroupIngress:
      - {IpProtocol: tcp, FromPort: 443, ToPort: 443, CidrIp: 0.0.0.0/0}

AppSg:
  Type: AWS::EC2::SecurityGroup
  Properties:
    GroupDescription: application tier
    VpcId: !Ref Vpc
    SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: 8080
        ToPort: 8080
        SourceSecurityGroupId: !Ref AlbSg      # only the ALB, by identity

DbSg:
  Type: AWS::EC2::SecurityGroup
  Properties:
    GroupDescription: database tier
    VpcId: !Ref Vpc
    SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: 5432
        ToPort: 5432
        SourceSecurityGroupId: !Ref AppSg      # only the app, never 0.0.0.0/0

10.2 Network ACLs#

A NACL is attached to a subnet, is stateless, supports both ALLOW and DENY, and evaluates numbered rules in ascending order with first-match-wins. Because it is stateless, return traffic must be permitted explicitly — typically the ephemeral port range 1024–65535 outbound for inbound connections, and the same inbound for outbound ones. Forgetting this is the classic NACL outage, and its signature is a connection that establishes and then times out.

The default NACL allows everything in both directions, which is the right setting for the vast majority of architectures. Reach for a NACL when you need something a security group structurally cannot express: blocking a specific IP range, or applying a subnet-wide deny during an incident.

10.3 The other network controls#

• AWS Network Firewall — a managed stateful firewall with Suricata-compatible rules, deployed into its own subnet and inserted into the route path. Used for deep packet inspection, domain-based egress filtering, and IPS signatures.
• AWS WAF — layer 7, attached to CloudFront, ALB, API Gateway, AppSync or Cognito. Managed rule groups for the OWASP top ten, rate-based rules, bot control, and geo-blocking.
• Shield Standard is automatic and free (L3/L4 DDoS); Shield Advanced adds L7 protection, a response team and billing protection for a substantial monthly fee.
• Gateway Load Balancer — inserts third-party virtual appliances transparently into the traffic path using GENEVE encapsulation.

Peering is not transitive, and this is the fact that drives most people to Transit Gateway. If A peers with B and B peers with C, A cannot reach C — you would need a third peering. Ten VPCs fully meshed is forty-five connections and forty-five sets of route-table entries to maintain. A Transit Gateway makes it ten attachments, and its route tables let you express policies like "every spoke may reach shared services, but spokes may not reach each other" — which is exactly the segmentation a full mesh cannot give you.

For hybrid connectivity, the production pattern is Direct Connect with a VPN backup. DX gives consistent latency and bandwidth but is a physical circuit that can be cut; the VPN costs little and takes over via BGP when DX drops. A single DX circuit with no backup is a single point of failure that takes weeks to replace.

12.1 Reading an instance type#

Graviton instances (the g suffix) are AWS's own ARM64 processors. They are typically 20% cheaper and 20–40% faster per dollar than the equivalent x86 instance. For anything running an interpreted language, a JVM, or a container built from a multi-arch base image, the migration is usually one line of a Dockerfile or one AMI change. It is the single highest-value, lowest-effort cost optimisation available on EC2.

T-family burstable instances accrue CPU credits while idle and spend them when busy. They are excellent for low-average, spiky workloads and a trap for sustained ones: when credits run out the instance is throttled to its baseline (as little as 5% of a vCPU for a t3.micro), or, in unlimited mode, silently billed for surplus credits. A production service pinned at 100% CPU on a T instance is either mysteriously slow or mysteriously expensive, and the CloudWatch metric to check is CPUCreditBalance.

12.2 EBS and instance store#

EBS volumes are zonal: a volume in us-east-1a can only attach to an instance in us-east-1a. Snapshots are stored in S3, are incremental, and are regional — which is how you move a volume between AZs or Regions. EBS Multi-Attach allows one io2 volume to attach to several instances, but only with a cluster-aware filesystem; mounting ext4 twice will corrupt it.

12.3 Purchase options#

Spot deserves more use than it gets. The interruption notice is two minutes, delivered through instance metadata and as an EventBridge event, which is enough time to drain a load balancer target, checkpoint a job, or requeue a message. Combine it with an Auto Scaling group using multiple instance types and a capacity-optimised allocation strategy, and interruptions become rare and harmless. Workloads that suit it: CI runners, batch and ETL, rendering, ML training with checkpointing, and stateless web tiers where a mixed on-demand/Spot fleet keeps a guaranteed baseline.

12.4 Operating instances without SSH#

The modern pattern removes SSH entirely. Systems Manager Session Manager gives you a shell through an outbound-only connection from the SSM agent — no bastion host, no open port 22, no key pairs to distribute or rotate, and every session logged to CloudTrail and optionally recorded to S3. It works in fully private subnets given SSM interface endpoints. Alongside it, Run Command executes scripts across a fleet, Patch Manager handles OS patching on a schedule, and Parameter Store holds configuration.

# a shell on a private instance with no bastion and no inbound rules
aws ssm start-session --target i-0abc123def456

# port-forward a private database to localhost, through the instance
aws ssm start-session --target i-0abc123def456 \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["prod-pg.abc.eu-west-1.rds.amazonaws.com"],
                 "portNumber":["5432"],"localPortNumber":["5432"]}'

# why is an instance not managed by SSM? Almost always one of these three.
aws ssm describe-instance-information \
  --query 'InstanceInformationList[].{Id:InstanceId,Ping:PingStatus,Agent:AgentVersion}'
# 1. no AmazonSSMManagedInstanceCore in the instance profile
# 2. no route to the SSM endpoints (no NAT and no interface endpoints)
# 3. the agent is not installed or not running

An Auto Scaling group maintains a desired number of instances between a minimum and a maximum, replaces unhealthy ones, spreads them across the subnets you give it, and adjusts the desired count according to scaling policies. A launch template describes what to launch: AMI, instance type, key pair, security groups, IAM instance profile, user data, block device mapping. Templates are versioned, which is what makes an instance refresh a controlled operation.

13.1 Scaling policy types#

The metric matters more than the policy. CPU is a poor proxy for load on an I/O-bound service; ALBRequestCountPerTarget is usually far better, because it scales on the thing that actually arrives rather than on a symptom. For queue-driven workers, scale on the queue's backlog per instance — approximate number of messages divided by running instances — which is the metric that keeps drain time constant as load changes.

13.2 Health checks, lifecycle hooks and instance refresh#

By default an ASG uses EC2 status checks, which only tell you the hypervisor thinks the instance is alive. Switching the health-check type to ELB makes the group defer to the load balancer's application-level check, so a process that has crashed but left the instance running is replaced. Set the health check grace period to slightly more than your boot time, or the group will kill instances while they are still starting and loop forever.

Lifecycle hooks pause an instance in Pending:Wait or Terminating:Wait so you can run something — register with a config system, warm a cache, drain connections, upload final logs. The instance stays paused until you call CompleteLifecycleAction or the timeout expires, which makes an instance stuck in Pending:Wait a sign that your hook handler failed silently.

Instance refresh rolls the whole group onto a new launch template version with a minimum healthy percentage and optional checkpoints — a rolling deployment for AMI changes, with automatic rollback if the new version fails to become healthy.

14.1 Target groups and health checks#

A target group is a set of targets (instances, IPs, Lambda functions or another ALB) plus a health-check configuration and an algorithm. Listener rules route to target groups; the target group decides which member gets the request. Round robin is the default; least outstanding requests is usually better when request durations vary widely, because it stops a slow request from queueing behind another slow one on the same target.

Health checks are the control loop. Configure the path to exercise the dependencies that matter — a check that returns 200 without touching the database will happily keep a database-less instance in rotation. Set the deregistration delay (the drain time) to just above your slowest normal request: too low and you cut off in-flight work on every deploy, too high (the default is 300 s) and every deployment crawls.

14.2 Connection behaviour and the 5xx you will meet#

15.1 The object model#

A bucket is a globally unique name in one Region. An object is a key (up to 1,024 UTF-8 bytes), a value (up to 5 TB), metadata, and a version ID if versioning is on. There are no directories; the console's folder view is synthesised from key prefixes using the delimiter parameter of ListObjectsV2. There is no rename — a rename is a copy followed by a delete, which for a large object is a server-side copy you pay for. And there is no append or partial write: PutObject replaces the whole object atomically.

Since December 2020, S3 provides strong read-after-write consistency for all operations, including list. Any advice you find about "S3 is eventually consistent" predates that change and can be discarded — along with the workarounds (like S3Guard) that used to be necessary.

15.2 Performance#

S3 scales to at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per prefix, and the number of prefixes is unlimited. Prefix here means the portion of the key up to the last delimiter, and partitions split automatically as load grows. Randomising the start of a key (the old advice) is no longer necessary; simply using multiple prefixes gives you linear scaling.

• Multipart upload is required above 5 GB and recommended above about 100 MB. Parts upload in parallel and retry individually, so a network blip costs one part rather than the whole file.
• Byte-range GETs let you fetch parts of a large object in parallel, and let columnar formats read only the columns they need.
• S3 Transfer Acceleration routes uploads through the nearest edge location and over the AWS backbone — worth it for cross-continent uploads, pointless for same-Region ones.
• S3 Express One Zone is a single-AZ, single-digit-millisecond storage class for latency-critical workloads — faster and dearer, and with no cross-AZ durability.

15.3 Querying data in place#

Three ways to avoid downloading data to process it. S3 Select runs a simple SQL projection and filter on a single object server-side. Athena runs full SQL across many objects using a Glue Data Catalog schema, charging per terabyte scanned — which makes partitioning and columnar formats a cost decision, not just a speed one. S3 Object Lambda puts a function in the response path so a single stored object can be transformed per requester — redacting personal data for one consumer and not another, for instance.

16.1 The layers of access control#

1. Block Public Access — an account-level and bucket-level override that defeats any policy or ACL granting public access. Leave it on at the account level. Every "company exposes data in S3 bucket" headline is this setting turned off.
2. Bucket policy — the resource-based policy; the primary tool. Use it to require TLS, require encryption headers, restrict to an organisation or a VPC endpoint, and grant cross-account access.
3. IAM identity policies — what your own principals may do.
4. ACLs — the legacy mechanism, disabled by default on new buckets via Object Ownership. Do not enable them.
5. Access Points — named endpoints with their own policies, so a shared bucket can present a narrow view per team without one enormous bucket policy.
6. Presigned URLs — time-limited delegated access using the signer's permissions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyUnencryptedTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": ["arn:aws:s3:::acme-data", "arn:aws:s3:::acme-data/*"],
      "Condition": { "Bool": { "aws:SecureTransport": "false" } }
    },
    {
      "Sid": "RequireKmsOnUpload",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::acme-data/*",
      "Condition": {
        "StringNotEquals": { "s3:x-amz-server-side-encryption": "aws:kms" }
      }
    },
    {
      "Sid": "OnlyMyOrganisation",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": ["arn:aws:s3:::acme-data", "arn:aws:s3:::acme-data/*"],
      "Condition": {
        "StringNotEquals": { "aws:PrincipalOrgID": "o-abc123xyz" }
      }
    }
  ]
}

16.2 Versioning, delete markers and Object Lock#

With versioning on, a PUT to an existing key creates a new version and keeps the old one; a DELETE without a version ID creates a delete marker that hides the object without removing it. Recovery is deleting the delete marker. This is the only protection against an application overwriting or deleting data, and it should be on for anything you would miss.

Two stronger controls exist. MFA Delete requires an MFA code to permanently delete a version or to suspend versioning — it can only be enabled by the root user with the CLI. Object Lock enforces write-once-read-many retention: in governance mode a privileged user can override, and in compliance mode nobody can, including the root user and AWS itself, until the retention period expires. Compliance mode is the ransomware and insider-threat answer, and it is genuinely irreversible — test it on a scratch bucket first.

16.3 Lifecycle and storage classes#

Lifecycle rules apply to a prefix, a tag, or the whole bucket, and can act on current versions, non-current versions and incomplete uploads independently. The two most valuable rules are usually the least glamorous: expire non-current versions after 30–90 days (versioning otherwise grows without bound), and abort incomplete multipart uploads after 7 days.

16.4 Replication#

Cross-Region Replication copies new objects to a bucket in another Region; Same-Region Replication copies within one, typically into a locked-down log-archive account. Both require versioning on both sides. Replication is asynchronous and applies only to objects created after it is enabled — use S3 Batch Replication for the backlog. If you need a latency SLA, Replication Time Control commits to 99.99% of objects within 15 minutes, at extra cost.

EFS is the one you are most likely to need. It grows and shrinks automatically, is mounted by any number of instances at once, and is the standard answer to "several containers need to share a directory". Its performance model is worth understanding: the default Elastic throughput mode scales with usage; Bursting mode ties throughput to stored size and can throttle a small-but-busy filesystem; Provisioned mode buys a fixed rate. It also has Infrequent Access and Archive classes with lifecycle policies, exactly like S3.

RDS is a managed relational database: AWS handles provisioning, patching, backups, failover and replication for PostgreSQL, MySQL, MariaDB, Oracle, SQL Server and Db2. You do not get the host, you do not get SUPERUSER, and you do get to stop thinking about most operational work.

18.1 Multi-AZ, replicas and backups#

Automated backups give point-in-time recovery to any second within the retention window (up to 35 days) by combining daily snapshots with continuously archived transaction logs. Manual snapshots persist until you delete them. A restore always creates a new instance with a new endpoint, which is why a recovery runbook has to include a step for repointing the application — and why you should have practised it.

18.2 Aurora#

Aurora keeps the PostgreSQL and MySQL engines but replaces the storage layer with a distributed, log-structured service that keeps six copies of every 10 GB segment across three AZs. The consequences are architectural rather than incremental:

• Storage grows automatically to 128 TB, and all instances in the cluster share it — so adding a read replica copies no data and takes minutes, not hours.
• Failover is typically under 30 seconds, because there is no data to promote, only a role to change.
• Backups are continuous to S3 and have no performance impact.
• Aurora Serverless v2 scales capacity in fine-grained increments, including scaling to a very low floor for idle databases.
• Global Database replicates to other Regions with typical lag under a second and cross-Region failover in about a minute.
• The cluster exposes two endpoints: a writer endpoint that follows the primary, and a reader endpoint that load-balances across replicas.

18.3 Connection management#

Relational databases have a hard limit on connections, and each one costs memory. Serverless compute is pathological here: a thousand concurrent Lambda invocations means a thousand attempted connections, each used for milliseconds. RDS Proxy sits in front, pools and multiplexes connections, and survives failovers transparently by holding the client connection while it reconnects underneath. For containers, an in-process pool with a sane maximum, multiplied by the number of tasks, must stay under the database's limit — a calculation worth doing explicitly rather than discovering.

19.1 Keys and access patterns#

An item is identified by a partition key, optionally with a sort key. The partition key is hashed to choose a physical partition; the sort key orders items within it. That is the entire storage model, and everything else follows: GetItem by full key is O(1); Query within one partition key with a sort-key condition is efficient; Scan reads the whole table and should be treated as a last resort.

Because a query must specify a partition key, you must know your access patterns before you design the table. This inverts the relational habit: in SQL you model the data and derive the queries; here you enumerate the queries and derive the keys. Single-table design takes this to its conclusion, storing several entity types in one table with generic PK/ SK attributes so that one query returns a whole object graph.

19.2 Secondary indexes#

A sparse index is the most useful trick here: an index only contains items that have its key attribute, so writing gsi1pk only on the items you care about produces a small, cheap index over a large table. Filtering "open orders" out of ten million orders becomes a query over the few thousand that are open.

19.3 Capacity, throttling and transactions#

On-demand mode charges per request and needs no planning — the right default, and the only sane choice for unpredictable traffic. Provisioned mode is cheaper for steady, well-understood load and can be combined with auto scaling and reserved capacity. One RCU is one strongly consistent read of up to 4 KB per second (or two eventually consistent reads); one WCU is one write of up to 1 KB per second.

Throttling has two shapes. Table-level throttling means you have exceeded provisioned capacity — raise it or switch to on-demand. Partition-level throttling means one partition key is hot, and no amount of extra capacity helps because the 3,000 RCU / 1,000 WCU per-partition ceiling is fixed. The fix is key design: add a shard suffix, or choose a higher-cardinality key. Enable CloudWatch Contributor Insights on the table to see the top keys by traffic.

Other features worth knowing by name: DynamoDB Streams emits an ordered change log per partition key, which is how you drive Lambdas, replicate, or maintain a search index; TTL deletes expired items for free; transactions give ACID across up to 100 items at double the capacity cost; global tables provide multi-Region active-active with last-writer-wins conflict resolution; and PartiQL offers a SQL-like syntax that does not change any of the underlying constraints.

A cache does not absorb load, it removes it — which is why it is usually a better first response to database pressure than a larger instance. AWS offers ElastiCache for Redis, Valkey and Memcached, MemoryDB as a durable Redis-compatible primary database, and DAX as a transparent write-through cache specifically for DynamoDB.

20.1 Caching patterns and their failure modes#

• Cache-aside (lazy loading) — read the cache; on a miss read the database and populate. Only requested data is cached, and a cache failure degrades rather than breaks. The default choice.
• Write-through — write to the cache and the database together. The cache is never stale, at the price of writing data that may never be read.
• Write-behind — write to the cache, flush to the database asynchronously. Fast, and risks losing writes.

The three failure modes worth designing against: thundering herd, where a popular key expires and a thousand requests all miss and hit the database at once — mitigate with a per-key lock or by refreshing slightly before expiry; cache stampede after a restart, where an empty cache means every request is a miss — mitigate by warming; and staleness, which is a correctness question you must answer explicitly by choosing a TTL and an invalidation strategy rather than by hoping.

21.1 The execution environment#

Each concurrent request gets its own execution environment — a Firecracker micro-VM with your code, your runtime, 512 MB to 10 GB of ephemeral /tmp, and memory from 128 MB to 10,240 MB. CPU is allocated proportionally to memory: roughly one full vCPU at 1,769 MB, and up to six vCPUs at the maximum. An environment handles one request at a time and is reused for subsequent requests while traffic continues, then frozen and eventually discarded.

The practical consequences: initialise SDK clients, database pools and configuration at module scope so they are shared across invocations; never store request state in a global, because the next request on that environment will see it; and remember that background work started but not awaited may be frozen mid-flight when the handler returns and resume unpredictably on the next invocation.

21.2 Invocation models#

Stream sources (Kinesis and DynamoDB Streams) are ordered per shard, which means a failing batch blocks that shard until it succeeds or expires. Configure BisectBatchOnFunctionError, MaximumRetryAttempts and an on-failure destination, or one poison record halts a partition for a day.

21.3 Concurrency and cold starts#

An account has a default limit of 1,000 concurrent executions per Region, shared across every function. Reserved concurrency carves out a guaranteed slice for one function and simultaneously caps it — which is how you both protect a critical function and protect a fragile downstream database from an aggressive one. Provisioned concurrency keeps a number of environments initialised and warm, eliminating cold starts at the cost of paying while idle.

Levers on cold start, in order of value: reduce package size; move rarely used imports inside the handler; increase memory (more CPU makes initialisation finish sooner); choose a fast-starting runtime; use SnapStart, which snapshots the initialised environment and restores it (available for Java, Python and .NET); and only then buy provisioned concurrency. Periodic "warming" pings are close to useless, because each ping warms exactly one environment while a burst needs hundreds.

API Gateway gives you throttling and quotas, request validation against a JSON schema, stages and canary deployments, custom domains with ACM certificates, authorizers, and CloudWatch and X-Ray integration. Its direct service integrations are underused: an endpoint can put a message on SQS or an item in DynamoDB with no Lambda at all, which removes a whole component from the critical path.

22.1 CORS, and why it is always the problem#

A browser making a cross-origin request with a non-simple method or header first sends an OPTIONS preflight. Three failures account for nearly every CORS ticket: the OPTIONS route is not configured, so the preflight 403s; the error responses (4xx and 5xx generated by API Gateway itself, not your code) carry no CORS headers, so a real error appears in the browser as a CORS error and sends you looking in the wrong place; and CORS is configured in both API Gateway and the Lambda, which then double-set the header and invalidate it.

23.1 ECR#

The Elastic Container Registry stores images, scans them for vulnerabilities (basic scanning on push, or enhanced scanning through Inspector for continuous rescanning as new CVEs are published), and supports immutable tags — which you should turn on, because a mutable latest means you cannot say what is running. Lifecycle policies expire untagged and old images so the registry does not grow forever. Authentication is an IAM-derived token, so a pull requires ecr:GetAuthorizationToken plus repository permissions, and that is why an ECS execution role missing ECR permissions produces a task that never starts.

23.2 ECS#

ECS has three objects. A task definition is an immutable revision describing one or more containers: image, CPU, memory, port mappings, environment, secrets, logging, and the task and execution roles. A task is a running instance of one. A service maintains a desired number of tasks, registers them with a load balancer, and performs deployments.

Fargate is a launch type with no instances: you declare CPU and memory per task and AWS runs it. EC2 launch type places tasks on instances you own, which is cheaper at sustained scale and necessary for GPUs, custom kernels, daemon containers and very high task density. Fargate is the right default; move to EC2 when a specific requirement or a modelled cost saving pushes you.

Networking on Fargate is always awsvpc mode: each task gets its own ENI with its own private IP and its own security groups, which makes per-task network policy straightforward. Service discovery comes from Cloud Map (DNS names for tasks) or from a load balancer. For one-off and scheduled work, ECS Exec gives you a shell inside a running task without SSH, and EventBridge Scheduler runs a task on a cron.

23.3 EKS#

EKS is managed Kubernetes: AWS runs the control plane across three AZs; you run the data plane as managed node groups, self-managed nodes, or Fargate profiles. The AWS-specific pieces you must understand are the ones that bridge Kubernetes concepts to AWS ones:

• Pod Identity or IRSA — maps a Kubernetes service account to an IAM role, so pods get AWS credentials without node-level roles.
• VPC CNI — pods get real VPC IP addresses, which is excellent for security-group integration and means IP exhaustion is a real capacity limit. Prefix delegation raises the density considerably.
• AWS Load Balancer Controller — turns Ingress objects into ALBs and Services into NLBs.
• Karpenter — provisions right-sized nodes in seconds based on pending pods, largely replacing the cluster autoscaler and node group capacity planning.
• EBS/EFS CSI drivers — persistent volumes.

Once a process has more than about three steps with retries, branches and partial failures, hand-rolling it in Lambda becomes a distributed-systems project you did not intend to start. Step Functions is a managed state machine: you declare states, transitions, retry policies and error handling, and AWS runs it durably, keeping the full execution history for inspection.

{
  "Comment": "Order fulfilment with compensation on failure",
  "StartAt": "ReserveStock",
  "States": {
    "ReserveStock": {
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:updateItem",
      "Parameters": { "TableName": "inventory", "Key": {"sku": {"S.$": "$.sku"}} },
      "Retry": [{
        "ErrorEquals": ["States.TaskFailed", "DynamoDB.ThrottlingException"],
        "IntervalSeconds": 1, "MaxAttempts": 4, "BackoffRate": 2.0, "JitterStrategy": "FULL"
      }],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "FailOrder" }],
      "Next": "ChargeCard"
    },
    "ChargeCard": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "charge", "Payload.$": "$" },
      "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "ReleaseStock" }],
      "Next": "ShipInParallel"
    },
    "ShipInParallel": {
      "Type": "Parallel",
      "Branches": [
        { "StartAt": "NotifyWarehouse", "States": {
            "NotifyWarehouse": {"Type": "Task",
              "Resource": "arn:aws:states:::sqs:sendMessage",
              "Parameters": {"QueueUrl": "https://sqs…/warehouse",
                             "MessageBody.$": "$"}, "End": true}}},
        { "StartAt": "SendEmail", "States": {
            "SendEmail": {"Type": "Task",
              "Resource": "arn:aws:states:::lambda:invoke",
              "Parameters": {"FunctionName": "email", "Payload.$": "$"},
              "End": true}}}
      ],
      "End": true
    },
    "ReleaseStock": { "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {"FunctionName": "release-stock", "Payload.$": "$"},
      "Next": "FailOrder" },
    "FailOrder": { "Type": "Fail", "Error": "OrderFailed" }
  }
}

Two capabilities to know by name. Callback patterns (.waitForTaskToken) pause a workflow until an external system — a human approver, a third party — calls back with the token, for up to a year. And Map state, especially distributed map, fans out over millions of items in S3 with controlled concurrency, which turns "process every object in this bucket" into a configuration.

25.1 SQS#

A queue holds messages up to 256 KB (larger payloads go to S3 with the extended client library) for up to 14 days. Consumers receive a message, which makes it invisible for the visibility timeout, and delete it when done — deletion is the acknowledgement. Failing to delete means redelivery, which is the entire durability model.

Use long polling (WaitTimeSeconds of 20) always: it reduces empty receives, cost and latency simultaneously, and there is no case where short polling is better. Configure a redrive policy to a dead-letter queue on every queue, and alarm on the DLQ's depth — a DLQ nobody watches is a folder of lost work.

The metric to alarm on is ApproximateAgeOfOldestMessage, not queue depth. Depth tells you how much work exists; age tells you whether you are keeping up, which is what actually matters to the person waiting.

25.2 SNS#

SNS is push-based publish/subscribe. A publisher sends to a topic and every subscriber gets a copy — Lambda, SQS, HTTPS endpoints, email, SMS, and mobile push. It has no storage of its own, so a subscriber that is down misses messages unless you set a retry policy and a DLQ. Message filtering on attributes (and now on the message body) lets subscribers receive only what they care about.

The canonical pattern is fan-out to queues: SNS topic → several SQS queues → one consumer each. That combines SNS's cheap duplication with SQS's durability and independent retries, and it is a better default than subscribing Lambdas directly.

25.3 EventBridge#

EventBridge routes structured JSON events using content-based rules, with three distinguishing capabilities: the default bus receives events from AWS services themselves (an EC2 state change, an ECS task stopping, a CodePipeline stage failing), partner event sources bring SaaS events in directly, and the schema registry can generate typed bindings from observed events. EventBridge Pipes connects a source to a target with optional filtering and enrichment, replacing a great deal of glue code, and EventBridge Scheduler is a managed cron for over 200 targets with millions of schedules.

Queues delete a message once it is processed. Streams keep an ordered log that many independent consumers read at their own positions, and that can be replayed. That difference — replay and multiple independent readers — is the reason to choose a stream.

A Kinesis shard supports 1 MB/s or 1,000 records per second in, and 2 MB/s out shared across consumers — or 2 MB/s per consumer with enhanced fan-out. Records are ordered within a shard, and the partition key determines the shard, so the same hot-key problem as DynamoDB applies: a partition key with poor cardinality creates a hot shard that throttles while the stream is mostly idle. On-demand mode removes shard management at higher cost.

The AWS analytics story is built on the idea that S3 is the data lake and the engines are interchangeable readers of it. Storage and compute are separated, so you can point several tools at the same files.

• Glue Data Catalog — the shared metastore: databases, tables, schemas, partitions. Hive-compatible, and used by Athena, Redshift Spectrum, EMR and Glue ETL alike.
• Glue ETL — serverless Spark for transformation, with crawlers that infer schemas from files.
• Athena — serverless SQL over S3, billed per terabyte scanned. No cluster, no loading, no maintenance.
• Redshift — a columnar MPP warehouse for repeated, complex analytical queries over curated data. Redshift Serverless removes cluster management; Spectrum queries S3 directly.
• EMR — managed Spark, Hive, Presto and HBase when you need the full ecosystem or fine-grained control.
• OpenSearch — full-text search, log analytics and dashboards.
• QuickSight — BI dashboards, priced per user or per session.
• Lake Formation — fine-grained (column, row and cell level) permissions over the lake, layered on top of the catalog.

The performance and cost rules for a lake are the same three every time: columnar format (Parquet or ORC) so a query reads only the columns it needs; partitioning by the columns you filter on, usually date; and file size around 128–512 MB, because thousands of small files cost far more in request overhead than they save in granularity. Open table formats — Iceberg, Hudi, Delta — add ACID transactions, schema evolution and time travel on top, and Iceberg has become the default choice on AWS.

Route 53 is a global, authoritative DNS service with a 100% availability SLA, plus domain registration and health checking. Hosted zones are public (resolvable from the internet) or private (associated with VPCs, resolvable only inside them).

28.1 Alias records#

An alias is a Route 53 extension that points at an AWS resource — ALB, NLB, CloudFront, S3 website, API Gateway, another record in the zone. It is resolved inside Route 53 rather than by chaining a lookup, so it is free to query, it tracks the target's changing addresses automatically, and — uniquely — it can exist at the zone apex, where the DNS standard forbids a CNAME. There is no situation in which a CNAME to an AWS endpoint is preferable.

28.2 Routing policies and health checks#

Health checks probe an endpoint (or watch a CloudWatch alarm, or aggregate other health checks) from multiple Regions. Attaching them to records is what turns a routing policy into a failover mechanism. Note the arithmetic: 30-second interval × 3 failures + the record TTL means realistic DNS failover takes roughly 90–150 seconds, and no configuration makes it seconds.

Inside a VPC, the Route 53 Resolver at base+2 answers for private hosted zones and forwards everything else. Resolver endpoints extend this to hybrid environments: inbound endpoints let on-premises resolvers query AWS private zones, outbound endpoints forward specified domains to on-premises DNS. DNS Firewall blocks queries to known-bad domains, which catches a great deal of malware command-and-control traffic.

CloudFront is a CDN with over 600 points of presence. It terminates TLS near the user, serves cached content from the edge, and carries cache misses to the origin over AWS's backbone. Its benefits divide into three, and only the first is what people expect:

1. Caching — a hit never reaches your origin, so a 95% hit ratio means your fleet handles a twentieth of the traffic.
2. Connection termination — even at a 0% hit ratio, terminating TLS a few milliseconds away and using the backbone typically saves 20–40% of latency for dynamic APIs.
3. Security and economics — it is where WAF and Shield attach, and CloudFront's data-transfer-out rate is lower than EC2's.

29.1 Cache keys and policies#

Three separate policies control behaviour and confusing them is the source of most CloudFront problems. The cache policy defines the cache key — which headers, cookies and query strings distinguish one cached variant from another — and the TTLs. The origin request policy defines what is forwarded to the origin, which can be more than is in the cache key. The response headers policy adds CORS and security headers at the edge without touching the origin.

Every value added to the cache key multiplies the number of variants and divides the hit ratio. Forwarding all headers is the single most common misconfiguration and effectively disables caching while still charging you for it.

29.2 Origin access and edge compute#

Origin Access Control lets CloudFront sign requests to a private S3 bucket, so the bucket never needs public access. Signed URLs and signed cookies restrict content to authorised users with an expiry. Field-level encryption encrypts specific form fields at the edge so only a designated service can read them.

Two edge runtimes exist and they are not interchangeable. CloudFront Functions are sub-millisecond JavaScript at the edge location for header manipulation, URL rewrites, redirects and cache-key normalisation — extremely cheap, no network access. Lambda@Edge runs full Node or Python at the regional edge cache, can make network calls and take up to 30 seconds, and is correspondingly more expensive. Reach for Functions first; almost every request-manipulation task fits.

30.1 Metrics and alarms#

A metric is identified by namespace, name and a set of dimensions. Note that each unique combination of dimensions is a separate metric, and a separate charge — which is why putting a user ID or a request ID in a dimension produces a spectacular bill. High-cardinality data belongs in logs, which are queryable, not in metric dimensions.

Alarms watch a metric or a metric-math expression over a number of periods, and support three types: static thresholds, anomaly detection (a learned band around expected behaviour, good for metrics with a daily cycle), and composite alarms that combine others with boolean logic to cut alert noise. Set treatMissingData deliberately for every alarm; the default of missing means a completely dead service produces INSUFFICIENT_DATA rather than a page.

30.2 Logs#

CloudWatch Logs organises streams into groups. Set a retention period on every group — the default is to keep data forever, and log storage silently becomes a major line item. Logs Insights queries with a purpose-built language; metric filters turn a log pattern into a metric you can alarm on; subscription filters stream logs onward to Firehose, OpenSearch or a third party in real time.

Emit structured JSON with a correlation ID, and Insights can aggregate and percentile it directly. The Embedded Metric Format goes further: a specially shaped JSON log line is extracted into a real CloudWatch metric with no separate API call, giving you metrics and the surrounding context from one write.

30.3 Tracing and auditing#

X-Ray propagates a trace header through your services and assembles segments into a service map with timing per hop. It is the only practical way to answer "where did the 3 seconds go?" across several services, and it is what turns a vague latency complaint into a named dependency. Application Signals layers SLOs on top of the same data, using OpenTelemetry instrumentation.

CloudTrail records every API call: who, what, when, from which IP, with which parameters, and whether it succeeded. Management events are recorded free for 90 days; a trail to S3 gives you long-term retention, and CloudTrail Lake gives you SQL over the history. Data events (S3 object-level access, Lambda invocations) are opt-in and charged, but are often exactly what an investigation needs.

31.1 KMS#

KMS holds keys inside FIPS-validated hardware security modules and never releases the key material. Because it only encrypts payloads up to 4 KB, everything of size uses envelope encryption: GenerateDataKey returns a data key in plaintext and encrypted under the KMS key; you encrypt locally, store the encrypted data key beside the ciphertext, and discard the plaintext.

A key policy is a resource policy on the key and is the primary access control — IAM alone is not sufficient unless the key policy delegates to IAM. This is why an AccessDenied reading an encrypted S3 object often has nothing to do with S3. Prefer customer managed keys where it matters: they give you a key policy, automatic annual rotation, grants, aliases and, decisively, the ability to disable the key and instantly revoke access to everything it protects. CloudHSM exists for single-tenant hardware and regulatory requirements that KMS cannot satisfy.

31.2 Secrets Manager and Parameter Store#

31.3 Detective and preventive services#

• GuardDuty — continuously analyses CloudTrail, VPC Flow Logs, DNS logs, EKS audit logs and S3 data events for credential exfiltration, crypto-mining, reconnaissance and known-bad destinations. One click, no agents, and it finds real things.
• Inspector — continuous vulnerability scanning of EC2, ECR images and Lambda functions against the CVE database.
• Macie — discovers personal and sensitive data in S3.
• Security Hub — aggregates findings from all of the above against CIS and AWS Foundational benchmarks, organisation-wide.
• Detective — builds a behaviour graph so an investigation can follow a finding across resources and time.
• IAM Access Analyzer — identifies resources reachable from outside the account or organisation, validates policies, and generates least-privilege policies from CloudTrail history.
• WAF — layer 7 filtering on CloudFront, ALB, API Gateway and AppSync, with managed rule groups, rate limiting and bot control.
• Shield Advanced — L7 DDoS protection, a response team, and cost protection against attack-driven scaling charges.
• Config — records configuration history and evaluates rules, with automatic remediation.

32.1 CloudFormation#

A template declares resources; CloudFormation computes the difference between the declaration and reality and converges. A stack is the deployed instance of a template and owns the resources it created — which is what makes deletion, rollback and drift detection possible.

Template sections you will use: Parameters for inputs (with AllowedValues and NoEcho), Mappings for lookup tables, Conditions for environment differences, Resources, and Outputs (with Export for cross-stack references). Intrinsic functions — !Ref, !GetAtt, !Sub, !If, !ImportValue, !FindInMap — build the dependency graph automatically, so explicit DependsOn is rarely needed.

Behaviours to know: an update may modify in place, modify with interruption, or replace a resource, and a change set tells you which; a failure triggers automatic rollback of the whole stack; drift detection reports resources changed outside CloudFormation; StackSets deploy one template across many accounts and Regions; nested stacks and cross-stack exports compose large estates — noting that an exported value cannot be changed while another stack imports it, which is a common source of stuck updates.

32.2 CDK, SAM and Terraform#

# AWS CDK - a real language that synthesises CloudFormation.
# L2 constructs encode sensible defaults; that is most of the value.
from aws_cdk import Stack, Duration, aws_ec2 as ec2, aws_ecs as ecs
from aws_cdk import aws_ecs_patterns as patterns
from constructs import Construct


class ApiStack(Stack):
    def __init__(self, scope: Construct, id: str, **kw):
        super().__init__(scope, id, **kw)

        # 3 AZs, public + private subnets, NAT, route tables - all of it
        vpc = ec2.Vpc(self, "Vpc", max_azs=3, nat_gateways=3)

        patterns.ApplicationLoadBalancedFargateService(
            self, "Api",
            vpc=vpc,
            cpu=512,
            memory_limit_mib=1024,
            desired_count=3,
            public_load_balancer=True,
            task_image_options=patterns.ApplicationLoadBalancedTaskImageOptions(
                image=ecs.ContainerImage.from_registry("acme/api:1.4.2"),
                container_port=8080,
            ),
            circuit_breaker=ecs.DeploymentCircuitBreaker(rollback=True),
            health_check_grace_period=Duration.seconds(60),
        )
        # ~20 lines replacing several hundred lines of YAML - and it is
        # still CloudFormation underneath, so change sets and rollback remain.

# AWS SAM - CloudFormation with a serverless-shaped shorthand.
# The Transform line expands these into full CloudFormation resources.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Globals:
  Function:
    Runtime: python3.12
    Timeout: 30
    MemorySize: 1024
    Architectures: [arm64]
    Tracing: Active                 # X-Ray, one line
    Environment:
      Variables:
        TABLE: !Ref Orders

Resources:
  Api:
    Type: AWS::Serverless::HttpApi
    Properties:
      StageName: prod

  CreateOrder:
    Type: AWS::Serverless::Function
    Properties:
      Handler: app.create
      Policies:
        - DynamoDBWritePolicy: {TableName: !Ref Orders}   # scoped, not s3:*
      Events:
        Post:
          Type: HttpApi
          Properties: {ApiId: !Ref Api, Path: /orders, Method: POST}

  Orders:
    Type: AWS::DynamoDB::Table
    DeletionPolicy: Retain          # the line that saves the data
    Properties:
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions: [{AttributeName: pk, AttributeType: S}]
      KeySchema: [{AttributeName: pk, KeyType: HASH}]

# Terraform - multi-cloud, its own state file, plan/apply.
# The tradeoff: no CloudFormation rollback, and the state file is
# now a critical, lockable, backed-up asset you own.
terraform init -backend-config="bucket=acme-tfstate" \
               -backend-config="dynamodb_table=tf-locks"
terraform plan -out=tfplan          # ALWAYS review before apply
terraform apply tfplan

# generate a least-privilege policy from the plan rather than guessing
terraform show -json tfplan > plan.json

# CDK and SAM equivalents
cdk diff                            # what will change
cdk deploy --require-approval any-change
sam build && sam deploy --guided
sam local start-api                 # run the API locally in Docker

The AWS-native chain is CodePipeline (orchestration) driving CodeBuild (build and test) and CodeDeploy (deployment strategies), with CodeArtifact for private packages and CodeConnections linking to GitHub, GitLab or Bitbucket. Many teams instead run GitHub Actions or GitLab CI and use AWS only as the deployment target — which is a perfectly good choice, and made much better by OIDC federation.

# GitHub Actions deploying to AWS with no stored credentials at all.
# The role's trust policy pins the repo and the branch (see section 6).
name: deploy
on:
  push:
    branches: [main]

permissions:
  id-token: write        # required to request the OIDC token
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v5
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-deploy
          aws-region: eu-west-1

      - uses: aws-actions/amazon-ecr-login@v2
        id: ecr

      - name: Build and push
        run: |
          IMAGE=${{ steps.ecr.outputs.registry }}/api:${{ github.sha }}
          docker build --platform linux/arm64 -t "$IMAGE" .
          docker push "$IMAGE"
          echo "IMAGE=$IMAGE" >> "$GITHUB_ENV"

      - name: Deploy
        run: |
          aws cloudformation deploy \
            --template-file infra/api.yaml \
            --stack-name api-prod \
            --parameter-overrides ImageUri="$IMAGE" \
            --capabilities CAPABILITY_IAM \
            --no-fail-on-empty-changeset

Deployment strategies, and what each buys you:

All of them are defeated by a database migration that only the new code understands. The discipline is expand, migrate, contract: deploy a schema change that both versions tolerate, deploy the code, backfill, and only then remove the old column in a later release. Two deployments and a wait, instead of one deployment and an outage.

Section 12.3 covered how compute is bought. This section is about seeing where the money actually goes and reducing it, which is a different skill: the bill is denominated in usage types, not in applications, and the largest surprises are rarely in the line item you expected.

34.1 The tools#

• Cost Explorer — the first stop. Group by usage type, not just by service, to see which meter moved.
• Budgets — alerts on actual and forecast spend, per account, service or tag.
• Cost Anomaly Detection — machine-learning alerts on unusual spend patterns. Free, and should be on everywhere.
• Cost and Usage Report — the full hourly, resource-level dataset in S3, queryable with Athena.
• Compute Optimizer — right-sizing recommendations for EC2, EBS, Lambda and ECS from actual utilisation.
• Trusted Advisor — idle resources, unassociated Elastic IPs, low utilisation, and a security checklist.
• Cost allocation tags — activate them early; they cannot be applied retroactively to already-billed usage.

34.2 The levers, in order of return#

1. Turn things off. Non-production environments stopped outside working hours cost about 70% less. Nothing else comes close, and it is a scheduled Lambda.
2. Delete orphans. Unattached EBS volumes, old snapshots, unassociated Elastic IPs, idle load balancers, unused NAT Gateways, forgotten dev clusters.
3. Fix data transfer. Add S3 and DynamoDB gateway endpoints, keep chatty services in one AZ where availability permits, and serve internet egress through CloudFront.
4. Buy Savings Plans for the steady baseline once you can see what it is.
5. Move to Graviton. Around 20% cheaper and usually faster; often a one-line change.
6. Use Spot for anything interruptible.
7. Right-size. Compute Optimizer will tell you which instances are oversized.
8. Storage lifecycle. S3 lifecycle rules, log retention periods, EBS snapshot expiry.
9. Re-architect. Serverless for spiky workloads, caching to remove database load, queues instead of over-provisioned capacity.

Six pillars, each of which is really a set of questions to ask about a design. The framework is worth knowing not because it is a certification topic but because it is the checklist every AWS design review runs, and it catches real omissions.

35.1 Three reference architectures#

36.1 Numbers worth knowing#

36.2 Errors and what they mean#

Design interviews and real design meetings are both pattern-matching exercises. These are the mappings from a stated requirement to the service that answers it.

Reading about AWS produces recognition; building on it produces knowledge. Each project below is deliberately small and adds exactly one new mechanism. Do them in order, in a personal account with a budget alarm set, and tear each one down afterwards.

1. Secure the account. Root MFA, no root keys, an IAM Identity Center user for yourself, a budget alert at $10, Cost Anomaly Detection on, CloudTrail on. Half an hour, and it is the habit that matters.
2. Static site. S3 + CloudFront with OAC, an ACM certificate in us-east-1, a Route 53 alias record. Deploy with hashed filenames and confirm you never need an invalidation.
3. Build the VPC by hand once. Three AZs, public and private subnets, IGW, one NAT, route tables, security groups. Then delete it and write it as CloudFormation. The second time teaches more than the first.
4. Serverless CRUD API. API Gateway + Lambda + DynamoDB with SAM. Add a GSI for a second access pattern and notice what it costs you to add it late.
5. Break IAM on purpose. Create a role with a deliberately narrow policy, watch it fail, and fix it using simulate-principal-policy. Then add a bucket policy that denies, and confirm that admin does not help.
6. Containerise something. Push an image to ECR, run it on ECS Fargate behind an ALB across three AZs, turn on the deployment circuit breaker, and deliberately deploy a broken image to watch it roll back.
7. Add asynchrony. S3 upload → EventBridge → SQS → Lambda → DynamoDB, with a DLQ and an alarm. Kill the consumer mid-message and watch the visibility timeout return it.
8. Add a database. RDS PostgreSQL Multi-AZ in private subnets, reached through Session Manager port forwarding. Trigger a failover from the console and time it.
9. Observe it. Structured JSON logs, a Logs Insights query for p99 by route, an alarm on p99 with treatMissingData: breaching, and X-Ray across the whole chain.
10. Optimise it. Add an S3 gateway endpoint and watch the NAT charge fall. Move a Lambda to arm64. Set log retention. Add S3 lifecycle rules.
11. Automate the deployment. GitHub Actions with OIDC — no stored credentials — building once and promoting the same image digest through two environments.
12. Break it deliberately. Terminate an instance, stop a task, revoke a security group rule, fill a disk. Write down what you saw, what alerted, and how long it took to notice. That document is worth more than another tutorial.

38.1 Certifications, if you want them#

Certifications are a reasonable syllabus and a poor substitute for building. Solutions Architect Associate is the broadest and most useful; Developer Associate goes deeper on Lambda, DynamoDB and deployment; SysOps Associate on operations and monitoring. The Professional and Specialty exams are genuinely difficult and assume real experience. If you do take one, build each service before you revise it — the questions are scenario-based, and recognition from having actually done it beats memorisation every time.