Common AWS errors and how to fix them

Old AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN are still set in the shell. The CLI uses environment variables before your profile, so it keeps sending the expired copy even after you log in again.

Check

aws configure list  # if access_key shows Type env, the AWS_* variables are being used, not your profile
env | grep -o '^AWS_[A-Z_]*'  # names only, never values: AWS_SESSION_TOKEN listed means pasted session credentials are set

Fix

Unset them: unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN, then run aws sts get-caller-identity again.

The SSO session behind the profile has ended. The role credentials the CLI cached from it expire, and it cannot get new ones until you sign in again.

Check

aws configure list  # Type sso or a profile with sso_session points here
aws sts get-caller-identity --profile <profile>  # still fails with ExpiredToken or an SSO token error

Fix

Sign in again: aws sso login --profile <profile>.

A long job outlived its role session. Credentials from assume-role last one hour by default, and a script that fetched them once at the start cannot renew them.

Check

aws iam get-role --role-name <role> --query Role.MaxSessionDuration  # 3600 means one hour, the default

Fix

Let the SDK refresh credentials itself from a profile with role_arn and source_profile, or from the instance or task role, instead of pasting keys. If the job really needs longer sessions, raise the role's MaxSessionDuration (up to 12 hours).

The key was deleted or set to Inactive. Rotating or disabling a key in IAM invalidates every copy of it at once.

Check

aws configure list  # note the last four characters of access_key, and where it comes from
aws iam list-access-keys --user-name <user>  # run with other, working credentials: if that key is missing or Inactive, this is your cause

Fix

Use a profile with a valid key (--profile), or create a new key for the user and replace the old one everywhere it was copied.

A temporary key (it starts with ASIA) was sent without AWS_SESSION_TOKEN. Temporary keys are only valid together with their token, so AWS treats the key alone as unknown.

Check

[ -n "$AWS_SESSION_TOKEN" ] && echo token set || echo no token  # a key starting ASIA with no token is this cause

Fix

Set AWS_SESSION_TOKEN with the key and secret from the same assume-role or SSO output, or better, use a profile that fetches all three itself.

The call goes to an opt-in Region (such as af-south-1 or ap-east-1) that is not enabled for the account, or uses there a session token from the global STS endpoint. By default, global STS tokens are valid only in Regions that are enabled by default.

Check

aws account list-regions --region-opt-status-contains ENABLED DISABLED --query 'Regions[].[RegionName,RegionOptStatus]'  # the Region you call shows DISABLED

Fix

Enable the Region for the account, or get session credentials from that Region's own STS endpoint (--region <region> with regional endpoints).

The secret access key was copied with a missing or extra character, or with a trailing space or newline from a file or variable. A secret that differs by one character signs every request differently.

Check

aws configure list  # the key ID (last four characters) and where it comes from
aws configure get aws_secret_access_key --profile <profile> | tr -d '\n' | wc -c  # length only: a secret access key is 40 characters; anything else means it was mangled

Fix

Enter the secret again with aws configure set aws_secret_access_key ..., with no quotes or spaces, or create a new key pair.

The secret belongs to a different key than this key ID, for example two variables filled from different places. AWS looks up the secret by the key ID, so the wrong secret never matches.

Check

aws configure list  # access_key and secret_key should come from the same Type and Location

Fix

Take the key ID and the secret from the same key, from one place: one profile, or one pair of variables.

A presigned S3 URL is used with a different method, headers or Content-Type than it was signed for. Those are part of what was signed, so changing them breaks the signature.

Check

curl -sS -o /dev/null -w '%{http_code}\n' '<presigned-url>'  # works with a plain GET but fails from your code: the code adds or changes a signed header

Fix

Send exactly the method and headers the URL was signed with, or sign it again with the headers your code sends.

A proxy or tool changes the request after it was signed (headers, path encoding). AWS checks the request it receives, not the one your client signed.

Check

env | grep -io '^\(https\?\|no\)_proxy'  # names only: a proxy variable set means requests go through it; try once without it

Fix

Exclude AWS endpoints from the proxy with NO_PROXY, or make it pass requests through unchanged.

No profile is configured on this machine, or the job runs as another user or in a container with a different home directory. The files in ~/.aws are per user, so a cron job or a container does not see yours.

Check

aws configure list-profiles  # an empty list means no profiles for this user
echo "$HOME"; whoami  # a different user or home than you expected is this cause

Fix

People: aws configure sso. Jobs: give them their own role (instance profile, task role, OIDC in CI) rather than copying your files.

AWS_PROFILE names a profile that has no credentials in it, for example one with only a Region. The CLI uses that profile and stops there.

Check

echo "$AWS_PROFILE"  # the profile in use
aws configure list  # access_key shows <not set> for that profile

Fix

Point --profile or AWS_PROFILE at a profile that has credentials, or add them to that profile.

An EC2 instance has no instance profile, or IMDS is turned off, so the last step of the chain finds no role.

Check

aws ec2 describe-instances --instance-ids <id> --query 'Reservations[].Instances[].[IamInstanceProfile.Arn,MetadataOptions.HttpEndpoint]'  # None, or HttpEndpoint disabled, is this cause

Fix

Attach an instance profile with only the permissions the server needs, and keep the metadata endpoint enabled with IMDSv2 required.

The SSO session reached its configured duration. Identity Center sessions last a set time (often 8 or 12 hours), after which you must sign in again.

Check

aws configure list  # confirms which profile is in use
aws sts get-caller-identity --profile <profile>  # fails with the same message

Fix

aws sso login --profile <profile>.

You signed out in the browser, or an administrator ended your session. That revokes the cached token too.

Check

aws sso login --profile <profile>  # if this works and the command then succeeds, the session had ended

Fix

Sign in again with aws sso login.

The profile in ~/.aws/config has no region line. The Region is read from the profile in use, so a profile with only credentials has none.

Check

aws configure get region --profile <profile>  # prints nothing: this is your cause

Fix

aws configure set region us-east-1 --profile <profile>, or pass --region us-east-1 on the command.

The code runs in a container, cron job or CI runner where neither AWS_REGION nor AWS_DEFAULT_REGION is set, and there is no config file.

Check

env | grep REGION  # nothing printed in that environment is this cause

Fix

Set AWS_DEFAULT_REGION (and AWS_REGION) in the image, job definition or workflow.

The code creates a client without a Region, using a profile that has credentials but no Region.

Check

aws configure list  # region shows <not set> for that profile

Fix

Name it when you create the client: boto3.client("ec2", region_name="us-east-1").

The profile name has a typo, or AWS_PROFILE is still set from another project or shell. The variable applies to every command until you unset it.

Check

echo "$AWS_PROFILE"  # a name you did not mean to use is this cause
aws configure list-profiles  # the names that exist

Fix

Use a name from aws configure list-profiles, or unset AWS_PROFILE.

In ~/.aws/config the section is written [dev]. There it must be [profile dev]; only ~/.aws/credentials uses a plain [dev], so the CLI does not see the profile.

Check

grep -n '^\[' ~/.aws/config ~/.aws/credentials  # [dev] in the config file is this cause

Fix

Rename the section in ~/.aws/config to [profile dev].

The command runs as another user, or in a container, with a different home directory, so it reads different files.

Check

whoami; echo "$HOME"  # not the user whose ~/.aws holds the profile

Fix

Create the profile for that user (aws configure --profile dev, or aws configure sso), or better, give the job its own role.

An Availability Zone was given as the Region (us-east-1a instead of us-east-1), or the Region has a typo. The CLI builds a host name that does not exist.

Check

aws configure get region  # anything ending in a letter, like us-east-1a, is this cause
nslookup sqs.eu-west-1.amazonaws.com  # NXDOMAIN for the host in the message means the name is wrong

Fix

aws configure set region us-east-1.

The machine is in a private subnet with no NAT gateway and no VPC endpoint for that service. The name resolves, but there is no route to the address.

Check

curl -sS -o /dev/null -w '%{http_code}\n' https://sqs.eu-west-1.amazonaws.com/  # hangs, then times out: no network path
aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=<subnet-id>  # no 0.0.0.0/0 route to a nat- gateway

Fix

Add a route to a NAT gateway, or an interface VPC endpoint for the service (gateway endpoint for S3 and DynamoDB).

A proxy or firewall is in the way, or DNS does not work on the machine.

Check

env | grep -io '^https\?_proxy'  # a proxy variable is set: the CLI must be able to use it
nslookup sts.amazonaws.com  # fails: DNS itself is broken

Fix

Set HTTPS_PROXY for the CLI, or allow the AWS endpoints through the firewall.

The service is not offered in that Region, so its endpoint does not exist there.

Check

aws ssm get-parameters-by-path --path /aws/service/global-infrastructure/services/<service>/regions --query 'Parameters[].Value'  # your Region missing is this cause

Fix

Use a Region where the service is available.

The machine's clock has drifted: a laptop after sleep, a VM restored from a snapshot, a container host without time sync. The request carries the wrong time, so AWS treats it as expired or not yet valid.

Check

date -u  # compare with the real UTC time: minutes off is this cause

Fix

Turn time sync on: sudo timedatectl set-ntp true, then run the command again. Changing the time zone does not help: signing uses UTC.

No time sync service is running (chrony or systemd-timesyncd), so the clock keeps drifting.

Check

timedatectl status  # System clock synchronized: no, or NTP service: inactive, is this cause

Fix

Enable chrony or systemd-timesyncd. On EC2, point chrony at the Amazon Time Sync Service, 169.254.169.123.

The call runs as another identity than you meant: another profile, leftover AWS_* variables, or the instance role instead of yours. That identity simply has no such permission.

Check

aws sts get-caller-identity  # if the Arn is not the identity you meant, this is your cause

Fix

Fix the credentials (profile, variables) rather than granting the wrong identity more.

The policy allows a different action, or the same action on a different resource. For S3, s3:ListBucket needs arn:aws:s3:::bucket while s3:GetObject needs arn:aws:s3:::bucket/*; an allow on one does not cover the other.

Check

aws iam simulate-principal-policy --policy-source-arn <arn> --action-names <action> --resource-arns <resource-arn>  # implicitDeny: no identity policy allows exactly this (resource policies are not included unless you pass one)

Fix

Allow exactly that action on exactly that resource, for example s3:ListBucket on arn:aws:s3:::nimbus-kitchen-reports. Not s3:* and not *.

A condition in the policy does not match the request (source IP, MFA, tags, Region). The statement exists but does not apply to this call.

Check

aws iam simulate-principal-policy --policy-source-arn <arn> --action-names <action> --resource-arns <resource-arn> --context-entries ContextKeyName=aws:SourceIp,ContextKeyValues=<ip>,ContextKeyType=ip  # allowed only with the right context: the condition is the cause

Fix

Make the request meet the condition, or correct the condition if it is wrong.

Across accounts, the other account's resource policy allows it but your own identity policy does not. Across accounts both sides must allow.

Check

aws sts get-caller-identity --query Account  # differs from the account in the resource ARN: check your own policy too

Fix

Add the allow to your identity policy as well, for that action and resource only.

in an identity-based policy: a guard-rail Deny on the user, one of its groups or the role, often with Resource "*" and no condition.

Check

aws sts get-caller-identity  # whose policies to read: for assumed-role/<role>/<session>, read the role
aws iam list-attached-role-policies --role-name <role>; aws iam list-role-policies --role-name <role>  # read each for an Effect Deny that matches (for a user: the -user- and -group- versions)

Fix

Narrow the Deny with a condition or a specific resource, rather than deleting it.

in a resource-based policy: the bucket, key, queue or secret policy denies, for example requests without TLS (aws:SecureTransport) or from outside a VPC endpoint.

Check

aws s3api get-bucket-policy --bucket <bucket> --query Policy --output text  # a Deny whose condition your request fails (or get-key-policy, get-resource-policy)

Fix

Make the request meet the condition (use HTTPS, go through the VPC endpoint), or narrow the Deny.

in a permissions boundary: the boundary set on the user or role contains a Deny for this action.

Check

aws iam get-role --role-name <role> --query Role.PermissionsBoundary  # the boundary policy to read

Fix

Change the boundary only if the guard rail is wrong; it is usually there on purpose.

in a session policy: a policy was passed with --policy or --policy-arns when the session was created, and it denies.

Check

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole --max-results 20  # requestParameters show the policy or policyArns passed for your session

Fix

Start the session again without that policy, or with one that allows the action.

in a VPC endpoint policy: the endpoint the request went through does not let it pass.

Check

aws ec2 describe-vpc-endpoints --query 'VpcEndpoints[].[VpcEndpointId,ServiceName,PolicyDocument]'  # a Deny, or no Allow, for your action or bucket

Fix

Allow that action and resource in the endpoint policy.

A permission on a related resource is missing: iam:PassRole for the instance profile, kms:CreateGrant for an encrypted volume, ec2:CreateTags for tags in the request. RunInstances checks all of them.

Check

aws sts decode-authorization-message --encoded-message <message> --query DecodedMessage --output text  # explicitDeny false, and context.action names the missing permission

Fix

Grant exactly context.action on context.resource from the decoded JSON.

An explicit deny matched, for example a guard rail on instance types or Regions.

Check

aws sts decode-authorization-message --encoded-message <message> --query DecodedMessage --output text  # explicitDeny true; matchedStatements shows which statement

Fix

Change the request to meet the guard rail, or narrow the deny if it is wrong.

The caller can create the resource but has no iam:PassRole at all. Creating an instance or function and handing it a role are two separate permissions.

Check

aws iam simulate-principal-policy --policy-source-arn <caller-arn> --action-names iam:PassRole --resource-arns <role-arn>  # implicitDeny is this cause

Fix

Allow iam:PassRole on that role's ARN, with an iam:PassedToService condition naming the service.

PassRole is allowed, but for a different role, or with an iam:PassedToService condition for another service.

Check

aws iam get-role-policy --role-name <caller-role> --policy-name <policy>  # check the PassRole Resource and its iam:PassedToService value

Fix

Add this role's ARN, or the right service (ec2.amazonaws.com, lambda.amazonaws.com), to that statement.

The trust policy does not name this caller: it names a different account or role, or a user that was deleted (it then shows as AIDA... or AROA..., an ID instead of an ARN).

Check

aws sts get-caller-identity  # the caller actually used
aws iam get-role --role-name <role> --query Role.AssumeRolePolicyDocument  # if that caller and its account are not in Principal, or it shows AIDA/AROA, this is your cause

Fix

Name the caller in the trust policy with aws iam update-assume-role-policy. A recreated user needs to be added again: the old ID will not match.

The trust policy has a condition the call does not meet, such as sts:ExternalId or aws:MultiFactorAuthPresent.

Check

aws iam get-role --role-name <role> --query 'Role.AssumeRolePolicyDocument.Statement[].Condition'  # a condition your call does not satisfy

Fix

Pass what it asks for: --external-id <id>, or sign in with MFA (--serial-number and --token-code).

The caller's own policy does not allow sts:AssumeRole on this role, and the role is in another account or its trust names only the account.

Check

aws iam simulate-principal-policy --policy-source-arn <caller-arn> --action-names sts:AssumeRole --resource-arns <role-arn>  # implicitDeny: the caller's side is missing (the trust policy is not part of this check)

Fix

Allow sts:AssumeRole on that one role ARN in the caller's policy.

The call passes session tags or a source identity, and the trust policy does not also allow sts:TagSession or sts:SetSourceIdentity.

Check

aws iam get-role --role-name <role> --query Role.AssumeRolePolicyDocument  # Action has only sts:AssumeRole, while your tool passes tags

Fix

Add sts:TagSession (or sts:SetSourceIdentity) to the trust statement's Action.

The sub condition names another repository, branch or environment. A job that uses an environment sends repo:owner/repo:environment:prod instead of the ref, and a pull request sends repo:owner/repo:pull_request.

Check

aws iam get-role --role-name <role> --query Role.AssumeRolePolicyDocument  # the sub value the role accepts
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRoleWithWebIdentity --max-results 5  # the failed call's userIdentity shows the subject the token carried; if it differs from sub, this is your cause

Fix

Set sub to exactly what the job sends, for example repo:owner/repo:ref:refs/heads/main or repo:owner/repo:environment:prod.

A wildcard is used in sub under StringEquals. Wildcards only work with StringLike, so repo:owner/repo:* under StringEquals matches nothing.

Check

aws iam get-role --role-name <role> --query Role.AssumeRolePolicyDocument  # a * inside a StringEquals block is this cause

Fix

Use StringLike for patterns, scoped to the repository and branch or environment, never repo:*.

The aud is not sts.amazonaws.com, or the Federated principal names an OIDC provider in another account or one that does not exist.

Check

aws iam list-open-id-connect-providers  # the provider for token.actions.githubusercontent.com must be in this account, and its ARN must match Principal.Federated

Fix

Create the provider in this account, and set the aud condition to sts.amazonaws.com.

The workflow lacks permissions: id-token: write, so it cannot request a token. The action then fails before it calls AWS, with a message about the id-token permission.

Check

grep -n -A3 'permissions:' .github/workflows/<workflow>.yml  # no id-token: write is this cause

Fix

Add permissions: id-token: write (and contents: read) to the job.

A Region restriction: an SCP denies requests outside approved Regions (a Deny on aws:RequestedRegion).

Check

aws organizations list-parents --child-id <account-id>  # the OU the account is in; SCPs on every OU up to the root apply
aws organizations list-policies-for-target --target-id <account-id> --filter SERVICE_CONTROL_POLICY  # from the management account or a delegated administrator; repeat for each OU, and look for aws:RequestedRegion

Fix

Use an allowed Region, or ask the platform team for an exception.

A guard rail blocks the action itself: disabling CloudTrail, leaving the organization, changing networking.

Check

aws organizations describe-policy --policy-id <policy-id> --query Policy.Content --output text  # a Deny naming your action

Fix

Work within the guard rail, or request an exception from the team that owns it. If you cannot read the SCPs, ask them which one applies.

An allow-list SCP, on the account or on any OU above it, does not list the service or action. Every level must allow it.

Check

aws organizations list-policies-for-target --target-id <ou-id> --filter SERVICE_CONTROL_POLICY  # an OU without FullAWSAccess, whose SCP does not list your service

Fix

Ask for the action to be added to the allow list at that level.

The key policy has no statement for the role, and no root-account statement delegating to IAM. IAM permissions alone then count for nothing.

Check

aws kms get-key-policy --key-id <key-id> --policy-name default --query Policy --output text  # neither your role nor arn:aws:iam::<account-id>:root is allowed the action: this is your cause

Fix

Add a key policy statement for the role, or the account-root statement and then an IAM allow, with aws kms put-key-policy.

The key policy delegates to IAM, but the role's own policy does not allow the action on this key. The message then says no identity-based policy allows.

Check

aws iam simulate-principal-policy --policy-source-arn <role-arn> --action-names kms:Decrypt --resource-arns <key-arn>  # implicitDeny is this cause

Fix

Allow kms:Decrypt (or the action named) on that key's ARN in the role's policy.

The data was encrypted with a different key than the one you allowed. You must be allowed the key that was actually used.

Check

aws s3api head-object --bucket <bucket> --key <key>  # SSEKMSKeyId: if it is not the key you allowed, this is your cause

Fix

Allow the key that was used, or re-encrypt the data with the intended key.

A condition in the key policy, such as kms:ViaService or kms:EncryptionContext, is not met by this request.

Check

aws kms get-key-policy --key-id <key-id> --policy-name default --query Policy --output text  # your statement has a Condition the call does not satisfy

Fix

Call through the service the condition names, or correct the condition.

Cross-account use, without both the key policy and the caller's IAM policy allowing it.

Check

aws sts get-caller-identity --query Account  # differs from the account in the key ARN

Fix

Allow the other account (or its role) in the key policy, and allow the key's ARN in the role's IAM policy.

The security group does not allow the port from your address. Security groups drop what they do not allow, with no reply.

Check

aws ec2 describe-security-groups --group-ids <sg-id> --query 'SecurityGroups[].IpPermissions'  # no rule for that port from your IP (or a security group you are in) is this cause

Fix

Allow that one port from your address or from the client's security group, not from 0.0.0.0/0. For SSH, use Session Manager instead.

The network ACL drops the request, or the reply. NACLs are stateless, so the return traffic to the client's ephemeral ports (1024-65535) needs its own outbound rule.

Check

aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=<subnet-id>  # a DENY before your ALLOW, or no outbound rule for 1024-65535, is this cause

Fix

Allow the port inbound and ephemeral ports outbound in the subnet's NACL.

The subnet has no route to an internet gateway, so replies cannot get back to the internet.

Check

aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=<subnet-id>  # no 0.0.0.0/0 route to an igw- is this cause (no row at all means it uses the main route table)

Fix

Add a 0.0.0.0/0 route to the VPC's internet gateway, or put the instance in a public subnet.

The instance has no public IP, or is stopped. There is then nothing at the address you are trying.

Check

aws ec2 describe-instances --instance-ids <id> --query 'Reservations[].Instances[].[State.Name,PublicIpAddress,SubnetId]'  # stopped, or no PublicIpAddress, is this cause

Fix

Start the instance, attach an Elastic IP, or reach it privately (Session Manager, a VPN, a load balancer). VPC Reachability Analyzer can trace the whole path for you.

The service is down or crashed, so nothing listens on the port.

Check

aws ssm start-session --target <instance-id>  # get a shell on the instance
sudo ss -tlnp  # no line for the port means nothing listens: this is your cause

Fix

Start the service and read its log for why it stopped (sudo journalctl -u <service> -n 50).

It listens on another port (8080, not 80).

Check

sudo ss -tlnp  # the process listens, but on a different port than you connect to

Fix

Connect to the port it uses, or change the load balancer or client to it.

It listens only on 127.0.0.1, so it accepts connections from the instance itself and refuses the rest.

Check

sudo ss -tlnp  # 127.0.0.1:80 is this cause; 0.0.0.0:80 or *:80 would accept from outside

Fix

Make the app bind to 0.0.0.0 (or the instance's private address).

A host firewall on the instance (iptables or nftables with REJECT) refuses the connection.

Check

sudo iptables -S | grep -i reject; sudo nft list ruleset | grep -i reject  # a REJECT rule covering the port

Fix

Allow the port in the host firewall, rather than turning the firewall off.

The private route table has no 0.0.0.0/0 route to a NAT gateway.

Check

aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=<subnet-id>  # no 0.0.0.0/0 to a nat- target is this cause

Fix

Create a NAT gateway in a public subnet with an Elastic IP, and route 0.0.0.0/0 from the private route table to it.

The NAT gateway exists but sits in a private subnet, or is not available. A NAT gateway needs a route to the internet gateway itself.

Check

aws ec2 describe-nat-gateways --query 'NatGateways[].[NatGatewayId,State,SubnetId]'  # State not available, or a SubnetId whose route table has no igw- route, is this cause

Fix

Create the NAT gateway in a public subnet and point the private route at the new one.

Outbound traffic is blocked by the instance's security group or the subnet's NACL.

Check

aws ec2 describe-security-groups --group-ids <sg-id> --query 'SecurityGroups[].IpPermissionsEgress'  # no outbound rule for 443

Fix

Allow outbound 443, and the NACL's ephemeral ports inbound for the replies.

The instance role lacks AmazonSSMManagedInstanceCore, or the instance has no role. The agent cannot register without it.

Check

aws ec2 describe-instances --instance-ids <id> --query 'Reservations[].Instances[].IamInstanceProfile'  # empty means no role
aws iam list-attached-role-policies --role-name <role>  # AmazonSSMManagedInstanceCore missing is this cause

Fix

Attach AmazonSSMManagedInstanceCore to the instance role. The agent picks it up within a few minutes.

There is no route to the SSM endpoints: a private subnet with no NAT and no interface endpoints.

Check

aws ec2 describe-vpc-endpoints --filters Name=vpc-id,Values=<vpc-id> --query 'VpcEndpoints[].ServiceName'  # no ssm, ssmmessages and ec2messages, and no NAT route, is this cause

Fix

Give the subnet a NAT route, or interface endpoints for ssm, ssmmessages and ec2messages (with a security group that allows 443 from the instance).

The instance is stopped, or the agent is not installed or not running.

Check

aws ssm describe-instance-information --filters Key=InstanceIds,Values=<id>  # no entry, or PingStatus ConnectionLost, is this cause

Fix

Start the instance. Use an AMI with the agent (Amazon Linux, recent Ubuntu), or install it from user data.

One VPC's route table has no route to the other VPC's CIDR through the peering. Traffic goes one way and the reply has no way back.

Check

aws ec2 describe-route-tables --filters Name=vpc-id,Values=<vpc-id>  # in both VPCs, look for the other CIDR with a pcx- target; missing on either side is this cause

Fix

Add a route to the other VPC's CIDR via the peering connection in both VPCs' route tables.

Security groups do not allow the other VPC's addresses.

Check

aws ec2 describe-security-groups --group-ids <sg-id>  # no rule for the other VPC's CIDR (or its security group, same Region)

Fix

Allow the other VPC's CIDR, or its security group when the peering is in the same Region.

The VPCs' CIDRs overlap. A peering between overlapping ranges cannot be created.

Check

aws ec2 describe-vpc-peering-connections --query 'VpcPeeringConnections[].Status'  # failed, Overlapping CIDRs

Fix

Re-address one VPC, or connect them through a private NAT or PrivateLink.

You expect traffic to pass through a middle VPC. Peering is not transitive: A to B and B to C does not give A to C.

Check

aws ec2 describe-vpc-peering-connections --query 'VpcPeeringConnections[].[RequesterVpcInfo.VpcId,AccepterVpcInfo.VpcId]'  # no direct peering between the two VPCs

Fix

Peer the two VPCs directly, or use a Transit Gateway.

A popular or large type is short in one AZ.

Check

aws ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values=<type>  # other AZs that offer the type

Fix

Launch in another AZ, or a similar type (c6i.8xlarge instead of c7i.8xlarge).

A capacity spike across the Region.

Check

aws ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values=<type> --region <region>  # the type is offered, yet every AZ fails: retry later

Fix

Retry with backoff, or use an Auto Scaling group with several types and AZs. For critical workloads, buy a capacity reservation.

Forgotten instances are still running and using the quota.

Check

aws ec2 describe-instances --filters Name=instance-state-name,Values=running --query 'Reservations[].Instances[].[InstanceId,InstanceType,Tags]'  # instances nobody needs

Fix

Stop or terminate what you do not need.

The account is new or the quota is low for what you run.

Check

aws service-quotas get-service-quota --service-code ec2 --quota-code L-1216C47A  # Value: the vCPU limit for standard (A, C, D, H, I, M, R, T, Z) instances

Fix

Request an increase: aws service-quotas request-service-quota-increase --service-code ec2 --quota-code L-1216C47A --desired-value <vcpus>.

System check failed: hardware or network trouble on the AWS host.

Check

aws ec2 describe-instance-status --instance-ids <id>  # SystemStatus impaired is this cause

Fix

Stop and start the instance (not reboot): that moves it to a new host. Instance store data is lost.

Instance check failed: the operating system is not healthy (kernel panic, full disk, broken network configuration).

Check

aws ec2 describe-instance-status --instance-ids <id>  # InstanceStatus impaired, SystemStatus ok
aws ec2 get-console-output --instance-id <id> --latest  # the boot log shows the panic, the full disk or the failed network

Fix

Fix the OS from what the console shows: free disk, revert the change, or attach the volume to another instance and repair it.

DesiredCapacity already equals MaxSize, so there is no room to add instances.

Check

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names <asg> --query 'AutoScalingGroups[].[MinSize,DesiredCapacity,MaxSize]'  # Desired equal to Max is this cause

Fix

Raise MaxSize: aws autoscaling update-auto-scaling-group --auto-scaling-group-name <asg> --max-size <n>.

The alarm watches the wrong dimension, often an old group name, so it never gets data.

Check

aws cloudwatch describe-alarms --alarm-names <alarm> --query 'MetricAlarms[].[StateValue,Dimensions]'  # INSUFFICIENT_DATA, or a dimension that is not this group, is this cause

Fix

Point the alarm at the right dimensions, or use a target tracking policy, which manages its own alarms.

The alarm has no action, or the policy is disabled.

Check

aws cloudwatch describe-alarms --alarm-names <alarm> --query 'MetricAlarms[].[AlarmActions,ActionsEnabled]'  # empty actions or false is this cause
aws autoscaling describe-scaling-activities --auto-scaling-group-name <asg>  # no recent activity when the alarm fired

Fix

Attach the scaling policy's ARN as the alarm action and enable actions.

The instance is stopped, stopping or terminated.

Check

aws ec2 describe-instances --instance-ids <id> --query 'Reservations[].Instances[].State.Name'  # a state the action cannot start from

Fix

Start it first, or wait for the transition to finish: aws ec2 wait instance-running --instance-ids <id>.

The wrong user name for the AMI. Each AMI has its own default user, and the key is only installed for that one.

Check

aws ec2 describe-images --image-ids <ami-id> --query 'Images[].[Name,Description]'  # Amazon Linux: ec2-user; Ubuntu: ubuntu; Debian: admin

Fix

Connect as the AMI's user: ssh -i web-key.pem ec2-user@<public-ip>.

The wrong private key, or the instance was launched with another key pair or none.

Check

aws ec2 describe-instances --instance-ids <instance-id> --query 'Reservations[].Instances[].KeyName'  # not the key pair your file belongs to, or None, is this cause
ssh -v -i web-key.pem ec2-user@<public-ip>  # Offering public key ... then no 'Server accepts key' line

Fix

Use the matching key. Lost it? Connect with Session Manager or EC2 Instance Connect and add a new public key to ~/.ssh/authorized_keys.

The key file can be read by other users, so ssh ignores it.

Check

ls -l web-key.pem  # anything wider than -r-------- (400) or -rw------- (600)

Fix

chmod 400 web-key.pem.

~/.ssh/authorized_keys on the instance was changed or has wrong permissions.

Check

aws ssm start-session --target <instance-id>  # then: ls -l ~ec2-user/.ssh  (700 for the folder, 600 for the file)

Fix

Restore the public key and permissions through Session Manager.

The key pair was created in another Region.

Check

aws configure get region  # the Region the request went to
aws ec2 describe-key-pairs --query 'KeyPairs[].KeyName' --region <other-region>  # the key is there, not here

Fix

Launch in that Region with --region <region>, or import your public key into this Region: aws ec2 import-key-pair --key-name web-key --public-key-material fileb://~/.ssh/id_ed25519.pub.

A typo in the name, or the key pair was deleted.

Check

aws ec2 describe-key-pairs --query 'KeyPairs[].KeyName'  # the names that exist in this Region

Fix

Use an existing name, or launch without a key and connect with Session Manager.

The ID was copied from a tutorial or from another Region.

Check

aws ec2 describe-images --image-ids <ami-id> --region <region>  # found in another Region but not this one is this cause

Fix

Look up this Region's current ID: aws ssm get-parameters --names /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 --query 'Parameters[0].Value'.

The AMI was deregistered.

Check

aws ec2 describe-images --owners self --query 'Images[].ImageId'  # your own AMI is no longer listed

Fix

Use a current AMI, or rebuild it.

A private AMI in another account that is not shared with you.

Check

aws ec2 describe-image-attribute --image-id <ami-id> --attribute launchPermission  # run in the owning account: your account not listed is this cause

Fix

Ask the owner to share it, or copy your own AMI into the Region: aws ec2 copy-image --source-image-id <ami-id> --source-region <region> --name web --region <target-region>.

A typo in code or configuration.

Check

aws s3 ls | grep <part-of-name>  # the real name differs from the one in the message

Fix

Use the right name in configuration.

The bucket was deleted.

Check

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteBucket  # a DeleteBucket for that name

Fix

Create it again (the name may be taken by then), or point the app at the new bucket.

An environment variable points at another stage's bucket.

Check

env | grep -i bucket  # a name for another stage (dev in prod)

Fix

Pass bucket names from infrastructure as code outputs rather than typing them.

The name is short or generic, so someone took it first.

Check

aws s3api head-bucket --bucket <name>  # 403: it exists in another account; 404: the name is free

Fix

Choose a unique name, for example organization, purpose, account and Region: nimbus-kitchen-reports-111122223333-eu-west-1.

No identity policy allows s3:GetObject on that key, or the call runs as another identity than you think.

Check

aws sts get-caller-identity  # the identity S3 judged
aws iam simulate-principal-policy --policy-source-arn <arn> --action-names s3:GetObject --resource-arns arn:aws:s3:::<bucket>/<key>  # implicitDeny is this cause

Fix

Grant s3:GetObject on arn:aws:s3:::<bucket>/* (or the prefix you need).

The key does not exist and you lack s3:ListBucket. S3 then answers 403 instead of 404, so a missing object looks like a permission problem.

Check

aws s3 ls s3://<bucket>/<prefix>/  # with an identity that can list: the key is not there

Fix

Fix the key name, and grant s3:ListBucket on the bucket so a missing key shows as 404.

The object is owned by another account (Object Ownership is ObjectWriter and another account uploaded it), so the bucket owner's policies do not apply to it.

Check

aws s3api get-bucket-ownership-controls --bucket <bucket>  # ObjectWriter or BucketOwnerPreferred, with cross-account uploads, is this cause

Fix

Set Object Ownership to BucketOwnerEnforced, and copy existing objects over themselves to take ownership.

The object is KMS-encrypted and the key policy does not allow you to decrypt.

Check

aws s3api head-object --bucket <bucket> --key <key>  # with an identity that can read it: SSEKMSKeyId names the key to check

Fix

Allow kms:Decrypt for your role in that key's policy.

A bucket policy Deny, or a VPC endpoint policy, blocks the request.

Check

aws s3api get-bucket-policy --bucket <bucket> --query Policy --output text  # a Deny whose condition you fail (aws:SecureTransport, aws:SourceVpce)

Fix

Meet the condition, or narrow the Deny.

aws s3api create-bucket --region eu-west-1 was run without --create-bucket-configuration.

Check

aws configure get region  # anything other than us-east-1, with no LocationConstraint in your command, is this cause

Fix

Add --create-bucket-configuration LocationConstraint=<region>, or use aws s3 mb s3://<bucket> --region <region>, which adds it for you.

A typo or a different prefix. Keys are exact and case-sensitive, and reports/2026/a.csv is not reports/2026//a.csv.

Check

aws s3 ls s3://<bucket>/<prefix>/  # the real key differs from the one requested

Fix

Use the exact key.

The object was deleted. In a versioned bucket it is hidden behind a delete marker.

Check

aws s3api list-object-versions --bucket <bucket> --prefix <key>  # a DeleteMarkers entry with IsLatest true is this cause

Fix

Delete the delete marker (aws s3api delete-object --bucket <bucket> --key <key> --version-id <marker-id>) or copy the previous version back.

The bucket has no CORS configuration, so S3 sends no Access-Control-Allow-Origin header.

Check

aws s3api get-bucket-cors --bucket <bucket>  # NoSuchCORSConfiguration is this cause

Fix

Add one with aws s3api put-bucket-cors, allowing your site's origin and the methods it uses.

AllowedOrigins does not match your site exactly, including scheme and port.

Check

curl -s -D - -o /dev/null -H 'Origin: https://shop.example.com' '<object or presigned URL>' | grep -i access-control  # no Access-Control-Allow-Origin line for your origin is this cause

Fix

Put the exact origin in AllowedOrigins, for example https://shop.example.com, rather than "*".

The method or headers are not allowed, for example a PUT with Content-Type. The browser's preflight (OPTIONS) is then refused.

Check

curl -s -D - -o /dev/null -X OPTIONS -H 'Origin: https://shop.example.com' -H 'Access-Control-Request-Method: PUT' -H 'Access-Control-Request-Headers: content-type' '<url>'  # 403, or no allow headers, is this cause

Fix

Add the method to AllowedMethods and the header to AllowedHeaders.

The profile's Region differs from the bucket's.

Check

aws s3api get-bucket-location --bucket <bucket>  # the bucket's Region; an empty or null LocationConstraint means us-east-1
aws configure get region  # differs from the bucket's: this is your cause

Fix

Send the request to the bucket's Region: aws s3 ls s3://<bucket> --region eu-west-1.

An SDK client was created for a fixed Region, or a hard-coded endpoint URL points at another Region.

Check

grep -rn "region_name\|endpoint_url" .  # a Region or endpoint that is not the bucket's

Fix

Create the client for the bucket's Region, and keep each bucket's Region in your config next to its name.

The task runs in private subnets with no NAT gateway and no VPC endpoints, so it has no way to reach ECR.

Check

aws ecs describe-tasks --cluster <cluster> --tasks <task-arn> --query 'tasks[].stoppedReason'  # ResourceInitializationError with i/o timeout
aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=<subnet-id>  # no 0.0.0.0/0 to a nat- gateway, and no endpoints below: this is your cause

Fix

Route the subnets through a NAT gateway, or add interface endpoints for ecr.api and ecr.dkr (and logs, secretsmanager if used) and a gateway endpoint for S3.

There are endpoints for ecr.api and ecr.dkr, but no S3 gateway endpoint. The image manifest is fetched, then the layer download from S3 times out.

Check

aws ec2 describe-vpc-endpoints --filters Name=vpc-id,Values=<vpc-id> --query 'VpcEndpoints[].[ServiceName,VpcEndpointType]'  # ecr endpoints present, com.amazonaws.<region>.s3 Gateway missing

Fix

Add the S3 gateway endpoint and associate it with the subnets' route tables.

The task is in a public subnet but has no public IP, so the internet gateway route cannot be used.

Check

aws ecs describe-services --cluster <cluster> --services <service> --query 'services[].networkConfiguration.awsvpcConfiguration.assignPublicIp'  # DISABLED in a public subnet is this cause

Fix

Prefer private subnets with NAT or endpoints. Otherwise set assignPublicIp to ENABLED.

The interface endpoints' security group does not allow 443 from the tasks.

Check

aws ec2 describe-vpc-endpoints --query 'VpcEndpoints[].Groups'  # then describe-security-groups on that group: no inbound 443 from the task's security group

Fix

Allow 443 on the endpoint security group from the tasks' security group.

AmazonECSTaskExecutionRolePolicy is not attached to the execution role, or the task definition has no execution role.

Check

aws ecs describe-task-definition --task-definition <family:revision> --query 'taskDefinition.[executionRoleArn,taskRoleArn]'  # executionRoleArn empty or not the role you think
aws iam list-attached-role-policies --role-name <execution-role>  # AmazonECSTaskExecutionRolePolicy missing is this cause

Fix

Attach arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy to the execution role, then force a new deployment.

The permissions were added to the task role instead. The task role is for your application's calls, after the container starts.

Check

aws iam list-attached-role-policies --role-name <task-role>  # the ECR permissions sit here, not on the execution role

Fix

Move them to the execution role.

The image is in another account and its ECR repository policy does not allow this role.

Check

aws ecr get-repository-policy --repository-name <repo> --registry-id <account-id>  # run in the owning account: no statement for your execution role

Fix

Allow ecr:BatchGetImage and ecr:GetDownloadUrlForLayer for the role in the repository policy.

There is no access entry for your role.

Check

aws sts get-caller-identity  # for assumed-role/<role>/<session>, the role is what the access entry must name
aws eks list-access-entries --cluster-name <cluster>  # that role's ARN missing is this cause

Fix

aws eks create-access-entry --cluster-name <cluster> --principal-arn <role-arn>, then aws eks associate-access-policy with only the access it needs.

kubeconfig gets its token with a different profile, role, cluster or Region than you think.

Check

kubectl config view --minify -o jsonpath='{.users[0].user.exec.args}'  # the cluster, Region and profile or role kubectl gets its token for

Fix

Run aws eks update-kubeconfig --name <cluster> --region <region> --profile <profile> with the identity that has access.

The access entry or aws-auth mapping names the session ARN (arn:aws:sts::...:assumed-role/...) instead of the role ARN (arn:aws:iam::...:role/...). The session ARN never matches.

Check

aws eks list-access-entries --cluster-name <cluster>  # an entry with arn:aws:sts:: is this cause

Fix

Use the role ARN.

The cluster uses only the aws-auth ConfigMap, and the identity that created it was deleted, so nobody can sign in.

Check

aws eks describe-cluster --name <cluster> --query cluster.accessConfig  # authenticationMode CONFIG_MAP

Fix

Switch the authentication mode to API_AND_CONFIG_MAP (aws eks update-cluster-config --access-config authenticationMode=API_AND_CONFIG_MAP) and create access entries.

The access entry exists but has no access policy associated.

Check

aws eks list-associated-access-policies --cluster-name <cluster> --principal-arn <role-arn>  # empty is this cause

Fix

aws eks associate-access-policy --cluster-name <cluster> --principal-arn <role-arn> --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy --access-scope type=cluster.

The policy is scoped to some namespaces and you asked about another, or about all of them (-A).

Check

aws eks list-associated-access-policies --cluster-name <cluster> --principal-arn <role-arn>  # accessScope type namespace, without the one you used
kubectl auth can-i list pods -n <namespace>  # yes here, no with -A

Fix

Add the namespace to the scope, or use cluster scope for read-only on-call access.

The entry maps you to Kubernetes groups, and no RoleBinding or ClusterRoleBinding gives those groups the permission.

Check

kubectl auth can-i --list  # what your identity may do; the verb and resource are missing

Fix

Create a RoleBinding for the group with the least role that works.

The login was never run, or was for another account or Region than the image tag.

Check

grep -o '[0-9]\{12\}\.dkr\.ecr\.[a-z0-9-]*\.amazonaws\.com' ~/.docker/config.json  # registry addresses you are logged in to: the tag's address missing is this cause

Fix

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 111122223333.dkr.ecr.us-east-1.amazonaws.com, with the account ID and Region from the image tag.

The login is more than 12 hours old.

Check

docker pull <account-id>.dkr.ecr.<region>.amazonaws.com/<repo>:<tag>  # denied: Your authorization token has expired

Fix

Log in again, or use the Amazon ECR credential helper so Docker fetches tokens itself.

Docker runs with sudo, which uses root's Docker login, not yours.

Check

sudo ls /root/.docker/config.json  # missing, or without the registry, while you logged in as yourself

Fix

Log in and push as the same user, or add yourself to the docker group.

Your identity cannot get a token or push to the repository.

Check

aws ecr get-authorization-token --query 'authorizationData[].expiresAt'  # AccessDenied: you lack ecr:GetAuthorizationToken

Fix

Allow ecr:GetAuthorizationToken (Resource "*"), plus push permissions on that repository only.

The certificate was requested in the app's Region (eu-west-1, for example). CloudFront cannot see it.

Check

aws acm list-certificates --region us-east-1  # your certificate missing here is this cause

Fix

Request (or import) the certificate in us-east-1 and use that ARN in ViewerCertificate.

The certificate is still PENDING_VALIDATION.

Check

aws acm describe-certificate --region us-east-1 --certificate-arn <arn> --query Certificate.Status  # anything but ISSUED

Fix

Create its validation record (see the ACM entry), then update the distribution.

It does not cover every alias on the distribution.

Check

aws acm describe-certificate --region us-east-1 --certificate-arn <arn> --query Certificate.SubjectAlternativeNames  # an alias not listed

Fix

Request a certificate that covers all the aliases.

The CNAME from DomainValidationOptions was never created.

Check

aws acm describe-certificate --certificate-arn <arn> --query 'Certificate.DomainValidationOptions[].ResourceRecord'  # the record ACM wants
dig +short <validation record name> CNAME  # nothing returned is this cause

Fix

Create that exact CNAME in the domain's hosted zone (change-resource-record-sets UPSERT). ACM issues within minutes.

It was created in the wrong hosted zone, or with a typo.

Check

dig +short <validation record name> CNAME  # a different value, or nothing, though you created it

Fix

Create it in the zone the domain is delegated to, with the exact name and value.

The domain's delegation is broken, so public DNS cannot see the zone.

Check

dig +short NS <domain>  # not the hosted zone's four name servers

Fix

Fix the registrar's name servers (see the SERVFAIL entry).

The files keep the same name across releases and have long TTLs, so edges keep the old copy.

Check

curl -sI https://<site>/index.html | grep -i '^\(x-cache\|age\|cache-control\)'  # Hit with a large Age is this cause

Fix

aws cloudfront create-invalidation --distribution-id <id> --paths /index.html (or "/*").

The deploy ran no invalidation.

Check

aws cloudfront list-invalidations --distribution-id <dist-id>  # none since the deploy

Fix

Add an invalidation of the entry points to the deploy step.

The distribution still uses the default *.cloudfront.net certificate.

Check

aws cloudfront get-distribution-config --id <dist-id> --query DistributionConfig.ViewerCertificate  # CloudFrontDefaultCertificate true is this cause

Fix

Attach a us-east-1 ACM certificate that covers the name.

The name is missing from the distribution's aliases or from the certificate.

Check

aws cloudfront get-distribution-config --id <dist-id> --query DistributionConfig.Aliases  # your name not listed

Fix

Add the name as an alias, with a certificate that covers it.

DNS points at the wrong distribution or load balancer.

Check

dig +short <name>  # not your distribution's d123.cloudfront.net address

Fix

Point the record (an alias) at the right distribution.

Targets are registered on a port where nothing listens.

Check

aws elbv2 describe-target-groups --names <tg> --query 'TargetGroups[].Port'  # the port traffic goes to
aws elbv2 describe-target-health --target-group-arn <tg-arn>  # unhealthy with Target.FailedHealthChecks, and nothing listens on that port on the instance

Fix

Register the targets on the port the app uses.

The application crashed or closed the connection mid-request.

Check

aws elbv2 describe-target-health --target-group-arn <tg-arn>  # healthy, yet 502s come and go

Fix

Read the application's logs for crashes at those times and fix the app.

The app's keep-alive timeout is shorter than the load balancer's idle timeout, so the app closes connections the load balancer still reuses.

Check

aws elbv2 describe-load-balancer-attributes --load-balancer-arn <alb-arn> --query "Attributes[?Key=='idle_timeout.timeout_seconds']"  # longer than the app's keep-alive (often 5 s in Node.js) is this cause

Fix

Set the app's keep-alive above the load balancer's idle timeout.

The target group has no registered targets.

Check

aws elbv2 describe-target-health --target-group-arn <tg-arn>  # an empty list is this cause

Fix

Register targets: aws elbv2 register-targets --target-group-arn <tg-arn> --targets Id=<instance-id>.

The listener rule points at a different, empty target group.

Check

aws elbv2 describe-listeners --load-balancer-arn <alb-arn>  # the TargetGroupArn in DefaultActions; then describe-rules for each listener

Fix

Point the rule at the target group that has targets.

The Auto Scaling group is attached to another target group.

Check

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names <asg> --query 'AutoScalingGroups[].TargetGroupARNs'  # not the listener's target group

Fix

Attach the Auto Scaling group to the right target group.

The target's security group does not allow the load balancer's security group.

Check

aws ec2 describe-security-groups --group-ids <target-sg>  # no inbound rule for the target port from the load balancer's security group is this cause

Fix

Allow the target port from the load balancer's security group.

A NACL drops the traffic between the load balancer's subnets and the targets.

Check

aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=<subnet-id>  # a DENY or no rule for the port and the ephemeral ports

Fix

Allow the port and the ephemeral ports in both directions.

The app takes longer than the idle timeout to answer.

Check

aws elbv2 describe-load-balancer-attributes --load-balancer-arn <alb-arn> --query "Attributes[?Key=='idle_timeout.timeout_seconds']"  # shorter than your slowest request

Fix

Make the request faster, or raise the idle timeout for it.

The health check path does not exist on the app.

Check

aws elbv2 describe-target-groups --names <tg> --query 'TargetGroups[].[HealthCheckPath,HealthCheckPort,Matcher.HttpCode]'  # the path being checked
curl -s -o /dev/null -w '%{http_code}\n' http://<target-ip>:<port><path>  # 404 is this cause

Fix

Point the health check at a path that returns 200 when the app can serve.

The app redirects on that path (301 or 302), for example to HTTPS or a login page.

Check

curl -s -o /dev/null -w '%{http_code}\n' http://<target-ip>:<port><path>  # 301 or 302 is this cause

Fix

Use a path that answers 200 without redirecting.

The health check uses the wrong port.

Check

aws elbv2 describe-target-groups --names <tg> --query 'TargetGroups[].HealthCheckPort'  # not the port the app listens on

Fix

Set it to traffic-port or the app's port. Do not widen the matcher to hide failures.

The registrar points at the name servers of a deleted or different hosted zone.

Check

aws route53domains get-domain-detail --domain-name <domain> --query Nameservers  # the name servers the registrar gives out
aws route53 get-hosted-zone --id <zone-id> --query DelegationSet.NameServers  # different from the line above: this is your cause

Fix

Update the registrar's name servers to the hosted zone's four.

DNSSEC is misconfigured: a DS record at the registrar that does not match the zone's signing key.

Check

dig +dnssec <domain> @1.1.1.1  # SERVFAIL here but an answer with +cd (checking disabled) points to DNSSEC

Fix

Fix the DS record, or remove it before you change or disable signing.

A records hold a load balancer's or an old instance's IP. Load balancer IPs change, and instances get new ones.

Check

aws route53 list-resource-record-sets --hosted-zone-id <zone-id> --query "ResourceRecordSets[?Name=='<name>.']"  # an A record with ResourceRecords instead of AliasTarget is this cause

Fix

Use an alias record to the load balancer (or a CNAME below the apex).

Resolvers still cache the old answer until its TTL ends.

Check

dig <name> @<one-of-the-zone-name-servers>  # the new address here, but the old one from your resolver: cached

Fix

Wait for the TTL to end. Before a planned change, lower the TTL first.

The database's security group does not allow the client's security group on the database port.

Check

aws rds describe-db-instances --db-instance-identifier <db> --query 'DBInstances[].VpcSecurityGroups'  # the database's security group
aws ec2 describe-security-groups --group-ids <db-sg> --query 'SecurityGroups[].IpPermissions'  # no rule for 5432 or 3306 from the client's security group is this cause

Fix

Allow the port from the client's security group, never from 0.0.0.0/0.

The client is outside the VPC and the database is private.

Check

aws rds describe-db-instances --db-instance-identifier <db> --query 'DBInstances[].[PubliclyAccessible,Endpoint.Address]'  # false, and you connect from a laptop or another network

Fix

Connect from inside the VPC (Session Manager port forwarding, a VPN), rather than making the database public.

NACLs between the subnets drop the traffic.

Check

aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=<db-subnet-id>  # no ALLOW for the port in, or the ephemeral ports out

Fix

Allow the database port and the ephemeral ports between the subnets.

The wrong port: 3306 is MySQL, 5432 is PostgreSQL, and an instance can use a custom port.

Check

aws rds describe-db-instances --db-instance-identifier <db> --query 'DBInstances[].[Engine,Endpoint.Port]'  # a different port than the one in the message is this cause

Fix

Use the endpoint's port.

The instance is rebooting, stopped or failing over.

Check

aws rds describe-db-instances --db-instance-identifier <db> --query 'DBInstances[].DBInstanceStatus'  # anything but available

Fix

Wait for available, or start it: aws rds start-db-instance --db-instance-identifier <db>.

The host name resolves to something other than the database, such as an old endpoint or localhost.

Check

nslookup <endpoint>  # not the instance's current address

Fix

Use the endpoint from describe-db-instances in your configuration.

The password is wrong or old: the secret was rotated, the master password was reset, or the app uses a copy from a config file. The server finds the account but the password does not match it.

Check

python3 -c "import os; print(repr(os.environ.get('DB_USER')), bool(os.environ.get('DB_PASSWORD')))"  # in the app's environment: the user it sends, and whether a password reaches the process, without printing it
aws secretsmanager describe-secret --secret-id <secret-arn> --query '[LastChangedDate,LastRotatedDate]'  # changed after the app last read it: this is your cause

Fix

Use the current password from its secret, read by the app at start-up (and again after a failed login), not a copy in a config file.

The master password is managed in Secrets Manager, and you used the one from creation time. RDS changes it on rotation, so the old one stops working.

Check

aws rds describe-db-instances --db-instance-identifier <db> --query 'DBInstances[].[MasterUsername,MasterUserSecret.SecretArn]'  # a SecretArn means Secrets Manager holds the master password

Fix

Read it from that secret. Lost it entirely? aws secretsmanager rotate-secret --secret-id <secret-arn>. Without a managed secret: aws rds modify-db-instance --db-instance-identifier <db> --master-user-password '<new-password>' --apply-immediately.

No account exists for that user at that host. MySQL accounts are user@host, so app from 10.0.12.77 only gets in if one of app's host patterns matches that address.

Check

SELECT user, host, plugin, ssl_type FROM mysql.user WHERE user IN ('app', '');  -- as the master user: if no host for app matches the address in the message, this is your cause

Fix

Create an account for the network the app connects from, not '%': CREATE USER 'app'@'10.0.%' IDENTIFIED BY '<password>';, then grant only what it needs.

Another account matched first. MySQL picks the account by its host-matching rules (the most specific host first), so a more specific host row for the same user, or an anonymous '' account, can win and carry a different password.

Check

mysql -h db.example.internal -P 3306 -u app -p -e 'SELECT USER(), CURRENT_USER();'  # where a login works: USER() is the identity the client supplied (user@client-host as the server sees it); CURRENT_USER() is the account actually used to authenticate. If they differ, or CURRENT_USER() is ''@..., another account matched
SELECT user, host FROM mysql.user WHERE user IN ('app', '') ORDER BY host;  -- two or more rows that could match your address

Fix

Fix that row's password, or drop it if nothing needs it. Adding yet another app row does not help while a more specific one still wins.

The user name is wrong, or DB_USER or DB_PASSWORD is empty in the app's environment. With an empty password the message says using password: NO.

Check

python3 -c "import os; print(repr(os.environ.get('DB_USER')), bool(os.environ.get('DB_PASSWORD')))"  # None or '' for the user, or False for the password, is this cause

Fix

Set both from the secret in the app's configuration (task definition, function settings), by reference rather than by value.

The account authenticates another way: it was set up for IAM database authentication, or requires TLS. The server rejects a plain password for it with the same error.

Check

SELECT user, host, plugin, ssl_type FROM mysql.user WHERE user = 'app';  -- MySQL: AWSAuthenticationPlugin means IAM tokens only; ssl_type ANY or X509 means TLS is required
SELECT pg_has_role('app', 'rds_iam', 'member');  -- PostgreSQL: true means it logs in with IAM tokens only

Fix

IAM user: connect with a token (see RDS IAM database authentication), or use a separate user with a password. TLS required: connect with TLS (see RDS requires TLS).

PostgreSQL: the role does not exist, or its password has passed VALID UNTIL. PostgreSQL gives the same message for both, so it does not reveal which users exist.

Check

SELECT rolname, rolvaliduntil FROM pg_roles WHERE rolname = 'app';  -- no row: no such role; a past rolvaliduntil: the password expired

Fix

Create the role (CREATE ROLE app LOGIN PASSWORD '<password>';), or set a new password and expiry: ALTER ROLE app PASSWORD '<password>' VALID UNTIL '<date>';. Do not switch the app to the master user to get past the error: it hides the cause and gives the app every privilege.

The token is more than 15 minutes old.

Check

grep -n generate-db-auth-token <your-code>  # the token made once at start-up and reused is this cause

Fix

Make a fresh token just before each connection opens.

The token was made for another host name, port, Region or user than the connection uses.

Check

aws rds describe-db-instances --db-instance-identifier <db> --query 'DBInstances[].[Endpoint.Address,Endpoint.Port]'  # compare with the --hostname, --port and --username used for the token

Fix

aws rds generate-db-auth-token --hostname <endpoint> --port 5432 --region <region> --username app, with exactly the values the connection uses.

The IAM identity has no rds-db:connect on arn:aws:rds-db:<region>:<account-id>:dbuser:<DbiResourceId>/<user>. The token is signed, but the database rejects it.

Check

aws iam simulate-principal-policy --policy-source-arn <role-arn> --action-names rds-db:connect --resource-arns arn:aws:rds-db:<region>:<account-id>:dbuser:<DbiResourceId>/app  # implicitDeny is this cause

Fix

Allow rds-db:connect on that dbuser ARN (it uses DbiResourceId, not the instance name).

IAM database authentication is not turned on for the instance.

Check

aws rds describe-db-instances --db-instance-identifier <db> --query 'DBInstances[].[IAMDatabaseAuthenticationEnabled,DbiResourceId]'  # false is this cause

Fix

aws rds modify-db-instance --db-instance-identifier <db> --enable-iam-database-authentication --apply-immediately.

The database user is not set up for IAM authentication.

Check

SELECT pg_has_role('app', 'rds_iam', 'member');  -- PostgreSQL: false is this cause
SELECT user, host, plugin FROM mysql.user WHERE user = 'app';  -- MySQL: a plugin other than AWSAuthenticationPlugin is this cause

Fix

PostgreSQL: GRANT rds_iam TO app;. MySQL: CREATE USER app IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';.

The client or driver connects without TLS: sslmode=disable, or a library that does not use TLS unless told to.

Check

grep -rn "sslmode\|ssl_mode\|ssl=" <your-config>  # disable, or no TLS setting at all, is this cause

Fix

Download the RDS certificate bundle (curl -sSO https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem) and connect with TLS: psql "host=<endpoint> dbname=orders user=app sslmode=verify-full sslrootcert=global-bundle.pem", or for MySQL mysql -h <endpoint> -u app -p --ssl-mode=VERIFY_IDENTITY --ssl-ca=global-bundle.pem.

An engine upgrade or a new parameter group turned TLS enforcement on, so clients that worked before now fail.

Check

aws rds describe-db-instances --db-instance-identifier <db> --query 'DBInstances[].[EngineVersion,DBParameterGroups]'  # a recent upgrade to PostgreSQL 15 or later, or a changed group
aws rds describe-db-parameters --db-parameter-group-name <group> --query "Parameters[?ParameterName=='rds.force_ssl'||ParameterName=='require_secure_transport'].[ParameterName,ParameterValue]"  # 1 or ON: TLS is required

Fix

Fix the clients to use TLS as above. Keep enforcement on rather than turning it off.

The instance was created without an initial database name (--db-name). PostgreSQL then has only postgres; MySQL has none of yours.

Check

aws rds describe-db-instances --db-instance-identifier <db> --query 'DBInstances[].DBName'  # null is this cause

Fix

Create it: CREATE DATABASE orders;, ideally in the first migration.

A typo in the name, or psql was run with no dbname: it then connects to a database named after the user (app).

Check

psql "host=<endpoint> user=<user> dbname=postgres" -c '\l'  # PostgreSQL: the names that exist
mysql -h <endpoint> -u <user> -p -e 'SHOW DATABASES;'  # MySQL: the names that exist

Fix

Use the exact name, and always pass dbname= in connection strings.

You connected to another instance, or to a restored copy.

Check

aws rds describe-db-instances --query 'DBInstances[].[DBInstanceIdentifier,Endpoint.Address]'  # the endpoint you used belongs to another instance

Fix

Point the app at the right endpoint.

Many Lambda invocations or containers each open their own connections.

Check

aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name DatabaseConnections --dimensions Name=DBInstanceIdentifier,Value=<db> --start-time <start> --end-time <end> --period 60 --statistics Maximum  # spikes in step with Lambda concurrency or task count

Fix

Put RDS Proxy in front of the database, and in Lambda open the connection outside the handler.

Connections are opened per request and never closed (a leak).

Check

SELECT state, count(*) FROM pg_stat_activity GROUP BY state;  -- PostgreSQL: many idle connections; MySQL: SHOW PROCESSLIST; with many Sleep rows

Fix

Close connections when done, and reuse them from a pool.

Each process's pool is larger than the database can take in total.

Check

SHOW VARIABLES LIKE 'max_connections';  -- MySQL (PostgreSQL: SHOW max_connections;); compare with pool size times process count

Fix

Size every app's pool so that together they stay under max_connections.

The instance class is too small for the number of clients.

Check

aws rds describe-db-instances --db-instance-identifier <db> --query 'DBInstances[].DBInstanceClass'  # a micro or small class

Fix

Move to a larger class, or raise max_connections in the parameter group if memory allows.

The data grew and storage autoscaling is off, or has reached its maximum.

Check

aws rds describe-db-instances --db-instance-identifier <db> --query 'DBInstances[].[DBInstanceStatus,AllocatedStorage,MaxAllocatedStorage]'  # storage-full, and no MaxAllocatedStorage (or AllocatedStorage equal to it), is this cause

Fix

aws rds modify-db-instance --db-instance-identifier <db> --allocated-storage <+10% or more> --max-allocated-storage <cap> --apply-immediately.

Logs or temporary files are filling the disk (binary logs kept long, large sorts, replication slots holding WAL).

Check

aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name FreeStorageSpace --dimensions Name=DBInstanceIdentifier,Value=<db> --start-time <start> --end-time <end> --period 3600 --statistics Minimum  # a fast drop with no matching data growth
SELECT slot_name, active FROM pg_replication_slots;  -- PostgreSQL: an inactive slot holds WAL

Fix

Drop unused replication slots, shorten binary log retention, then add storage.

The default 3-second timeout was never changed.

Check

aws lambda get-function-configuration --function-name <function> --query '[Timeout,MemorySize]'  # 3 is this cause

Fix

Raise the timeout above the slowest real run: aws lambda update-function-configuration --function-name <function> --timeout 30.

A downstream call (an API, a database) is slow or hangs.

Check

aws logs tail /aws/lambda/<function> --since 1h  # the last log line before the timeout shows which call it was in

Fix

Set timeouts on downstream calls shorter than the function's, and fix or retry the slow dependency.

A function in a VPC waits for a network path that does not exist: no NAT gateway and no VPC endpoint for the service it calls.

Check

aws lambda get-function-configuration --function-name <function> --query VpcConfig  # subnets set
aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=<subnet-id>  # no route to a NAT gateway, and no endpoint for the service: this is your cause

Fix

Add a NAT route or a VPC endpoint for the service, or take the function out of the VPC if it needs nothing there.

Reserved concurrency is set to 0 (switched off) or too low.

Check

aws lambda get-function-concurrency --function-name <function>  # ReservedConcurrentExecutions 0 or a small number is this cause

Fix

aws lambda put-function-concurrency --function-name <function> --reserved-concurrent-executions <n>, or delete-function-concurrency to remove the cap.

The burst is above the account's concurrency limit for the Region.

Check

aws lambda get-account-settings --query 'AccountLimit.ConcurrentExecutions'  # compare with the ConcurrentExecutions metric at the time

Fix

Smooth the load (a queue in front), or request a higher limit.

The library was installed on your machine but not zipped with the function.

Check

unzip -l function.zip  # the package's folder (requests/) missing is this cause

Fix

pip install -r requirements.txt -t package/, then cd package && zip -r ../function.zip ., and add your code at the zip's root.

The files sit inside a folder in the zip, not at its root.

Check

unzip -l function.zip  # paths like myapp/lambda_function.py instead of lambda_function.py

Fix

Zip from inside the folder, so the files are at the root.

The handler setting does not match the file and function.

Check

aws lambda get-function-configuration --function-name <function> --query Handler  # lambda_function.lambda_handler needs lambda_function.py with def lambda_handler

Fix

aws lambda update-function-configuration --function-name <function> --handler lambda_function.lambda_handler.

A package with compiled code was built for another operating system or architecture.

Check

aws lambda get-function-configuration --function-name <function> --query '[Runtime,Architectures]'  # arm64 or x86_64; a wheel built on macOS or for the other architecture is this cause

Fix

Download wheels built for Lambda: pip install --platform manylinux2014_x86_64 --only-binary=:all: -t package/ <package> (manylinux2014_aarch64 for arm64).

Traffic grew past the provisioned WCU or RCU.

Check

aws dynamodb describe-table --table-name <table> --query 'Table.[BillingModeSummary,ProvisionedThroughput]'  # PROVISIONED with low units
aws cloudwatch get-metric-statistics --namespace AWS/DynamoDB --metric-name WriteThrottleEvents --dimensions Name=TableName,Value=<table> --start-time <t> --end-time <t> --period 300 --statistics Sum  # above 0 while traffic is high

Fix

Switch to on-demand (aws dynamodb update-table --table-name <table> --billing-mode PAY_PER_REQUEST) or raise capacity with auto scaling.

A hot partition key concentrates writes on one partition, which throttles even when the table total is under its limit.

Check

aws cloudwatch get-metric-statistics --namespace AWS/DynamoDB --metric-name ConsumedWriteCapacityUnits --dimensions Name=TableName,Value=<table> --start-time <t> --end-time <t> --period 60 --statistics Sum  # throttling while consumption is well below provisioned

Fix

Spread writes over a better-distributed key, for example by adding a suffix to busy keys.

A global secondary index has too little capacity. When a GSI throttles, writes to the table throttle too.

Check

aws dynamodb describe-table --table-name <table> --query 'Table.GlobalSecondaryIndexes[].[IndexName,ProvisionedThroughput]'  # an index with fewer units than the table writes

Fix

Give the index capacity to match the table's writes.

The visibility timeout is shorter than the processing time.

Check

aws sqs get-queue-attributes --queue-url <queue-url> --attribute-names VisibilityTimeout  # shorter than your slowest run (for Lambda, the message above says so) is this cause

Fix

Raise VisibilityTimeout above the slowest processing time; for a Lambda consumer, six times the function timeout.

The consumer deletes the message only after a slow step, or never on some code paths (errors).

Check

aws sqs get-queue-attributes --queue-url <queue-url> --attribute-names ApproximateNumberOfMessagesNotVisible  # stays high while the consumer is idle

Fix

Delete every message you finished, including on handled errors.

Standard queues can deliver a message twice even when everything is right.

Check

aws sqs get-queue-attributes --queue-url <queue-url> --attribute-names FifoQueue  # no FifoQueue: a standard queue

Fix

Make processing idempotent, keyed on a message or business ID.

The password was pasted into an environment variable or config file. Rotation cannot update copies.

Check

aws lambda get-function-configuration --function-name <function> --query 'keys(Environment.Variables)'  # names only: a DB_PASSWORD here is a pasted copy, and this is your cause

Fix

Remove the copy. Read the secret at run time (with a short cache) and grant secretsmanager:GetSecretValue on that secret only.

The app reads the secret once and caches it forever.

Check

aws secretsmanager describe-secret --secret-id <secret-id> --query LastRotatedDate  # rotated after the app last started

Fix

Re-read the secret when a login fails, and cache it for minutes, not for the life of the process.

The rotation Lambda failed halfway, so the database and the secret's AWSCURRENT disagree.

Check

aws secretsmanager describe-secret --secret-id <secret-id> --query VersionIdsToStages  # AWSPENDING still on a version after the rotation window is this cause
aws logs tail /aws/lambda/<rotation-function> --since 1d  # which step failed: createSecret, setSecret, testSecret or finishSecret

Fix

Fix the error in the log, then aws secretsmanager rotate-secret --secret-id <secret-id> to run it again.

The policy does not cover the secret's ARN. Secret ARNs end in a random six-character suffix (prod/orders/db-AbCdEf), so a Resource without it does not match.

Check

aws iam simulate-principal-policy --policy-source-arn <role-arn> --action-names secretsmanager:GetSecretValue --resource-arns <secret-arn>  # implicitDeny is this cause

Fix

Allow secretsmanager:GetSecretValue on the full secret ARN, or on arn:aws:secretsmanager:<region>:<account-id>:secret:prod/orders/db-*.

The secret uses a customer managed KMS key the role cannot decrypt.

Check

aws secretsmanager describe-secret --secret-id <secret-arn> --query KmsKeyId  # empty means the AWS managed key; a key ARN means the role also needs kms:Decrypt on it

Fix

Allow kms:Decrypt on that key, in the key policy or (if it delegates to IAM) in the role's policy.

A resource policy on the secret denies the role. The message then says explicit deny in a resource-based policy.

Check

aws secretsmanager get-resource-policy --secret-id <secret-arn>  # a Deny that matches the role

Fix

Narrow the Deny, or call from where it allows (a VPC endpoint, a tagged role).

Cross-account access: the secret needs a resource policy and a customer managed KMS key, because the default aws/secretsmanager key cannot be used from another account.

Check

aws sts get-caller-identity --query Account  # differs from the account in the secret ARN

Fix

Add a resource policy for the role, encrypt the secret with a customer managed key that allows the role, and allow both in the role's policy.

A resource failed during the first create: a name already taken, a missing permission, a bad parameter.

Check

aws cloudformation describe-stack-events --stack-name <stack> --query "StackEvents[?ResourceStatus=='CREATE_FAILED'].[LogicalResourceId,ResourceStatusReason]"  # the oldest CREATE_FAILED is the real cause; later ones are usually cancelled

Fix

Fix the template or parameters for that reason, then aws cloudformation delete-stack --stack-name <stack>, aws cloudformation wait stack-delete-complete --stack-name <stack>, and create it again.

You deployed again without fixing what failed.

Check

aws cloudformation describe-stacks --stack-name <stack> --query 'Stacks[0].StackStatus'  # ROLLBACK_COMPLETE again after the redeploy

Fix

Read the first CREATE_FAILED reason before deploying again.

The template creates IAM roles, users, groups or policies (CAPABILITY_IAM), or gives them custom names (CAPABILITY_NAMED_IAM).

Check

grep -nE 'AWS::IAM::|RoleName|PolicyName|UserName|GroupName' template.yaml  # IAM resources; a fixed name needs CAPABILITY_NAMED_IAM

Fix

Read which IAM resources it creates, then deploy with --capabilities CAPABILITY_NAMED_IAM (or CAPABILITY_IAM).

The template uses a transform such as AWS::Serverless (CAPABILITY_AUTO_EXPAND).

Check

grep -n 'Transform' template.yaml  # a Transform line is this cause

Fix

Add CAPABILITY_AUTO_EXPAND to --capabilities.

The pipeline ran again with no template change.

Check

aws cloudformation get-template --stack-name <stack> --query TemplateBody  # identical to your template

Fix

With aws cloudformation deploy, add --no-fail-on-empty-changeset so a run with no changes exits 0.

You changed a file the template only points at, such as Lambda code in S3 under the same key. CloudFormation compares the template, not the file.

Check

aws cloudformation describe-stacks --stack-name <stack> --query 'Stacks[0].Parameters'  # the same S3 key or version as before

Fix

Upload the new code under a new key or object version and pass that to the template.

A script polls in a tight loop.

Check

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=<operation> --max-results 50  # many calls a second from one identity or user agent is this cause

Fix

Slow down, use waiters (aws ec2 wait) and events (EventBridge) instead of polling.

Many tools share one account and Region, and together exceed the limit.

Check

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=<operation> --max-results 50  # many different identities calling the same API

Fix

Set AWS_RETRY_MODE=adaptive and AWS_MAX_ATTEMPTS=10, batch calls, and spread heavy tools over time.