Production Fixes

Real issues from real deployments. Each fix is tagged by tool so you can find what you need fast.

🔍
Docker

Container exits immediately after start

Container starts but exits within seconds. docker ps shows nothing.

Read fix →
Docker Linux

permission denied: /var/run/docker.sock

Running docker commands fails with permission denied on the socket.

Read fix →
Docker

Docker build using stale cache after code change

Image rebuilds skip your code changes and serve old content.

Read fix →
Docker

Container cannot reach host or other containers

App inside container fails to connect to localhost or sibling containers.

Read fix →
Docker

Docker image is several GB — much larger than expected

Built image is 2-3x bigger than it needs to be.

Read fix →
Docker

Environment variable not available inside container

App inside container cannot read env variables you set on the host.

Read fix →
Docker

Data lost when container restarts

Uploads or database data disappear every time the container is recreated.

Read fix →
Docker

Port is already in use when starting container

docker run fails because the host port is already bound.

Read fix →
Kubernetes

Pod stuck in CrashLoopBackOff

Pod keeps restarting and never reaches Running state.

Read fix →
Kubernetes

Pod stuck in ImagePullBackOff or ErrImagePull

Kubernetes cannot pull the container image.

Read fix →
Kubernetes

Service is not reachable inside the cluster

Pods cannot reach a Service by its DNS name or ClusterIP.

Read fix →
Kubernetes

Pod stuck in Pending — insufficient resources

Pod never schedules because no node has enough CPU or memory.

Read fix →
Kubernetes

Secret value is empty or unavailable in pod

Environment variable backed by a Secret is empty inside the container.

Read fix →
Kubernetes

Container killed with OOMKilled

Container keeps dying because it exceeds its memory limit.

Read fix →
Kubernetes

Ingress returns 404 or does not route traffic

Ingress is created but traffic hits 404 or goes to the wrong backend.

Read fix →
Kubernetes

Deployment update not rolling out — same old pods

You updated the image tag but pods still run the old version.

Read fix →
Linux

Disk full — no space left on device

Server throws "no space left on device" but du shows plenty of space.

Read fix →
Linux

kill command does not stop the process

Process keeps running after kill. Even kill -9 does not work.

Read fix →
Linux

SSH: Permission denied (publickey)

SSH login fails with publickey error even with correct key.

Read fix →
Linux

High CPU but no obvious process in top

Server is slow and CPU is high but top does not show the culprit.

Read fix →
Linux

Cron job not running as expected

Cron entry looks correct but the job never runs.

Read fix →
Linux

Port already in use — cannot start service

Service fails to start because another process holds the port.

Read fix →
Linux

Permission denied even as root

Even with sudo you cannot write to a file.

Read fix →
Linux

Load average is high but CPU usage is low

top shows load average of 20+ but CPUs are mostly idle.

Read fix →
Nginx

Nginx returns 502 Bad Gateway

Nginx is running but proxying to your app returns 502.

Read fix →
Nginx SSL

Nginx fails to start: SSL cert file not found

nginx -t fails because cert paths in config do not exist yet.

Read fix →
Nginx

413 Request Entity Too Large on file upload

File uploads fail with 413 error.

Read fix →
Nginx

ERR_TOO_MANY_REDIRECTS — redirect loop

Browser shows too many redirects when hitting the site.

Read fix →
Nginx

nginx -t fails with cryptic emerg error

nginx config test throws emerg error and nginx will not reload.

Read fix →
Nginx SSL

Connection refused on port 443 (HTTPS)

HTTP works but HTTPS gives connection refused.

Read fix →
Nginx

Nginx 504 Gateway Timeout

Slow requests result in 504 from nginx before the app responds.

Read fix →
Nginx

Static files return 404 through nginx

CSS, JS, or image files return 404 when served via nginx.

Read fix →
CI/CD

Pipeline passes but production is broken

All CI tests go green but the deployed app crashes in production.

Read fix →
CI/CD

CI builds are very slow — taking 15-20 minutes

Pipeline takes too long, blocking developer feedback.

Read fix →
CI/CD

Secret accidentally printed in CI logs

A token or password appears in plain text in the pipeline output.

Read fix →
CI/CD

Deployment causes downtime — requests drop during rollout

Every deployment causes a brief outage as the old container is replaced.

Read fix →
CI/CD

Pipeline fails: environment variable not set

Build script errors because a required env var is undefined in CI.

Read fix →
CI/CD

Stale branch causes pipeline failures not related to your change

Your PR fails tests that your change did not touch.

Read fix →
CI/CD Docker

docker push fails with authentication required in CI

Pipeline cannot push image to registry — auth error.

Read fix →
CI/CD

Need to rollback but no previous version available

Production is broken but you cannot rollback because old image was overwritten.

Read fix →
AWS

Cannot SSH into EC2 instance

SSH connection to EC2 times out or is refused.

Read fix →
AWS

S3 returns 403 Access Denied

Reading or writing to S3 fails with 403 even though permissions look right.

Read fix →
AWS

Application cannot connect to RDS

App gets connection timeout when connecting to RDS database.

Read fix →
AWS

Lambda function times out

Lambda hits timeout limit and never completes the task.

Read fix →
AWS

AWS assume role fails with AccessDenied

Cross-account or service assume-role call is denied.

Read fix →
AWS

ALB returns 504 Gateway Timeout

Application Load Balancer returns 504 for slow requests.

Read fix →
AWS

Logs not appearing in CloudWatch

Application logs not showing up in CloudWatch Log Groups.

Read fix →
AWS

Unexpected high AWS bill from EC2

AWS bill is much higher than expected due to EC2 usage.

Read fix →
Git

Secret accidentally committed to Git

API key or password committed to the repo — need to remove it from history.

Read fix →
Git

Stuck in detached HEAD state

git status shows "HEAD detached" and commits seem to disappear.

Read fix →
Git

Merge conflict blocks PR merge

PR cannot be merged due to conflicts with the target branch.

Read fix →
Git

Need to undo a commit already pushed to main

Bad commit was merged to main — need to remove it without rewriting history.

Read fix →
Git

Push rejected: file too large

git push fails because a large file was committed.

Read fix →
Git

Committed to main instead of feature branch

Accidentally committed directly to main (or wrong branch).

Read fix →
Git

Branches have diverged — push rejected

git push rejected because remote has commits your local branch does not.

Read fix →
Git

Submodule directory is empty after git clone

After cloning a repo, submodule folders are empty.

Read fix →