← Back to all fixes
CI/CD

Pipeline passes but production is broken

The Issue

CI pipeline passes every check. Deployment goes through. Production serves 500 errors.

Root Cause

Tests pass against mocked dependencies. Real production has different env vars, live database schema, secrets that differ from test values, or the deploy target has different OS/package versions.

How to Fix
  1. Run integration tests against a real database, not mocks
  2. Add a smoke test stage after deployment — curl a health endpoint: `curl -f https://yourapp.com/health`
  3. Use the same base Docker image in CI and production
  4. Check if a missing env var in production is causing the crash
  5. Add a rollback step: if smoke test fails, automatically redeploy previous version
← More fixes