CI/CD Pipeline Best Practices: From Code to Production

CI/CD Pipeline Best Practices: From Code to Production

BySanjay Goraniya•
4 min read
devopscicdautomationdeploymenttesting
Share:

CI/CD Pipeline Best Practices: From Code to Production

A well-designed CI/CD pipeline is the backbone of modern software development. It's the difference between deploying with confidence and deploying with fear. After setting up and maintaining CI/CD pipelines for multiple teams, I've learned what works and what doesn't. Let me share the practices that have proven most valuable.

The Foundation: What CI/CD Should Do

A good CI/CD pipeline should:

  1. Catch bugs early - Before they reach production
  2. Deploy consistently - Same process every time
  3. Provide feedback quickly - Developers shouldn't wait hours
  4. Enable rollbacks - When things go wrong
  5. Maintain quality - Enforce standards automatically

Pipeline Stages

Stage 1: Linting and Formatting

Catch style issues before code review:

Code
# .github/workflows/ci.yml
lint:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-node@v3
    - run: npm ci
    - run: npm run lint
    - run: npm run format:check

Why it matters: Consistent code style reduces cognitive load and makes reviews faster.

Stage 2: Type Checking

For TypeScript projects:

Code
type-check:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-node@v3
    - run: npm ci
    - run: npm run type-check

Why it matters: Catch type errors before runtime.

Stage 3: Unit Tests

Fast feedback on logic errors:

Code
test:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-node@v3
    - run: npm ci
    - run: npm test -- --coverage
    - uses: codecov/codecov-action@v3
      with:
        files: ./coverage/lcov.info

Best practices:

  • Run tests in parallel when possible
  • Set coverage thresholds (but don't obsess over 100%)
  • Fail fast on first error

Stage 4: Integration Tests

Test components working together:

Code
integration-test:
  runs-on: ubuntu-latest
  services:
    postgres:
      image: postgres:15
      env:
        POSTGRES_PASSWORD: test
      options: >-
        --health-cmd pg_isready
        --health-interval 10s
        --health-timeout 5s
        --health-retries 5
  steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-node@v3
    - run: npm ci
    - run: npm run test:integration

Stage 5: Build

Ensure code compiles and bundles correctly:

Code
build:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-node@v3
    - run: npm ci
    - run: npm run build
    - uses: actions/upload-artifact@v3
      with:
        name: build-artifacts
        path: dist/

Stage 6: Security Scanning

Code
security:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-node@v3
    - run: npm audit --audit-level=high
    - uses: aquasecurity/trivy-action@master
      with:
        scan-type: 'fs'
        scan-ref: '.'

Deployment Strategies

Blue-Green Deployment

Maintain two identical production environments:

Code
deploy-blue:
  runs-on: ubuntu-latest
  steps:
    - name: Deploy to blue environment
      run: |
        # Deploy to blue
        kubectl set image deployment/app app=myapp:${{ github.sha }} -n blue
        
    - name: Health check
      run: |
        # Wait for blue to be healthy
        kubectl wait --for=condition=available deployment/app -n blue --timeout=300s
        
    - name: Switch traffic
      run: |
        # Switch load balancer to blue
        kubectl patch service app -p '{"spec":{"selector":{"version":"blue"}}}'

Advantages:

  • Zero-downtime deployments
  • Easy rollback (switch back to green)
  • Test new version before switching traffic

Canary Deployments

Gradually roll out to a percentage of users:

Code
deploy-canary:
  runs-on: ubuntu-latest
  steps:
    - name: Deploy canary (10%)
      run: |
        kubectl set image deployment/app-canary app=myapp:${{ github.sha }}
        # Route 10% traffic to canary
        
    - name: Monitor metrics
      run: |
        # Wait and check error rates
        sleep 300
        # If error rate < threshold, continue
        
    - name: Full rollout
      if: success()
      run: |
        kubectl set image deployment/app app=myapp:${{ github.sha }}
        # Route 100% traffic

Environment Management

Separate Environments

Code
Development → Staging → Production

Each environment should mirror production as closely as possible.

Environment Variables

Never commit secrets. Use secret management:

Code
deploy:
  runs-on: ubuntu-latest
  env:
    DATABASE_URL: ${{ secrets.DATABASE_URL }}
    API_KEY: ${{ secrets.API_KEY }}
  steps:
    - name: Deploy
      run: |
        # Use environment variables
        echo "Deploying with secure credentials"

Rollback Strategies

Automated Rollback

Code
deploy:
  runs-on: ubuntu-latest
  steps:
    - name: Deploy
      id: deploy
      run: |
        # Deploy and capture version
        VERSION=$(kubectl rollout status deployment/app)
        echo "version=$VERSION" >> $GITHUB_OUTPUT
        
    - name: Health check
      run: |
        # Check if deployment is healthy
        if ! kubectl rollout status deployment/app --timeout=5m; then
          echo "Deployment failed, rolling back"
          kubectl rollout undo deployment/app
          exit 1
        fi

Manual Rollback

Always have a one-command rollback:

Code
# Kubernetes
kubectl rollout undo deployment/app

# Docker
docker service rollback app

# Custom script
./scripts/rollback.sh production v1.2.3

Testing in Production

Feature Flags

Deploy code behind flags:

Code
// Deploy code, but don't enable yet
if (featureFlags.isEnabled('new-checkout-flow', userId)) {
  return <NewCheckoutFlow />;
}
return <OldCheckoutFlow />;

Monitoring and Alerts

Set up alerts for:

  • Error rates
  • Response times
  • Resource usage
  • Business metrics
Code
monitor:
  runs-on: ubuntu-latest
  steps:
    - name: Check error rate
      run: |
        ERROR_RATE=$(curl -s metrics/api/error-rate)
        if (( $(echo "$ERROR_RATE > 0.05" | bc -l) )); then
          echo "Error rate too high: $ERROR_RATE"
          exit 1
        fi

Pipeline Optimization

Parallel Execution

Run independent jobs in parallel:

Code
jobs:
  lint:
    # ...
  test:
    # ...
  build:
    # ...
  security:
    # ...

# All run in parallel

Caching

Cache dependencies to speed up builds:

Code
- uses: actions/cache@v3
  with:
    path: node_modules
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-node-

Conditional Execution

Skip unnecessary steps:

Code
test:
  if: contains(github.event.head_commit.message, '[skip tests]') == false
  runs-on: ubuntu-latest
  steps:
    - run: npm test

Common Pitfalls

1. Testing in Production

Don't use production as your test environment. Always test in staging first.

2. Ignoring Failed Tests

If tests are flaky, fix them. Don't disable them.

3. Manual Deployments

Automate everything. Manual steps are error-prone.

4. No Rollback Plan

Always have a way to rollback. Test your rollback process.

5. Slow Pipelines

If your pipeline takes hours, developers will skip it. Optimize for speed.

Real-World Example

Challenge: Deployment pipeline taking 45 minutes, frequent production issues.

Solution:

  1. Parallelized tests - Reduced test time from 30min to 8min
  2. Added caching - Build time reduced from 10min to 2min
  3. Implemented canary deployments - Catch issues before full rollout
  4. Added automated rollback - Rollback time reduced from 15min to 30sec
  5. Improved monitoring - Detect issues within 1 minute

Result:

  • Pipeline time: 45min → 12min
  • Production incidents: 5/month → 1/month
  • Deployment confidence: High

Best Practices Summary

  1. Fail fast - Catch issues early in the pipeline
  2. Run in parallel - Don't wait unnecessarily
  3. Cache dependencies - Speed up builds
  4. Test everything - Unit, integration, e2e
  5. Monitor deployments - Know when something's wrong
  6. Automate rollbacks - Recover quickly
  7. Document everything - Future you will thank you

Conclusion

A good CI/CD pipeline is invisible when it works and invaluable when it doesn't. The investment in setting up a robust pipeline pays dividends in:

  • Reduced bugs in production
  • Faster development cycles
  • Higher team confidence
  • Better code quality

Remember: Your pipeline is only as good as your tests. Invest in good tests, and your pipeline will catch issues before they reach production.

What CI/CD challenges have you faced? What practices have worked best for your team?

Share:

Related Posts