From ClickOps to GitOps: Building a Culture of Infrastructure as Code

git log --oneline terraform/rds.tf
 
a3b2c1d (HEAD -> main) Increase RDS storage for incident INC-12345
b4c3d2e Add performance insights to production database
c5d4e3f Enable automated backups with 30-day retention

#!/bin/bash
# Weekly drift detection report
 
cd /path/to/terraform
 
# Run plan and capture output
terraform plan -detailed-exitcode > /dev/null 2>&1
EXIT_CODE=$?
 
# Exit codes:
# 0 = No changes (no drift)
# 1 = Error
# 2 = Changes detected (drift exists)
 
if [ $EXIT_CODE -eq 2 ]; then
  echo "⚠️  DRIFT DETECTED"
 
  # Generate detailed drift report
  terraform plan -no-color > drift-report-$(date +%Y%m%d).txt
 
  # Count number of resources with drift
  DRIFT_COUNT=$(terraform plan -no-color | grep -c "# .* will be updated in-place")
 
  echo "Resources with drift: $DRIFT_COUNT"
 
  # Alert team
  curl -X POST "$SLACK_WEBHOOK" \
    -H 'Content-Type: application/json' \
    -d "{\"text\":\"🚨 Drift detected: $DRIFT_COUNT resources differ from code\"}"
fi

#!/usr/bin/env python3
"""
Analyze CloudTrail to identify console vs. programmatic changes
"""
import boto3
from datetime import datetime, timedelta
from collections import defaultdict
 
cloudtrail = boto3.client('cloudtrail')
 
# Look back 7 days
start_time = datetime.now() - timedelta(days=7)
 
# Fetch events
events = cloudtrail.lookup_events(
    StartTime=start_time,
    LookupAttributes=[
        {'AttributeKey': 'EventName', 'AttributeValue': 'ModifyDBInstance'},
        {'AttributeKey': 'EventName', 'AttributeValue': 'ModifySecurityGroup'},
        {'AttributeKey': 'EventName', 'AttributeValue': 'PutBucketPolicy'},
        # Add more mutation events
    ]
)
 
console_changes = 0
programmatic_changes = 0
 
for event in events['Events']:
    user_agent = event.get('UserAgent', '')
 
    if 'console.aws.amazon.com' in user_agent:
        console_changes += 1
        print(f"🖱️  Console: {event['Username']} - {event['EventName']}")
    else:
        programmatic_changes += 1
 
total = console_changes + programmatic_changes
console_percentage = (console_changes / total * 100) if total > 0 else 0
 
print(f"\n📊 Console Changes: {console_changes} ({console_percentage:.1f}%)")
print(f"📊 Programmatic: {programmatic_changes} ({100-console_percentage:.1f}%)")
 
# Set target: < 5% console changes (excluding break-glass scenarios)
if console_percentage > 5:
    print("⚠️  Console usage above target threshold")

# ClickOps Cost Analysis
 
## Current State
- 47 resources with configuration drift
- 34% of infrastructure changes via console
- Average 8 hours to create production-like environment
- 6 of last 10 incidents caused by or complicated by manual changes
 
## Projected Costs (Annual)
- Engineering time on drift debugging: $85,000
- Delayed incident resolution: $120,000
- Failed compliance audit risk: $250,000+
- Total: $455,000+
 
## GitOps Investment
- Tooling and automation: $30,000
- Training and transition: $50,000
- First year total: $80,000
 
## ROI
- Year 1 savings: $375,000
- Payback period: 2.5 months
- Additional benefits: Faster deployment, reduced risk, better compliance

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOnlyConsoleAccess",
      "Effect": "Allow",
      "Action": [
        "ec2:Describe*",
        "rds:Describe*",
        "s3:List*",
        "s3:Get*",
        "iam:List*",
        "iam:Get*",
        "cloudwatch:Describe*",
        "cloudwatch:Get*",
        "cloudwatch:List*",
        "logs:Describe*",
        "logs:FilterLogEvents",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DenyAllMutations",
      "Effect": "Deny",
      "NotAction": [
        "ec2:Describe*",
        "rds:Describe*",
        "s3:List*",
        "s3:Get*",
        "iam:List*",
        "iam:Get*",
        "cloudwatch:Describe*",
        "cloudwatch:Get*",
        "cloudwatch:List*",
        "logs:Describe*",
        "logs:FilterLogEvents",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EmergencyAccess",
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:RequestedRegion": ["us-east-1", "us-west-2"]
        }
      }
    }
  ]
}

# Lambda function triggered by CloudWatch Events
import boto3
import json
 
sns = boto3.client('sns')
 
def lambda_handler(event, context):
    """Alert team when break-glass role is assumed"""
 
    detail = event['detail']
    user_identity = detail['userIdentity']
 
    if 'break-glass' in user_identity.get('arn', '').lower():
        message = f"""
        🚨 BREAK-GLASS ROLE ASSUMED
 
        User: {user_identity.get('principalId')}
        Time: {detail['eventTime']}
        Source IP: {detail.get('sourceIPAddress')}
 
        REMINDER: All emergency changes must be backported to IaC within 24 hours.
        Create tracking ticket: https://your-issue-tracker.com/new
        """
 
        sns.publish(
            TopicArn='arn:aws:sns:us-east-1:123456789012:security-alerts',
            Subject='🚨 Break-Glass Role Assumed',
            Message=message
        )
 
        # Also post to Slack
        # ... slack webhook code ...

# Policy Update: Read-Only Console Access
 
Effective Date: [DATE]
 
## What's Changing
- Developer AWS console access is now read-only by default
- All infrastructure changes must be made via Pull Requests to infrastructure repository
- Emergency access available via break-glass role
 
## Why This Change
- Reduces configuration drift (currently 47 resources)
- Provides audit trail for all changes
- Enables faster environment reproduction
- Improves security and compliance posture
 
## How to Make Changes
1. Clone infrastructure repository
2. Make changes to Terraform/CDK code
3. Run `terraform plan` locally to verify
4. Create Pull Request
5. Get review and approval
6. Merge triggers automated deployment
 
## Emergency Procedures
If production is down and immediate console access needed:
1. Assume break-glass role: `aws sts assume-role ...`
2. Make necessary changes
3. Immediately notify team in #incidents channel
4. Create ticket to backport changes to code (within 24 hours)
 
## Training
- Terraform workshop: [DATE]
- Office hours: Daily 2-3 PM
- Documentation: https://wiki/gitops
 
## Questions
Contact platform-team@company.com

# .github/workflows/drift-detection.yml
name: Infrastructure Drift Detection
 
on:
  schedule:
    - cron: '0 */6 * * *'  # Every 6 hours
  workflow_dispatch:  # Manual trigger
 
jobs:
  detect-drift:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        environment: [development, staging, production]
        region: [us-east-1, us-west-2]
 
    steps:
      - uses: actions/checkout@v4
 
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/TerraformReadOnly
          aws-region: ${{ matrix.region }}
 
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
 
      - name: Terraform Init
        run: terraform init
        working-directory: environments/${{ matrix.environment }}
 
      - name: Detect Drift
        id: drift
        run: |
          terraform plan -detailed-exitcode -no-color > plan.txt 2>&1
          EXIT_CODE=$?
 
          if [ $EXIT_CODE -eq 2 ]; then
            echo "drift=true" >> $GITHUB_OUTPUT
            echo "Drift detected in ${{ matrix.environment }} - ${{ matrix.region }}"
          else
            echo "drift=false" >> $GITHUB_OUTPUT
          fi
        continue-on-error: true
        working-directory: environments/${{ matrix.environment }}
 
      - name: Post to Slack
        if: steps.drift.outputs.drift == 'true'
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "🚨 Infrastructure Drift Detected",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*Environment:* ${{ matrix.environment }}\n*Region:* ${{ matrix.region }}\n*Status:* Configuration drift detected"
                  }
                },
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "Please review and reconcile the drift. All infrastructure changes should be made via code."
                  }
                }
              ]
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}
 
      - name: Create Issue
        if: steps.drift.outputs.drift == 'true'
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `Drift detected: ${{ matrix.environment }} - ${{ matrix.region }}`,
              body: `Configuration drift detected in ${{ matrix.environment }} environment (${{ matrix.region }} region).\n\nPlease investigate and reconcile.`,
              labels: ['drift', 'infrastructure', '${{ matrix.environment }}']
            });

# Lambda-backed AWS Config rule
import boto3
import json
 
def evaluate_compliance(config_item, rule_parameters):
    """
    Check if resource has required ManagedBy tag
    """
    tags = config_item.get('tags', {})
    managed_by = tags.get('ManagedBy', '')
 
    # Resources must be managed by Terraform/CDK
    if managed_by.lower() in ['terraform', 'cdk', 'cloudformation']:
        return 'COMPLIANT'
    else:
        return 'NON_COMPLIANT'
 
def lambda_handler(event, context):
    """AWS Config Lambda handler"""
    config = boto3.client('config')
 
    invoking_event = json.loads(event['invokingEvent'])
    config_item = invoking_event['configurationItem']
 
    compliance_status = evaluate_compliance(config_item, {})
 
    config.put_evaluations(
        Evaluations=[
            {
                'ComplianceResourceType': config_item['resourceType'],
                'ComplianceResourceId': config_item['resourceId'],
                'ComplianceType': compliance_status,
                'OrderingTimestamp': config_item['configurationItemCaptureTime']
            }
        ],
        ResultToken=event['resultToken']
    )

# Drift Reconciliation Runbook
 
## When Drift is Detected
 
### Step 1: Identify the Change
```bash
# Run terraform plan to see what's different
cd environments/production
terraform plan -no-color > drift.txt
 
# Review the drift
cat drift.txt

# Check CloudTrail for recent modifications
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=<resource-id> \
  --max-results 10 \
  --query 'Events[*].[EventTime,Username,EventName]' \
  --output table

# Terraform policy
resource "aws_organizations_policy" "require_managed_by_tag" {
  name        = "RequireManagedByTag"
  description = "Require all resources to have ManagedBy tag"
 
  content = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "RequireManagedByTag"
        Effect = "Deny"
        Action = [
          "ec2:RunInstances",
          "rds:CreateDBInstance",
          "s3:CreateBucket",
          "dynamodb:CreateTable"
        ]
        Resource = "*"
        Condition = {
          StringNotEquals = {
            "aws:RequestTag/ManagedBy" = ["Terraform", "CDK"]
          }
        }
      }
    ]
  })
}

#!/usr/bin/env python3
"""
Auto-remediate resources without proper ManagedBy tags
"""
import boto3
from datetime import datetime, timedelta
 
ec2 = boto3.client('ec2')
sns = boto3.client('sns')
 
def find_unmanaged_resources():
    """Find EC2 instances without ManagedBy tag"""
    response = ec2.describe_instances(
        Filters=[
            {
                'Name': 'instance-state-name',
                'Values': ['running']
            }
        ]
    )
 
    unmanaged = []
 
    for reservation in response['Reservations']:
        for instance in reservation['Instances']:
            tags = {tag['Key']: tag['Value'] for tag in instance.get('Tags', [])}
 
            if 'ManagedBy' not in tags or tags['ManagedBy'] not in ['Terraform', 'CDK']:
                unmanaged.append({
                    'InstanceId': instance['InstanceId'],
                    'LaunchTime': instance['LaunchTime'],
                    'Tags': tags
                })
 
    return unmanaged
 
def remediate(unmanaged_resources, dry_run=True):
    """Terminate unmanaged resources (with grace period)"""
    now = datetime.now(tz=timezone.utc)
    grace_period = timedelta(hours=24)
 
    for resource in unmanaged_resources:
        launch_time = resource['LaunchTime']
        age = now - launch_time
 
        if age > grace_period:
            if dry_run:
                print(f"[DRY RUN] Would terminate: {resource['InstanceId']}")
            else:
                print(f"Terminating unmanaged instance: {resource['InstanceId']}")
                ec2.terminate_instances(InstanceIds=[resource['InstanceId']])
 
                # Alert team
                sns.publish(
                    TopicArn='arn:aws:sns:us-east-1:123456789012:security-alerts',
                    Subject='Unmanaged resource terminated',
                    Message=f"Terminated {resource['InstanceId']} - missing ManagedBy tag"
                )
        else:
            # Tag for pending termination
            ec2.create_tags(
                Resources=[resource['InstanceId']],
                Tags=[
                    {'Key': 'PendingTermination', 'Value': 'true'},
                    {'Key': 'TerminationDate', 'Value': (launch_time + grace_period).isoformat()}
                ]
            )
 
            print(f"Tagged for termination: {resource['InstanceId']}")
 
# Run
unmanaged = find_unmanaged_resources()
print(f"Found {len(unmanaged)} unmanaged resources")
 
# Start with dry run for 2 weeks, then enable actual termination
remediate(unmanaged, dry_run=True)

# CLI tool to make infrastructure changes easier
#!/bin/bash
# iac-change - Helper script for infrastructure changes
 
function iac_change() {
  echo "🏗️  Infrastructure Change Wizard"
  echo ""
 
  # Guide user through process
  echo "What would you like to change?"
  echo "1. Add new resource"
  echo "2. Modify existing resource"
  echo "3. Delete resource"
  read -p "Choice: " choice
 
  read -p "Which environment? (dev/staging/prod): " env
  read -p "Short description: " description
 
  # Create branch
  branch="infra/${env}/${description// /-}"
  git checkout -b "$branch"
 
  # Open editor
  echo "Opening Terraform files..."
  code "environments/$env"
 
  echo ""
  echo "After making changes:"
  echo "1. Run: terraform plan"
  echo "2. Review the changes"
  echo "3. Run: git add . && git commit"
  echo "4. Run: git push origin $branch"
  echo "5. Create PR at: https://github.com/company/infrastructure/compare/$branch"
}
 
iac_change

# Weekly Infrastructure Newsletter
 
## This Week in GitOps 🎉
 
### Achievements
- 🎯 Zero drift detected for 7 consecutive days!
- 🚀 12 infrastructure PRs merged this week
- ⚡ Average PR review time: 2.3 hours (down from 4.1)
 
### Shoutouts
- @alice for excellent documentation in her VPC PR
- @bob for catching a security issue in code review
- @charlie for backporting emergency change within 1 hour
 
### By The Numbers
- Console usage: 2% (target: <5%) ✅
- Drift incidents: 0 (target: 0) ✅
- Time to provision new environment: 23 minutes (target: <30min) ✅
 
### This Week's Learning
@diane wrote a great guide on Terraform modules:
https://wiki/terraform-modules-guide

# New Engineer Onboarding - Infrastructure
 
Day 1:
- [ ] Clone infrastructure repository
- [ ] Read GitOps principles document
- [ ] Watch "Why We Don't Click" video (15 min)
 
Week 1:
- [ ] Complete Terraform basics course
- [ ] Shadow senior engineer on infrastructure PR
- [ ] Get read-only AWS console access
 
Week 2:
- [ ] Make first infrastructure PR (add self to team IAM group)
- [ ] Attend infrastructure office hours
- [ ] Review break-glass procedures
 
Week 3:
- [ ] Make substantial infrastructure change
- [ ] Review someone else's infrastructure PR
- [ ] Get full commit access
 
The goal: New engineers contributing infrastructure changes within 3 weeks

#!/bin/bash
# break-glass - Emergency access script
 
echo "🚨 EMERGENCY ACCESS PROCEDURE 🚨"
echo ""
echo "This grants full AWS console access for emergency situations."
echo "All actions are logged and audited."
echo ""
read -p "Incident ticket number (required): " incident
read -p "Brief description: " description
 
if [ -z "$incident" ]; then
  echo "❌ Incident ticket required"
  exit 1
fi
 
# Assume role
aws sts assume-role \
  --role-arn "arn:aws:iam::123456789012:role/BreakGlass" \
  --role-session-name "emergency-${USER}-${incident}" \
  --duration-seconds 3600
 
# Alert team
curl -X POST $SLACK_WEBHOOK \
  -H 'Content-Type: application/json' \
  -d "{
    \"text\": \"🚨 Break-glass access assumed\",
    \"blocks\": [{
      \"type\": \"section\",
      \"text\": {
        \"type\": \"mrkdwn\",
        \"text\": \"*User:* ${USER}\n*Incident:* ${incident}\n*Description:* ${description}\n*Time:* $(date)\"
      }
    }]
  }"
 
echo ""
echo "✅ Emergency access granted for 1 hour"
echo "⚠️  REMINDER: Backport all changes to IaC within 24 hours"
echo "📝 Create backport ticket: https://issues/new?template=backport&incident=${incident}"

# Lambda to create backport tickets automatically
import boto3
from datetime import datetime, timedelta
 
dynamodb = boto3.resource('dynamodb')
github = # GitHub API client
 
def lambda_handler(event, context):
    """Track break-glass usage and create backport tickets"""
 
    # Triggered by CloudWatch Events when break-glass role assumed
    detail = event['detail']
 
    session_name = detail['userIdentity']['principalId']
    username = session_name.split('-')[1]
    incident = session_name.split('-')[2]
    timestamp = detail['eventTime']
 
    # Create GitHub issue for backport
    issue = github.create_issue(
        title=f"Backport break-glass changes from {incident}",
        body=f"""
## Break-Glass Usage
 
- **User:** {username}
- **Incident:** {incident}
- **Time:** {timestamp}
- **Deadline:** {(datetime.now() + timedelta(hours=24)).isoformat()}
 
## Required Actions
 
- [ ] Review CloudTrail logs for all changes made during emergency session
- [ ] Update Terraform/CDK to match changes
- [ ] Create PR with changes
- [ ] Get review and merge
- [ ] Verify drift is resolved
 
## CloudTrail Query
 
aws cloudtrail lookup-events \\
  --lookup-attributes AttributeKey=Username,AttributeValue={session_name} \\
  --start-time {timestamp} \\
  --max-results 50
 
**This issue will be escalated if not resolved within 24 hours.**
        """,
        labels=['break-glass', 'backport', 'high-priority'],
        assignees=[username]
    )
 
    # Store in DynamoDB for tracking
    table = dynamodb.Table('BreakGlassAudit')
    table.put_item(Item={
        'session_id': session_name,
        'username': username,
        'incident': incident,
        'timestamp': timestamp,
        'backport_ticket': issue.number,
        'backport_deadline': (datetime.now() + timedelta(hours=24)).isoformat(),
        'status': 'pending'
    })

# Daily check for overdue backports
def check_overdue_backports():
    """Alert on overdue break-glass backports"""
 
    table = dynamodb.Table('BreakGlassAudit')
 
    # Scan for pending backports past deadline
    response = table.scan(
        FilterExpression=Attr('status').eq('pending') &
                        Attr('backport_deadline').lt(datetime.now().isoformat())
    )
 
    for item in response['Items']:
        # Escalate to engineering manager
        send_escalation_email(
            to='engineering-managers@company.com',
            subject=f"OVERDUE: Break-glass backport from {item['username']}",
            body=f"""
            Break-glass usage from {item['timestamp']} has not been backported to IaC.
 
            User: {item['username']}
            Incident: {item['incident']}
            Deadline: {item['backport_deadline']}
            Status: OVERDUE
 
            Please follow up immediately.
            """
        )