Automation Debt

Automation debt is test code that is:
  - Hard to understand (no structure, magic values everywhere)
  - Hard to maintain (breaks when unrelated code changes)
  - Hard to trust (flaky, slow, or tests the wrong things)
  - Hard to extend (copy-paste to add new scenarios)

Unlike production code debt, automation debt is often invisible to stakeholders.
"Tests are passing" looks the same whether tests are pristine or held together with duct tape.
The cost shows up as: sprint velocity drops when "just adding a test" takes a day.

Signal                          | Likely cause
────────────────────────────────|────────────────────────────────────────
Tests take > 20 min in CI      | Missing parallelism, over-broad E2E coverage
Flaky test rate > 5%           | Race conditions, missing waits, shared state
"Don't touch the test" culture | Tests coupled to implementation details
Adding a feature breaks 20+ tests | Lack of abstraction, missing fixtures
Test setup > 50 lines          | Missing factories, fixtures not composable
Same selector in 30 tests      | No Page Object or component object layer
Tests that test the test tool  | Testing mock setup, not real behaviour
No-one can explain what a test | Missing intent in naming + assertion messages
 actually verifies             |
Tests that always pass         | Assertions that never fail (soft assertions, wrong comparisons)

# Identify debt programmatically before deciding what to fix

import subprocess, re
from pathlib import Path

def find_magic_strings(test_dir: str) -> list[str]:
    """Hard-coded selectors and values that should be in fixtures/constants."""
    result = subprocess.run(
        ["rg", "--type", "py", r'find_element_by|By\.(XPATH|CSS_SELECTOR)\s*,\s*"[^"]{40,}"',
         test_dir],
        capture_output=True, text=True,
    )
    return result.stdout.splitlines()

def count_duplicate_selectors(test_dir: str) -> dict[str, int]:
    """Selectors repeated across tests — Page Object candidates."""
    selectors: dict[str, int] = {}
    for f in Path(test_dir).rglob("*.py"):
        for match in re.finditer(r'"(#[\w-]+|\.[\w-]+|\[[\w-]+=)', f.read_text()):
            sel = match.group(1)
            selectors[sel] = selectors.get(sel, 0) + 1
    return {k: v for k, v in sorted(selectors.items(), key=lambda x: -x[1]) if v > 3}

def tests_without_assertions(test_dir: str) -> list[str]:
    """Tests that call the system but assert nothing."""
    findings = []
    for f in Path(test_dir).rglob("test_*.py"):
        text = f.read_text()
        for m in re.finditer(r'def (test_\w+)', text):
            # Find the function body up to the next def
            start = m.start()
            body = text[start:start + 2000]
            if not re.search(r'assert |expect\(', body):
                findings.append(f"{f}::{m.group(1)}")
    return findings

Debt quadrant:
                  High Impact
                       |
   Slow to fix         |         Quick to fix
   High impact         |         High impact
   (schedule time)     |         (fix now)
                       |
─────────────────────────────────────────── Effort
                       |
   Slow to fix         |         Quick to fix
   Low impact          |         Low impact
   (maybe never)       |         (batch with related work)
                       |
                  Low Impact

High-impact debt:
  - Flaky tests (undermine confidence in entire suite)
  - Tests that take > 5 minutes (block CI feedback loop)
  - Missing test for a defect that escaped to production

Low-impact debt:
  - Inconsistent naming conventions
  - Missing docstrings
  - Duplicated test data that doesn't cause failures

# Pattern 1: Extract Page Objects from inline selectors
# Before (debt)
async def test_checkout_flow(page):
    await page.click("#checkout-btn")
    await page.fill("input[name='card-number']", "4242 4242 4242 4242")
    await page.click(".submit-payment")
    assert await page.locator(".order-confirmation").is_visible()

# After (debt paid)
from pages.checkout import CheckoutPage

async def test_checkout_flow(page, checkout_page: CheckoutPage):
    await checkout_page.fill_card("4242 4242 4242 4242")
    await checkout_page.submit_payment()
    assert await checkout_page.confirmation_visible()

# Pattern 2: Replace magic values with constants / factories
# Before
response = client.post("/api/orders", json={
    "product_id": "prod_abc123", "quantity": 3, "user_id": "usr_xyz789"
})

# After
from tests.factories import OrderFactory

response = client.post("/api/orders", json=OrderFactory.valid_order())
# Factory owns the valid structure; tests describe the variation
response_big = client.post("/api/orders", json=OrderFactory.valid_order(quantity=100))

# Pattern 3: Quarantine and fix flaky tests
# pytest.ini
[pytest]
markers =
    flaky: mark test as known-flaky, runs in quarantine only

# conftest.py
def pytest_collection_modifyitems(items, config):
    if not config.getoption("--run-flaky"):
        skip_flaky = pytest.mark.skip(reason="Quarantined flaky test")
        for item in items:
            if "flaky" in item.keywords:
                item.add_marker(skip_flaky)

# CI: main suite never includes flaky; nightly run includes --run-flaky for tracking

## Automation Debt Reduction — Q2 2026

### Current State
- Flaky test rate: 8% (target: < 1%)
- Mean CI time: 24 minutes (target: < 10 min)
- Tests with no assertions: 17
- Duplicate selectors (> 3 uses): 43 selectors across 8 files

### Sprint 14 (Quick Wins)
- [ ] Quarantine 12 known-flaky tests (unblock CI confidence)
- [ ] Fix the 5 no-assertion tests (trivial — assertions were commented out)
- [ ] Extract CheckoutPage object (used in 14 tests, 6 selectors duplicated)

### Sprint 15
- [ ] Introduce parallel test execution (target: 24 min → 10 min)
- [ ] Fix root cause of top 3 flaky tests (timing issues in payment flow)
- [ ] Add OrderFactory to replace 22 instances of hard-coded order JSON

### Sprint 16
- [ ] Refactor auth setup into session-scoped fixture (saves ~3s per test)
- [ ] Add Allure tagging to all tests (enables filtering and reporting)
- [ ] Fix remaining 9 flaky tests

### Success Metrics
| Metric | Now | Sprint 14 | Sprint 16 |
|--------|-----|-----------|-----------|
| Flaky rate | 8% | 5% | 1% |
| CI time | 24 min | 20 min | 10 min |
| Duplicate selectors | 43 | 30 | 10 |