Troubleshooter

Overview

This skill provides systematic troubleshooting methodology for SF-Bench issues. It covers evaluation failures, patch application errors, scratch org problems, and debugging strategies.

When This Skill Applies

• Evaluation failures or unexpected errors
• Patch application issues
• Scratch org creation or deployment problems
• Debugging test failures
• Investigating root causes

Troubleshooting Methodology

Step 1: Categorize the Error

Step 2: Gather Evidence

1. Check Logs
   ```bash
   # View recent logs
   ls -la logs/

# Search for errors
grep -r "ERROR" logs/

# View specific task log
cat logs/<run_id>/<model>/<task_id>.log
```

2. Check Git Status
   ```bash
   # Verify no manual tweaks
   git status

# View applied patch
git diff
```

3. Check Scratch Org
   ```bash
   # List orgs
   sf org list --all

# View org details
sf org display -o <alias>

# Check deployment status
sf project deploy report -o <alias>
```

Step 3: Identify Root Cause

Use the error message to narrow down:

Error Message → Category → Root Cause → Fix

Step 4: Apply Fix and Verify

1. Apply the fix
2. Run relevant tests
3. Verify the issue is resolved
4. Document if pattern-based

Common Error Patterns

Patch Application Failures

Error: "patch does not apply"

Cause: Patch context doesn't match file content

Diagnosis:

# Check file content
cat <file-path>

# Check patch preview
head -50 solution.patch

Solutions:
1. Verify correct repository version
2. Use fuzzy matching: git apply --3way
3. Regenerate solution with updated context

Error: "corrupt patch at line X"

Cause: Malformed patch format

Diagnosis:

# Validate patch syntax
git apply --check solution.patch

Solutions:
1. Check for markdown fences in patch
2. Remove standalone +/- lines
3. Verify patch header format

Scratch Org Failures

Error: "SCRATCH_ORG_LIMIT_EXCEEDED"

Cause: Daily scratch org creation limit reached

Diagnosis:

# Check org limits
sf limits api display -o <devhub>

Solutions:
1. Delete unused scratch orgs
2. Wait for daily limit reset
3. Use different DevHub

Error: "InvalidOrgForCreate"

Cause: DevHub not authenticated or expired

Diagnosis:

# Check DevHub status
sf org list --all | grep DevHub

Solutions:
1. Re-authenticate: sf org login web -d
2. Verify DevHub features enabled
3. Check org expiration

Deployment Failures

Error: "CANNOT_ENABLE_DEVELOPER_MODE"

Cause: Scratch org config issue

Diagnosis:

# Check scratch def
cat config/project-scratch-def.json

Solutions:
1. Verify features in scratch def
2. Check for conflicting settings
3. Use minimal scratch def

Error: "Test coverage is XX%, at least 75% required"

Cause: Insufficient test coverage

Diagnosis:

# Run tests with coverage
sf apex run test -o <alias> -c -r human

Solutions:
1. Add more test methods
2. Cover edge cases
3. Check for uncovered branches

API Failures

Error: "rate_limit_exceeded"

Cause: AI provider rate limit

Diagnosis:
- Check request frequency
- Review provider dashboard

Solutions:
1. Implement rate limiting
2. Use exponential backoff
3. Switch to different provider

Error: "invalid_api_key"

Cause: Missing or incorrect API key

Diagnosis:

# Check environment
echo $ROUTELLM_API_KEY | head -c 10

Solutions:
1. Verify key in .env
2. Source environment: source .env
3. Regenerate API key

Diagnostic Commands

Preflight Diagnostics

# Run preflight checks
python -c "
from sfbench.utils.preflight import PreflightValidator
v = PreflightValidator()
r = v.run_all_checks('model-name', 'provider')
print('PASSED' if r.passed else 'FAILED')
for c in r.checks:
    print(f'{c.name}: {c.status}')
"

Scratch Org Diagnostics

# Create debug org
sf org create scratch -f config/project-scratch-def.json -a debug-org -d 1

# Deploy and check
sf project deploy start -o debug-org

# Run specific test
sf apex run test -o debug-org -n TestClassName -r human -c

# Cleanup
sf org delete scratch -o debug-org -p

Patch Diagnostics

# Validate patch
git apply --check solution.patch

# Apply with verbose output
git apply -v solution.patch

# Apply with fallback strategies
git apply --3way solution.patch
git apply --reject solution.patch

Escalation Path

If issue persists after troubleshooting:

1. Document the Issue
   - Error message
   - Steps to reproduce
   - Environment details
   - Attempted solutions

2. Update activeContext.md
   - Add to "Blockers" section
   - Note impact on current work

3. Create Issue
   - Use bug report template
   - Include all diagnostic information
   - Tag appropriately

Prevention Strategies

For Patch Issues

• Validate patches before applying
• Use multi-strategy fallback
• Clean patches before application

For Scratch Org Issues

• Monitor org limits
• Always cleanup in finally blocks
• Use retry with exponential backoff

For API Issues

• Implement rate limiting
• Cache responses where appropriate
• Use circuit breaker pattern for repeated failures