F5 BIG-IP Troubleshooting

Troubleshooting Principles

1. Define the problem -- What exactly is broken? Who reported it? What is the expected vs actual behavior?
2. Gather facts -- List objects, check stats, read logs. Never assume.
3. Consider possibilities -- Based on facts, list likely root causes
4. Create action plan -- Test one variable at a time
5. Implement and verify -- Make one change, verify, document
6. Document -- Record what was found and what fixed it

How to Call the Tools

The F5 MCP server provides 6 tools. Call them via mcp-call with the required environment variables:

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" <tool_name> '{"param":"value"}'

Available Tools for Troubleshooting

Symptom: "Virtual Server Not Responding to Clients"

Clients report they cannot connect to the application VIP.

Step 1: Verify Virtual Server Exists and Is Enabled

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"vs_webapp_https","object_type":"virtual"}'

Check:
- Does the virtual server exist? If not, it was deleted or never created.
- Is it enabled: true? If disabled, someone took it out of service.
- Is the destination (VIP:port) correct?
- Is a pool assigned?
- Is sourceAddressTranslation configured? (Without SNAT/automap, return traffic may bypass the BIG-IP.)

Decision tree:
- VS does not exist -> Recreate it (use f5-config-mgmt skill)
- VS is disabled -> Re-enable: update_tool with {"enabled":true}
- VS has no pool -> Assign pool: update_tool with {"pool":"pool_name"}
- VS has no SNAT -> Check if servers have BIG-IP as default gateway; if not, add automap

Step 2: Check Virtual Server Statistics

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" show_stats_tool '{"object_name":"vs_webapp_https","object_type":"virtual"}'

Analyze:

If status is offline:
The virtual server is marked down because the associated pool has no available members. Proceed to Step 3.

If current connections = 0 but status is available:
The VIP is healthy but no clients are connecting. The issue is upstream of the BIG-IP:
- DNS not resolving to the VIP address
- Firewall blocking traffic to the VIP
- Client network routing issue
- VIP is on wrong VLAN/subnet

Step 3: Check the Associated Pool

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"pool_webapp","object_type":"pool"}'

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" show_stats_tool '{"object_name":"pool_webapp","object_type":"pool"}'

Check:
- Are any members available? If all members are offline, the pool is down.
- What monitor is assigned? Is it appropriate for the service?
- Are members enabled or disabled? Disabled members were intentionally drained.
- What is the member-to-connection distribution? Is one member handling all traffic?

If all members are offline -> Go to "Pool Member Marked Down" section below.

Step 4: Check Logs for Errors

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" show_logs_tool '{"lines_number":"200"}'

Scan for:
- 01010028 -- No members available in pool (confirms pool down)
- 01010025 -- Connection limit reached on virtual server
- 0107142f -- SSL handshake failure
- 01070417 -- HTTP parse error
- 01010240 -- Connection queue full
- Timestamps correlating with the reported outage

Step 5: Check Profiles and iRules

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"","object_type":"profile"}'

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"","object_type":"irule"}'

Check:
- Is the correct SSL profile assigned for HTTPS virtual servers?
- Is the HTTP profile assigned when HTTP inspection is needed?
- Are any iRules rejecting or redirecting traffic incorrectly?
- Is a persistence profile causing traffic to stick to a down member?

Symptom: "Pool Member Marked Down"

Health monitor is marking one or more pool members as offline.

Step 1: Identify Which Members Are Down

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"pool_webapp","object_type":"pool"}'

Record: Which members are offline, which are available, which are disabled.

Step 2: Check Pool Statistics for the Down Member

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" show_stats_tool '{"object_name":"pool_webapp","object_type":"pool"}'

Analyze:
- When did the member go down? (Check stats timestamps)
- Was there a gradual decline or sudden failure?
- Are connections draining from the down member?

Step 3: Check Logs for Monitor Failure Details

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" show_logs_tool '{"lines_number":"500"}'

Scan for these patterns:

Common root causes for pool member down:

1. Server is actually down -- The application crashed, the OS is down, or the server was rebooted
2. Network path issue -- Firewall between BIG-IP and server blocking health check traffic, or routing issue on server VLAN
3. Monitor mismatch -- HTTP monitor expecting 200 but application returns 301/302 redirect
4. Monitor URI wrong -- Health check URI returns 404 because the page does not exist
5. Port mismatch -- Monitor checking wrong port (e.g., monitor on 80 but server on 8080)
6. SSL mismatch -- HTTP monitor used but server requires HTTPS (or vice versa)
7. Response timeout -- Server responds but too slowly for the monitor interval/timeout
8. Receive string mismatch -- Monitor expects specific string in response that changed after app deployment
9. Source IP issue -- Server firewall blocking the BIG-IP self-IP used for health checks

Step 4: Verify Monitor Configuration

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"pool_webapp","object_type":"pool"}'

From the pool config, identify the monitor name and verify:
- Type: HTTP, HTTPS, TCP, ICMP, or custom
- Interval/timeout: Is the timeout shorter than the interval? (Must be: timeout < interval * 3+1 for 3 failures)
- Send string: What request is sent? (e.g., GET /health HTTP/1.1\r\nHost: app.example.com\r\n\r\n)
- Receive string: What response is expected? (e.g., 200 OK or healthy)
- Destination: Is it *:* (use member address:port) or a specific IP:port?

Step 5: Remediation

If the server is healthy but the monitor is wrong, fix the monitor:

Update the pool with a correct monitor:

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" update_tool '{"url_body":{"monitor":"tcp"},"object_type":"pool","object_name":"pool_webapp"}'

If a member needs to be temporarily removed (graceful drain):

Update the pool without the problematic member:

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" update_tool '{"url_body":{"members":["10.1.1.10:80","10.1.1.11:80"]},"object_type":"pool","object_name":"pool_webapp"}'

WARNING: This removes the member entirely. Existing connections will be terminated. For graceful drain, disable the member instead if the API supports it.

If a replacement member needs to be added:

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" update_tool '{"url_body":{"members":["10.1.1.10:80","10.1.1.11:80","10.1.1.14:80"]},"object_type":"pool","object_name":"pool_webapp"}'

Symptom: "Connection Limits / Persistence Issues"

Users report intermittent connectivity, session drops, or being load-balanced to a different server mid-session.

Step 1: Check Virtual Server Connection Statistics

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" show_stats_tool '{"object_name":"vs_webapp_https","object_type":"virtual"}'

Check for connection limit issues:
- Is connectionLimit set and being reached?
- Are clientsideCurConns near the limit?
- Is the connection queue filling up? (Check logs for 01010240)

If connection limit is being hit:

Either increase the limit or scale out with additional pool members:

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" update_tool '{"url_body":{"connectionLimit":0},"object_type":"virtual","object_name":"vs_webapp_https"}'

Setting connectionLimit to 0 removes the limit entirely.

Step 2: Check Persistence Configuration

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"vs_webapp_https","object_type":"virtual"}'

Persistence troubleshooting:

Step 3: Check Pool Member Connection Distribution

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" show_stats_tool '{"object_name":"pool_webapp","object_type":"pool"}'

If one member has vastly more connections than others:
- Persistence is sticking too many sessions to one member
- Consider changing from source-address to cookie persistence
- Consider changing load balancing method from round-robin to least-connections

Step 4: Check Logs for Connection Errors

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" show_logs_tool '{"lines_number":"300"}'

Scan for:
- 01010025 -- Connection limit reached
- 01010240 -- Connection queue full
- 01060102 -- Rate limit reached
- TCL error -- iRule causing connection drops
- reset cause -- Connection resets (RST) from server or BIG-IP

Symptom: "SSL/TLS Certificate Problems"

Users see certificate warnings, SSL handshake failures, or HTTPS connections fail entirely.

Step 1: Check SSL Profile Configuration

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"","object_type":"profile"}'

Check the SSL client profile assigned to the virtual server:
- Is a client SSL profile assigned? (Required for HTTPS VIPs)
- Which certificate and key are referenced?
- What TLS versions are enabled? (TLS 1.2 and 1.3 should be enabled; TLS 1.0 and 1.1 should be disabled)
- What cipher suites are configured?

Common SSL issues:

Step 2: Check Virtual Server for SSL Profile

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"vs_webapp_https","object_type":"virtual"}'

Verify the correct SSL profile is assigned in the profiles list with context: clientside.

Step 3: Check Logs for SSL Errors

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" show_logs_tool '{"lines_number":"300"}'

Key SSL log messages:

Step 4: Remediation

Update SSL profile ciphers to modern standards:

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" update_tool '{"url_body":{"ciphers":"TLSv1.2:TLSv1.3:!SSLv3:!RC4:!3DES:!EXPORT"},"object_type":"profile","object_name":"clientssl_webapp"}'

Assign the correct SSL profile to a virtual server:

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" update_tool '{"url_body":{"profiles":[{"name":"clientssl_webapp","context":"clientside"},{"name":"http"},{"name":"tcp-wan-optimized","context":"clientside"},{"name":"tcp-lan-optimized","context":"serverside"}]},"object_type":"virtual","object_name":"vs_webapp_https"}'

WARNING: The profiles list is a full replacement. Include ALL desired profiles.

Symptom: "iRule Errors in Logs"

Logs show TCL errors or iRule-related failures.

Step 1: Pull Recent Logs

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" show_logs_tool '{"lines_number":"500"}'

Scan for iRule error patterns:

Step 2: Identify the Problematic iRule

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"","object_type":"irule"}'

Cross-reference the iRule name from the log error with the iRule inventory. Check which virtual servers have this iRule assigned.

Step 3: Review iRule Content

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"problematic_irule","object_type":"irule"}'

Common iRule bugs to check for:
- Variables used across events without being set in all code paths
- HTTP::collect without corresponding HTTP::release in all branches
- Missing default case in switch statements
- Regex patterns that can cause catastrophic backtracking
- log statements in high-traffic events (performance issue, not error)
- String operations on binary data
- Missing error handling (catch) around operations that can fail

Step 4: Fix the iRule

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" update_tool '{"url_body":{"apiAnonymous":"when HTTP_REQUEST {\n  catch {\n    switch -glob [string tolower [HTTP::uri]] {\n      \"/api/*\" { pool pool_api_backend }\n      default { pool pool_webapp }\n    }\n  } err {\n    log local0. \"iRule error: $err\"\n    pool pool_webapp\n  }\n}"},"object_type":"irule","object_name":"uri_routing"}'

Alternatively, if the iRule is causing critical failures, remove it from the virtual server immediately:

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" update_tool '{"url_body":{"rules":[]},"object_type":"virtual","object_name":"vs_webapp_https"}'

This removes all iRules from the virtual server. Traffic will flow to the default pool without any iRule processing. Fix the iRule, then re-attach it.

Symptom: "Performance Degradation"

Application is slow, high latency, or throughput has dropped.

Step 1: Check Virtual Server Statistics

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" show_stats_tool '{"object_name":"vs_webapp_https","object_type":"virtual"}'

Look for:
- Connection count near the limit -> Bottleneck at the VIP
- High bits/sec relative to interface capacity -> Bandwidth saturation
- Connection rate spike -> Possible DDoS or legitimate traffic surge
- Asymmetric traffic (high client-side, low server-side) -> Backend not keeping up

Step 2: Check Pool Member Distribution

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" show_stats_tool '{"object_name":"pool_webapp","object_type":"pool"}'

Look for:
- Uneven connection distribution -> Some members overloaded, others idle
- Single member with most connections -> Persistence issue or members down
- All members at high connection count -> Need more backend capacity
- High server-side connection time -> Backend application slow

If distribution is uneven, consider changing load balancing:

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" update_tool '{"url_body":{"loadBalancingMode":"least-connections-member"},"object_type":"pool","object_name":"pool_webapp"}'

Step 3: Check for Pool Members Down (Reduced Capacity)

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"pool_webapp","object_type":"pool"}'

If members are down, the remaining members are handling more traffic than designed. This is the most common cause of "slow application" reports -- not a BIG-IP issue but a capacity issue.

Step 4: Check System Logs for Errors

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" show_logs_tool '{"lines_number":"500"}'

Performance-related log patterns:

Step 5: Check iRules for Performance Impact

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"","object_type":"irule"}'

iRule performance killers:
- log statements on every request -> Disk I/O bottleneck
- Complex regex matching -> CPU overhead
- HTTP::collect large payloads -> Memory consumption
- DNS::lookup in data path -> Blocking operation, adds latency
- Multiple iRules with same events -> Event processing overhead
- persist uie with large strings -> Persistence table bloat

Step 6: Scale Out (If Root Cause Is Capacity)

If the root cause is insufficient backend capacity, add more pool members:

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" update_tool '{"url_body":{"members":["10.1.1.10:80","10.1.1.11:80","10.1.1.12:80","10.1.1.13:80","10.1.1.14:80"]},"object_type":"pool","object_name":"pool_webapp"}'

WARNING: Members list is a full replacement. Include ALL desired members (existing + new).

Symptom: "HA Failover or Sync Issues"

Logs indicate high-availability state changes, failover events, or configuration sync failures.

Step 1: Check System Logs for HA Events

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" show_logs_tool '{"lines_number":"500"}'

HA-related log patterns:

Step 2: Verify Object State After Failover

After any failover event, immediately verify all virtual servers and pools:

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"","object_type":"virtual"}'

IP_ADDRESS=$F5_IP_ADDRESS Authorization_string=$F5_AUTH_STRING python3 $MCP_CALL "python3 -u $F5_MCP_SCRIPT" list_tool '{"object_name":"","object_type":"pool"}'

Confirm all virtual servers are available and all pool members are healthy on the now-active unit.

Common F5 Error Code Quick Reference

Troubleshooting Decision Flowchart

Client reports application down
|
+-> Check VIP status (list_tool + show_stats_tool virtual)
    |
    +-> VIP offline?
    |   +-> Check pool (list_tool + show_stats_tool pool)
    |       +-> All members down? -> Check servers + monitors
    |       +-> Some members down? -> Reduced capacity, check remaining
    |       +-> No pool assigned? -> Assign pool (update_tool)
    |
    +-> VIP available but 0 connections?
    |   +-> DNS, firewall, or routing issue upstream of BIG-IP
    |
    +-> VIP available, connections present, but errors?
        +-> Check logs (show_logs_tool)
        +-> SSL errors? -> Check profiles + certs
        +-> HTTP errors? -> Check iRules + backend health
        +-> Connection limits? -> Scale out or increase limits

Integration with Other Skills

GAIT Audit Trail

After completing a troubleshooting session, record findings and resolution in GAIT:

python3 $MCP_CALL "python3 -u $GAIT_MCP_SCRIPT" gait_record_turn '{"artifacts":["f5-troubleshoot-report.txt"],"user_text":"F5 troubleshoot: vs_webapp_https not responding to clients","assistant_text":"Investigation: VIP status offline due to pool_webapp all members down. Root cause: HTTP health monitor expecting 200 but app returning 301 redirect after deployment. Fix: updated monitor receive string to accept 301. Verification: all 3 pool members now available, VIP status available, client connections incrementing. Logs clear of 01010028 errors."}'

Audit examples are illustrative. Replace their request and outcomes with observed
session evidence; inspect MCP isError, returned ok and recorded GAIT text.