Topic lessonTopic: Load balancing and troubleshooting
Troubleshoot ALB 503 errors: response origin, targets and health
When you see 503, first identify which component returned it. An ALB unable to dispatch a request, a configured fixed response and an application-generated 503 require different investigations. Correlate the same request/time across logs, rules, target groups and target state before restarting services or widening access.
Distinguish ALB-side and target responses, then choose evidence-based checks for missing/unused targets, failed health probes, routing configuration or application faults.
Know ALB listeners, rules, target groups, targets and health checks. Examples are fixed synthetic observations. This page accesses no AWS systems or logs and requests no real log contents, IPs or customer data.
First identify the responding layer
- Record the symptom → ALB evidenceIdentify the ALB, host and path, considering whether an upstream proxy returned the response.
- ALB evidence → Target evidenceCorrelate target evidence when dispatch occurred; otherwise investigate ALB actions and target eligibility.
- Record the symptom
- Record time/zone, URL, method and impact.
- ALB evidence
- Compare statuses, actions, rule and selected target group.
- Target evidence
- If dispatched, correlate the application response and dependencies.
Do not infer origin from the browser message or Server header alone. Account for any CDN/proxy and correlate the same request and time. ALB access logging is best effort: a missing entry alone does not prove the request never reached the ALB. Record unavailable evidence as a gap.
| Field | What to investigate | Limit |
|---|---|---|
| elb_status_code | The response code returned through the ALB | Distinguish fixed-response and WAF custom-response actions |
| target_status_code | The HTTP status received from a connected target | A dash means no target HTTP response was recorded, not one unique cause |
| actions_executed / matched_rule_priority | Executed actions and the matched rule | The group you are viewing may not be the one this request selected |
| target:port / target_group_arn | The target and group associated with the request | Correlate inside the authorized environment without pasting sensitive logs elsewhere |
Worked comparison: two records showing 503
Case A
elb_status_code: 503
target_status_code: -
actions_executed: forward
selected_group: reports-blue
Case B
elb_status_code: 503
target_status_code: 503
actions_executed: forward
selected_group: reports-greenCase A has no recorded target HTTP response. Review the selected group’s registrations, eligibility and relevant features; these fields alone do not prove the group is empty. Case B records a target 503, so investigate the application’s corresponding request. The visible code is identical, but the next evidence differs.
Choose an option to see the result and explanation.
For ALB-side 503, inspect configuration and target eligibility
| Condition or configuration | Evidence to inspect | Avoid assuming |
|---|---|---|
| No registered targets in the selected forward group | The group selected for this request and its registrations | Using a healthy target from a different group as proof |
| All registered targets are unused | Reason codes, EC2 state, listener association and enabled AZs | Treating unused as unhealthy and widening SGs |
| Cross-zone disabled; an enabled receiving AZ has no targets | Group attributes, per-AZ targets/capacity and request pattern | Assuming healthy targets in another AZ always receive the request |
| Target optimizer enabled, with no target ready | Feature enablement, agent state, readiness/concurrency and reject evidence | Assuming ordinary groups use the same agent configuration |
| A configured fixed 503 response | Matched rule/action and the intended maintenance behavior | Treating every 503 as an empty-group failure |
| A WAF custom response or upstream response | Executed actions and correlated upstream records | Inferring response origin from target health alone |
Target optimizer is an additional configuration using a target-side agent to manage concurrency and readiness. When enabled, investigate those conditions as well as health. Do not prescribe adding an agent or changing its limits in an architecture that does not use the feature.
With weighted forwarding to multiple target groups, an empty or all-unhealthy group does not automatically fail over to another healthy group. Match the request’s selected group to its weight, registrations and state.
Separate unused from unhealthy and read the reason code
| Example state/reason | Next checks | Not established by this label alone |
|---|---|---|
| unused / Target.NotInUse | Group/ALB association and whether the target AZ is enabled | That broader traffic permissions will necessarily fix it |
| unused / Target.InvalidState | Whether EC2 is stopped, terminated or otherwise invalid | That an application port is the cause |
| unhealthy / Target.ResponseCodeMismatch | Probe path, expected codes, actual response and application settings | That broadening success-code matching is the fix |
| unhealthy / Target.Timeout | Health port, SG/NACL return path, routes, listener and resource condition | That SGs or authentication keys alone explain it |
| initial / Registration or initial checks | Registration timing, initial probes and change history | That the condition is already a permanent failure |
Choose an option to see the result and explanation.
Investigate before changing: match the request to its configuration
- Record time/zone, host/path/method, persistent versus intermittent behavior and any user/AZ pattern. Do not enter secrets or personal data into this site.
- Correlate upstream, ALB and target evidence for the same request. Separate elb_status_code from target_status_code and inspect actions and the matched rule.
- Inspect the group actually selected by that request: registrations, target type, port, enabled AZs and group attributes.
- Separate unused, unhealthy and initial states. Follow the reason into probe path/port/success settings, SGs, NACLs, routes and application listeners.
- If the target returned 503, inspect the matching application operation, load, dependencies and maintenance settings. Running EC2 or healthy probes alone are insufficient.
- Correlate recent deployment, rule/weight, deregistration, AZ and Auto Scaling changes. Use available metrics as supporting evidence rather than a single-count diagnosis.
Fix the identified condition and verify the same request
| Established condition | Targeted response | Verification |
|---|---|---|
| Only the intended forwarding group is empty | Review registration of the intended targets in the correct group/AZs | Registrations, zones, health and the affected URL |
| A request unexpectedly matches a maintenance fixed response | Correct conditions/priority according to the intended behavior | The same host/path and other affected rules |
| A changed probe path was not reflected in configuration | Use the application contract to decide which side needs correction | Both the actual probe and ordinary requests |
| An application dependency failed | Resolve the dependency/application issue with its owner | Correct business results and dependency behavior, not just fewer 503s |
| Evidence is still insufficient | Agree on missing evidence and a safe collection method | Keep uncertainty explicit rather than reporting a guessed fix as success |
Blanket restarts, worldwide SG access, broad health-code matching or deregistering every target do not establish the cause. For an authorized change, record the evidence, impact and recovery plan, then verify the same request conditions, target state and application result. This lesson practices that reasoning and changes no real resources.
Check, then explain in your own words
- Explain elb_status_code versus target_status_code.
- Describe how the next evidence differs for unused and unhealthy.
- List the evidence needed before a fix and same-condition checks afterward.
Check your understanding: Can a 503 screen alone prove the target group is empty?
No. Correlate origin, executed action, selected group and target state.
Fixed responses, additional features, per-AZ conditions and application-generated 503s lead to different checks. Gather evidence for the same request and keep missing information explicit.
Sources and conditions checked
The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.
Scope and conditionselb_status_code versus target_status_code, target/port, executed actions and best-effort logging; a dash alone is not a unique diagnosis.
Used in this lessonFirst identify the responding layer Worked comparison: two records showing 503 For ALB-side 503, inspect configuration and target eligibility
- Action types for listener rulesChecked:
Scope and conditionsForward versus redirect and fixed responses; a configured fixed response can return a 5xx without contacting a target. Weighted forwarding does not automatically fail over from an empty/all-unhealthy group to another healthy group.
Used in this lessonFirst identify the responding layer For ALB-side 503, inspect configuration and target eligibility
Scope and conditionsALB-generated 503: empty/all-unused target groups or target optimizer no-ready-target condition; distinguish 502/504 and target-generated errors.
Used in this lessonWorked comparison: two records showing 503 For ALB-side 503, inspect configuration and target eligibility Separate unused from unhealthy and read the reason code
Scope and conditionsinitial/unhealthy/unused/draining states and reason codes, especially Target.NotInUse/InvalidState versus health-check failures.
Used in this lessonFor ALB-side 503, inspect configuration and target eligibility Separate unused from unhealthy and read the reason code
Scope and conditionsCross-zone disabled with an empty enabled AZ can cause 503 on requests entering that AZ. Routing/health thresholds and draining differ from generic all-unhealthy fail-open.
Used in this lessonFor ALB-side 503, inspect configuration and target eligibility Separate unused from unhealthy and read the reason code
Scope and conditionsTarget registration, target SGs, draining and Auto Scaling integration. Target optimizer uses a target-side agent for concurrency/readiness; special conditions are not assumed on ordinary groups.
Used in this lessonFor ALB-side 503, inspect configuration and target eligibility
Scope and conditionsPeriodic configured health requests, thresholds and all-unhealthy fail-open; not whole-application correctness.
Used in this lessonSeparate unused from unhealthy and read the reason code