INFRASTRUCTURE LEARNINGINFRASTRUCTURE LEARNING
インフラ大全インフラ大全IT INFRASTRUCTURE / LEARNING
Infrastructure learning guide
Find a lesson or lab/

Esc to close · Search terms stay on your device

LEARNING日本語

Topic lessonTopic: Load balancing and troubleshooting

Troubleshoot ALB 503 errors: response origin, targets and health

When you see 503, first identify which component returned it. An ALB unable to dispatch a request, a configured fixed response and an application-generated 503 require different investigations. Correlate the same request/time across logs, rules, target groups and target state before restarting services or widening access.

What you will learn

Distinguish ALB-side and target responses, then choose evidence-based checks for missing/unused targets, failed health probes, routing configuration or application faults.

Prerequisite

Know ALB listeners, rules, target groups, targets and health checks. Examples are fixed synthetic observations. This page accesses no AWS systems or logs and requests no real log contents, IPs or customer data.

First identify the responding layer

Investigation order: arrows describe human checks, not a traffic path
Investigation order: arrows describe human checks, not a traffic path1: Record the symptom — Identify the ALB, host and path, considering whether an upstream proxy returned the response. — ALB evidence. 2: ALB evidence — Correlate target evidence when dispatch occurred; otherwise investigate ALB actions and target eligibility. — Target evidence12Record the symptomALB evidenceTarget evidenceInvestigation order: arrows describe human checks, not a traffic path1: Record the symptom — Identify the ALB, host and path, considering whether an upstream proxy returned the response. — ALB evidence. 2: ALB evidence — Correlate target evidence when dispatch occurred; otherwise investigate ALB actions and target eligibility. — Target evidence12Record thesymptomALB evidenceTarget evidence
  1. Record the symptom → ALB evidenceIdentify the ALB, host and path, considering whether an upstream proxy returned the response.
  2. ALB evidence → Target evidenceCorrelate target evidence when dispatch occurred; otherwise investigate ALB actions and target eligibility.
Record the symptom
Record time/zone, URL, method and impact.
ALB evidence
Compare statuses, actions, rule and selected target group.
Target evidence
If dispatched, correlate the application response and dependencies.

Do not infer origin from the browser message or Server header alone. Account for any CDN/proxy and correlate the same request and time. ALB access logging is best effort: a missing entry alone does not prove the request never reached the ALB. Record unavailable evidence as a gap.

Useful access-log fields to correlate
FieldWhat to investigateLimit
elb_status_codeThe response code returned through the ALBDistinguish fixed-response and WAF custom-response actions
target_status_codeThe HTTP status received from a connected targetA dash means no target HTTP response was recorded, not one unique cause
actions_executed / matched_rule_priorityExecuted actions and the matched ruleThe group you are viewing may not be the one this request selected
target:port / target_group_arnThe target and group associated with the requestCorrelate inside the authorized environment without pasting sensitive logs elsewhere

Worked comparison: two records showing 503

Synthetic selected fields for investigation; this is not the raw AWS log-line format
Case A
elb_status_code: 503
target_status_code: -
actions_executed: forward
selected_group: reports-blue

Case B
elb_status_code: 503
target_status_code: 503
actions_executed: forward
selected_group: reports-green

Case A has no recorded target HTTP response. Review the selected group’s registrations, eligibility and relevant features; these fields alone do not prove the group is empty. Case B records a target 503, so investigate the application’s corresponding request. The visible code is identical, but the next evidence differs.

Choose an option to see the result and explanation.

As in Case B, target_status_code also contains 503. What should you investigate first?
Read the answer and reason

Answer: Correlate the matching target request, application and dependencies · The record includes a target-generated 503. Investigate that request’s application load, dependencies or maintenance behavior using evidence; recreating the ALB does not explain the cause.

For ALB-side 503, inspect configuration and target eligibility

Candidate conditions and evidence; no single row explains every 503
Condition or configurationEvidence to inspectAvoid assuming
No registered targets in the selected forward groupThe group selected for this request and its registrationsUsing a healthy target from a different group as proof
All registered targets are unusedReason codes, EC2 state, listener association and enabled AZsTreating unused as unhealthy and widening SGs
Cross-zone disabled; an enabled receiving AZ has no targetsGroup attributes, per-AZ targets/capacity and request patternAssuming healthy targets in another AZ always receive the request
Target optimizer enabled, with no target readyFeature enablement, agent state, readiness/concurrency and reject evidenceAssuming ordinary groups use the same agent configuration
A configured fixed 503 responseMatched rule/action and the intended maintenance behaviorTreating every 503 as an empty-group failure
A WAF custom response or upstream responseExecuted actions and correlated upstream recordsInferring response origin from target health alone

Target optimizer is an additional configuration using a target-side agent to manage concurrency and readiness. When enabled, investigate those conditions as well as health. Do not prescribe adding an agent or changing its limits in an architecture that does not use the feature.

With weighted forwarding to multiple target groups, an empty or all-unhealthy group does not automatically fail over to another healthy group. Match the request’s selected group to its weight, registrations and state.

Separate unused from unhealthy and read the reason code

Use state and reason to narrow the next check
Example state/reasonNext checksNot established by this label alone
unused / Target.NotInUseGroup/ALB association and whether the target AZ is enabledThat broader traffic permissions will necessarily fix it
unused / Target.InvalidStateWhether EC2 is stopped, terminated or otherwise invalidThat an application port is the cause
unhealthy / Target.ResponseCodeMismatchProbe path, expected codes, actual response and application settingsThat broadening success-code matching is the fix
unhealthy / Target.TimeoutHealth port, SG/NACL return path, routes, listener and resource conditionThat SGs or authentication keys alone explain it
initial / Registration or initial checksRegistration timing, initial probes and change historyThat the condition is already a permanent failure

Choose an option to see the result and explanation.

Every target is unhealthy. Which conclusion is justified by that fact alone?
Read the answer and reason

Answer: Health reasons and actual response origin still need separate investigation · Fail-open and other configured behavior mean states and HTTP codes are not one-to-one. Correlate reason codes with the affected request and keep unconfirmed causes unconfirmed.

Investigate before changing: match the request to its configuration

  1. Record time/zone, host/path/method, persistent versus intermittent behavior and any user/AZ pattern. Do not enter secrets or personal data into this site.
  2. Correlate upstream, ALB and target evidence for the same request. Separate elb_status_code from target_status_code and inspect actions and the matched rule.
  3. Inspect the group actually selected by that request: registrations, target type, port, enabled AZs and group attributes.
  4. Separate unused, unhealthy and initial states. Follow the reason into probe path/port/success settings, SGs, NACLs, routes and application listeners.
  5. If the target returned 503, inspect the matching application operation, load, dependencies and maintenance settings. Running EC2 or healthy probes alone are insufficient.
  6. Correlate recent deployment, rule/weight, deregistration, AZ and Auto Scaling changes. Use available metrics as supporting evidence rather than a single-count diagnosis.

Fix the identified condition and verify the same request

Examples after evidence is established; changes require authority
Established conditionTargeted responseVerification
Only the intended forwarding group is emptyReview registration of the intended targets in the correct group/AZsRegistrations, zones, health and the affected URL
A request unexpectedly matches a maintenance fixed responseCorrect conditions/priority according to the intended behaviorThe same host/path and other affected rules
A changed probe path was not reflected in configurationUse the application contract to decide which side needs correctionBoth the actual probe and ordinary requests
An application dependency failedResolve the dependency/application issue with its ownerCorrect business results and dependency behavior, not just fewer 503s
Evidence is still insufficientAgree on missing evidence and a safe collection methodKeep uncertainty explicit rather than reporting a guessed fix as success

Blanket restarts, worldwide SG access, broad health-code matching or deregistering every target do not establish the cause. For an authorized change, record the evidence, impact and recovery plan, then verify the same request conditions, target state and application result. This lesson practices that reasoning and changes no real resources.

Check, then explain in your own words

  1. Explain elb_status_code versus target_status_code.
  2. Describe how the next evidence differs for unused and unhealthy.
  3. List the evidence needed before a fix and same-condition checks afterward.
Check your understanding: Can a 503 screen alone prove the target group is empty?

No. Correlate origin, executed action, selected group and target state.

Fixed responses, additional features, per-AZ conditions and application-generated 503s lead to different checks. Gather evidence for the same request and keep missing information explicit.

Sources and conditions checked

The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.