INFRASTRUCTURE LEARNINGINFRASTRUCTURE LEARNING
インフラ大全インフラ大全IT INFRASTRUCTURE / LEARNING
Infrastructure learning guide
Find a lesson or lab/

Esc to close · Search terms stay on your device

LEARNING日本語

Topic lessonTopic: Amazon Bedrock readiness, privacy and evaluation

Evaluate Amazon Bedrock changes with fixed cases and severe failure limits

A higher average can conceal a serious operational mistake. Compare model or prompt changes on the same cases and decide against predefined acceptance gates.

What you will learn

Compare model or prompt changes on the same cases and decide against predefined acceptance gates.

Prerequisite

This exercise uses fictional evidence and makes no changes in a live environment.

Define evaluation conditions before running

Bedrock evaluation uses the selected resources and data. Match the metrics and decision rules to the intended workload before reviewing results.

From comparison to decision
From comparison to decision1: Input — Fix input version — Variants. 2: Variants — Compare on equal conditions — Evaluate. 3: Evaluate — Record reproducibility — Decision123InputVariantsEvaluateDecisionFrom comparison to decision1: Input — Fix input version — Variants. 2: Variants — Compare on equal conditions — Evaluate. 3: Evaluate — Record reproducibility — Decision123InputVariantsEvaluateDecision
  1. Input → VariantsFix input version
  2. Variants → EvaluateCompare on equal conditions
  3. Evaluate → DecisionRecord reproducibility
Input
Fixed synthetic cases
Variants
Current and proposed
Evaluate
Accuracy and severe errors
Decision
Check independent gates

Twenty synthetic operations FAQ cases

Acceptance requires zero severe mistakes in addition to ordinary accuracy.
MetricCurrent to candidateDecision
Correct answers16 to 18Record improved cases
Severe mistakes0 to 1Violates acceptance
p95 latency700 to 900 msCheck service limits
EvidenceInput, model and configurationPreserve reproducibility

The candidate answers more cases correctly but fails a required safety gate, so hold it for review.

Evidence for the decision

Define the metrics Record the success criteria, severe failures and representative input scope.

Freeze the evaluation input Preserve synthetic documents, expected answers, configurations and duplicate rules.

Review important cases manually Inspect severe examples and dataset validity alongside aggregate metrics.

Check your decision

Choose an option to see the result and explanation.

Would you accept 18 correct answers versus 16?
Read the answer and reason

Answer: Hold and investigate the severe case · The candidate violates an independent condition.

What should be held constant when comparing candidates?
Read the answer and reason

Answer: Case version and scoring criteria · Control variables other than the intended change.

What do you retain after retesting?
Read the answer and reason

Answer: Version, conditions, severe cases and decision · The evidence must explain the decision.

Check, then explain in your own words

  1. Define acceptance gates first.
  2. Use identical synthetic cases.
  3. Preserve severe examples and change history.
Check your understanding: Do twenty passing synthetic cases guarantee production behavior?

Record representativeness and untested scenarios.

Results apply to the conditions exercised.

Sources and conditions checked

The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.