INFRASTRUCTURE LEARNINGINFRASTRUCTURE LEARNING
インフラ大全インフラ大全IT INFRASTRUCTURE / LEARNING
Infrastructure learning guide
Find a lesson or lab/

Esc to close · Search terms stay on your device

LEARNING日本語

Topic lessonTopic: AWS data pipelines and SQL Server integration

Kinesis ordering and checkpoints for SQL Server changes

Preserve the order of changes to an order and avoid counting replayed sales twice. Assign distinct responsibilities to the key, sequence number, durable write and checkpoint.

What you will learn

Assign distinct responsibilities to the key, sequence number, durable write and checkpoint.

Prerequisite

This exercise uses fictional evidence and makes no changes in a live environment.

Make the write durable before checkpointing

Partition keys control shard placement. Retain a business event ID and avoid treating sequence numbers as one global order.

Where a restarted consumer resumes
Where a restarted consumer resumes1: Extract — Attach key — Send. 2: Send — Read the assigned shard — Process. 3: Process — Commit the result — Resume123ExtractSendProcessResumeWhere a restarted consumer resumes1: Extract — Attach key — Send. 2: Send — Read the assigned shard — Process. 3: Process — Commit the result — Resume123ExtractSendProcessResume
  1. Extract → SendAttach key
  2. Send → ProcessRead the assigned shard
  3. Process → ResumeCommit the result
Extract
Order and event ID
Send
Partition by order ID
Process
Deduplicate and commit
Resume
Checkpoint committed work

A crash after commit

Orders A and B reach separate shards. A17 is committed just before the consumer stops without checkpointing.
EvidenceAfter restartInterpretation
A16Committed and checkpointedResume boundary
A17Read againRecognize the committed ID
A18First readApply and checkpoint
B9Progresses independentlyDo not compare sequence numbers across shards

Keep the durable event ID consistent with the side effect to prevent double counting.

Conditions for replay

Define the ordering scope Route by order ID and specify what happens when a change arrives late.

Commit before checkpointing Keep the result and event ID consistent, then checkpoint completed work.

Exercise restart Stop just after commit and verify both duplicates and first-time events.

Check your decision

Choose an option to see the result and explanation.

What partitions changes to the same order?
Read the answer and reason

Answer: Order ID · The ordering key differs from the event identifier.

How do you handle A17 after replay?
Read the answer and reason

Answer: Recognize the committed event ID · Use the recorded ID as evidence of its earlier commit.

When can you move the checkpoint?
Read the answer and reason

Answer: After the write commits · A checkpoint represents completed work.

Check, then explain in your own words

  1. Define the ordering scope.
  2. Commit the side effect with its ID.
  3. Verify restart behavior.
Check your understanding: Can you order A and B globally by sequence number?

Use a business-defined ordering field for a global order.

A shard sequence and business chronology have different meanings.

Sources and conditions checked

The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.