Topic lessonTopic: AWS data pipelines and SQL Server integration
Kinesis ordering and checkpoints for SQL Server changes
Preserve the order of changes to an order and avoid counting replayed sales twice. Assign distinct responsibilities to the key, sequence number, durable write and checkpoint.
Assign distinct responsibilities to the key, sequence number, durable write and checkpoint.
This exercise uses fictional evidence and makes no changes in a live environment.
Make the write durable before checkpointing
Partition keys control shard placement. Retain a business event ID and avoid treating sequence numbers as one global order.
- Extract → SendAttach key
- Send → ProcessRead the assigned shard
- Process → ResumeCommit the result
- Extract
- Order and event ID
- Send
- Partition by order ID
- Process
- Deduplicate and commit
- Resume
- Checkpoint committed work
A crash after commit
| Evidence | After restart | Interpretation |
|---|---|---|
| A16 | Committed and checkpointed | Resume boundary |
| A17 | Read again | Recognize the committed ID |
| A18 | First read | Apply and checkpoint |
| B9 | Progresses independently | Do not compare sequence numbers across shards |
Keep the durable event ID consistent with the side effect to prevent double counting.
Conditions for replay
Define the ordering scope Route by order ID and specify what happens when a change arrives late.
Commit before checkpointing Keep the result and event ID consistent, then checkpoint completed work.
Exercise restart Stop just after commit and verify both duplicates and first-time events.
Check your decision
Choose an option to see the result and explanation.
Check, then explain in your own words
- Define the ordering scope.
- Commit the side effect with its ID.
- Verify restart behavior.
Check your understanding: Can you order A and B globally by sequence number?
Use a business-defined ordering field for a global order.
A shard sequence and business chronology have different meanings.
Sources and conditions checked
The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.
Scope and conditionsPartition keys, shards and sequence numbers.
Used in this lessonMake the write durable before checkpointing A crash after commit Conditions for replay