Topic lessonTopic: CI/CD design: from checks to release and recovery
AWS deployment rollback: application recovery and database compatibility
Rollback switches to a previous version as a way to recover from a harmful change. Database writes can remain after application rollback. Use two failures of the same ECS application to identify what can be restored, which versions can read the data and how to verify recovery.
Distinguish source history, the deployed version and stored data; explain when to roll back and when a compatible forward fix is needed.
The preceding fictional system has an ECS/Fargate business application, an ALB routing requests and a database shared by old and new versions. A is old; B is new. No live operations or customer data are used.
First separate the states you intend to restore
Reverting a Git change, switching an application and restoring a database are separate operations. Editing source does not change the running version; running the old application does not rewind database rows. Recovery decisions must account for artifacts, configuration, data and user outcomes.
- Git history → Deployable artifactBuild and validate
- Deployable artifact → Running applicationDeploy the identified output
- Running application → Current shared dataRead and write
- Current shared data → User outcomesVerify correct outcomes
- Git history
- A commit reverses a source change.
- Deployable artifact
- Identify the build and configuration.
- Running application
- Switch between A and B.
- Current shared data
- Verify compatibility of existing writes.
- User outcomes
- Check reads, writes and missing records.
R1 and R2: the same old version has different recovery conditions
| Condition | R1: presentation defect only | R2: a new stored value |
|---|---|---|
| States understood by A | new / done | new / done |
| States written by B | Only new / done | Adds archived |
| Supplied test result | A correctly reads every relevant value | A cannot process archived |
| Recovery candidate | Switch to validated A with compatible configuration, then verify outcomes | Contain B’s impact; validate a compatible forward fix or explicit data repair |
| Preserve | Valid orders created while B ran | Valid orders and evidence needed to investigate |
R1 explicitly supplies verified compatibility between A and current data, making A a recovery candidate. Also verify artifact availability and current configuration, connections and permissions. “It worked before” alone does not establish recovery now.
In R2, archived remains in the database after traffic returns to A, so A’s reads do not recover. Contain the affected writes or feature, then evaluate a reader-compatible forward fix or an explicit data repair that preserves meaning. Do not change records before defining the method, authority and validation.
Prepare readers before introducing new stored values
During coexistence, test whether the required readers can process data from both writers. Preparing readers before introducing a new format can preserve more recovery options.
| Stage | Fictional application change | Condition to advance |
|---|---|---|
| Preparation C | Reads new/done/archived but still writes only new/done | Every required reader and retained recovery target supports the values |
| Enablement D | Starts writing archived | Reads and updates work during coexistence and after recovery |
| Later cleanup | Decide when to remove obsolete behavior | Old readers, queued work and recovery targets no longer need it |
Distinguish rollback, a forward fix and database restore
Define the unsuccessful-change plan before deployment, distinguishing known-good recovery from a forward fix under specified conditions. Identify authority, scope, execution conditions and recovery checks.
| Option | What changes | What it does not establish by itself |
|---|---|---|
| Switch application to A | Running application and selected configuration | Whether A reads current data or users have recovered |
| Compatible forward fix | Validate and deploy code that handles current values | The safety of an untested emergency edit |
| Database restore | Create data at a recovery point in a restore destination | Preservation of later valid writes, business cutover and reconciliation |
| Git revert | Add a commit reversing a source change | Automatic changes to a destination or its database |
A reverted deployment still needs recovery verification
- Contain the deployment or affected writes within the agreed scope.
- Record the revision, artifact, configuration, data changes and affected user operations.
- Test current-data compatibility as in R1/R2; confirm recovery method and authority.
- Verify artifact availability and compatible configuration, then execute the agreed method.
- Match the observed revision and destination; validate reads, writes, preserved orders and retries.
- Record when impact ended and the recovery evidence; carry forward unresolved repairs and unknowns.
The ECS deployment circuit breaker can detect failures of applicable rolling deployments and roll back to an eligible COMPLETED deployment. It does not remove archived values from the database. If an old revision is already terminated or removed, an immediate traffic switch may not be available. Check retention and recovery rules for the strategy.
Check which state can safely be restored
Choose an option to see the result and explanation.
Check, then explain in your own words
- Distinguish source history, the running application and shared data.
- Compare reads after switching to A in R1 and R2.
- Explain C/D ordering and recovery targets compatible with current values.
- Carry user impact and recovery evidence into the post-incident lesson.
Check your understanding: R2 switches to a compatible fix C and gets HTTP 200. Is recovery complete?
Not yet. Match the observed version and destination; verify reads including archived, required updates, preserved valid orders and affected user operations.
Successful deployment and recovery of the affected business behavior require separate checks.
Sources and conditions checked
The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.
Scope and conditionsRead/write compatibility during coexistence and after rollback; original author’s article now served through Builder Center. Documentation review, not live AWS/database validation.
Used in this lessonPrepare readers before introducing new stored values
Scope and conditionsKnown-good recovery versus explicit fix-forward conditions, small changes and recovery planning. Documentation review, not live AWS/database validation.
Used in this lessonDistinguish rollback, a forward fix and database restore
Scope and conditionsRolling startup/health failures and prior COMPLETED target; not a guarantee of business correctness or database recovery. Documentation review, not live AWS/database validation.
Used in this lessonA reverted deployment still needs recovery verification
Scope and conditionsSeparate service revisions, test traffic, managed shifting and old-revision cleanup after bake time; resources and permissions are required. Documentation review, not live AWS/database validation.
Used in this lessonA reverted deployment still needs recovery verification