Topic lessonTopic: Monitoring and recovering event processing
Design CloudWatch Logs for investigation, retention, and sensitive data
Decide what evidence investigators need before you emit logs. Use a fictional image-processing system to choose useful fields while controlling sensitive data and log costs.
Distinguish log groups from streams and choose retention, access, and detail based on investigation requirements.
The examples use fictional request identifiers. You do not need real logs or personal information.
Choose what to record and who needs to read it
A log group in CloudWatch Logs shares settings such as retention and access. A log stream is a sequence of events from one source. Limiting what the application emits can keep unnecessary sensitive data out of storage.
- Operation → Useful fieldsSelect the necessary fields
- Useful fields → Storage and accessCheck retention and access
- Storage and access → InvestigationRead the relevant scope
- Operation
- Intake and saved result
- Useful fields
- Time, request ID, and stage
- Storage and access
- Set requirements by group
- Investigation
- Find the request in the relevant period
Select fields based on the investigation task
| Information | Treatment | Reason |
|---|---|---|
| Intake time, request ID img-17, and processing stage | Keep | These fields let an investigator correlate the same request. |
| Image body and authentication header | Omit from general investigation logs | These are unnecessary for finding the failing stage and increase exposure. |
| Error category and fictional code | Keep | The error can be classified without the underlying payload. |
| Query the incident window | Narrow the groups and time range | Read relevant records and limit the data scanned. |
Shorter retention changes how long events are stored. Check ingestion separately when you change emitted log volume, and check scanned data separately when you narrow a query.
Separate masking after ingestion from limiting original output
Handle sensitive content. CloudWatch Logs data protection can mask matches to selected identifiers when ingested events are viewed. It does not apply retroactively to existing events. Authorized viewers may be able to see the original values.
Keep the investigation window available. Consider delayed incident reports and organizational retention requirements. A retention change does not change the volume that was already ingested.
Choose how to record and query the evidence
Choose an option to see the result and explanation.
Check, then explain in your own words
- Choose which fields the investigation needs and which to omit.
- Check retention and access by log group.
- Measure ingestion, storage, and query scanning separately.
Check your understanding: Does adding masking make it appropriate to log authentication credentials?
Omit credentials that the investigation does not need before emitting the event.
Masking has a defined scope, and the stored value still needs to be protected.
Sources and conditions checked
The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.
Scope and conditionsLog groups, streams, and the scope of retention and access settings.
Used in this lessonChoose what to record and who needs to read it Select fields based on the investigation task Separate masking after ingestion from limiting original output
Scope and conditionsMasking scope, limits on historical events, and unmask permission.
Used in this lessonChoose what to record and who needs to read it Separate masking after ingestion from limiting original output
- Amazon CloudWatch PricingChecked:
Scope and conditionsIngestion, storage, and query billing dimensions.
Used in this lessonSelect fields based on the investigation task Separate masking after ingestion from limiting original output