Topic lessonTopic: SQL Server recovery and analytics data
AWS Glue Data Catalog and ETL: distinguish schema changes from transformed data
Seeing a column definition does not mean its values were transformed. Check file changes and type mismatches separately in the crawler, catalog and job.
Check file changes and type mismatches separately in the crawler, catalog and job.
Only fictional order records are used.
Follow files, metadata and transformation outputs
The Data Catalog stores metadata such as table structure and location. Crawlers infer definitions. ETL jobs extract, transform and write data.
- Input files → CrawlerObserve input
- Crawler → CatalogDecide type policy
- Catalog → ETL outputVerify transformation
- Input files
- Order amount is a string
- Crawler
- Discovers column and type
- Catalog
- Table definition is available
- ETL output
- Validated numeric column
Compare evidence before and after transformation
| Component | State | Interpretation |
|---|---|---|
| Input amount | Stored as a string | Numeric interpretation needs validation |
| Crawler | Infers string type | The contents were not transformed |
| ETL job | Converts 100 and isolates unknown | Check counts and handling |
| Analytics definition | Uses separate numeric output | Do not mix it with the raw input table |
Crawler settings control how catalog definitions change. Values only change through the transformation and its resulting output.
Review structure and values together
Compare locations and definitions. Tables with the same name may differ in paths, columns, types and partitions.
Check automatic update scope. For incremental crawls or configurations that suppress schema updates, decide which tables and files need separate verification.
Report values that cannot be converted. Track counts and reasons. Successful conversions do not establish success for the entire input.
Check the worked example
Choose an option to see the result and explanation.
Check, then explain in your own words
- Separate input, catalog and output.
- Set type and rejected-value handling.
- Match output counts and definitions.
Check your understanding: Does a Data Catalog type change transform strings inside the original files?
Verify the transformation and the definition that points to its output separately.
Changing metadata does not rewrite stored values.
Sources and conditions checked
The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.
- AWS Glue conceptsChecked:
Scope and conditionsThe catalog, crawler and ETL job responsibilities.
Used in this lessonFollow files, metadata and transformation outputs Compare evidence before and after transformation Review structure and values together
- Customizing crawler behaviorChecked:
Scope and conditionsSchema change behavior and incremental crawling.
Used in this lessonFollow files, metadata and transformation outputs Review structure and values together