INFRASTRUCTURE LEARNINGINFRASTRUCTURE LEARNING
インフラ大全インフラ大全IT INFRASTRUCTURE / LEARNING
Infrastructure learning guide
Find a lesson or lab/

Esc to close · Search terms stay on your device

LEARNING日本語

Topic lessonTopic: SQL Server recovery and analytics data

AWS Glue Data Catalog and ETL: distinguish schema changes from transformed data

Seeing a column definition does not mean its values were transformed. Check file changes and type mismatches separately in the crawler, catalog and job.

What you will learn

Check file changes and type mismatches separately in the crawler, catalog and job.

Prerequisite

Only fictional order records are used.

Follow files, metadata and transformation outputs

The Data Catalog stores metadata such as table structure and location. Crawlers infer definitions. ETL jobs extract, transform and write data.

From input discovery to verified output
From input discovery to verified output1: Input files — Observe input — Crawler. 2: Crawler — Decide type policy — Catalog. 3: Catalog — Verify transformation — ETL output123Input filesCrawlerCatalogETL outputFrom input discovery to verified output1: Input files — Observe input — Crawler. 2: Crawler — Decide type policy — Catalog. 3: Catalog — Verify transformation — ETL output123Input filesCrawlerCatalogETL output
  1. Input files → CrawlerObserve input
  2. Crawler → CatalogDecide type policy
  3. Catalog → ETL outputVerify transformation
Input files
Order amount is a string
Crawler
Discovers column and type
Catalog
Table definition is available
ETL output
Validated numeric column

Compare evidence before and after transformation

One fictional input includes a numeric-looking 100 and a string value unknown.
ComponentStateInterpretation
Input amountStored as a stringNumeric interpretation needs validation
CrawlerInfers string typeThe contents were not transformed
ETL jobConverts 100 and isolates unknownCheck counts and handling
Analytics definitionUses separate numeric outputDo not mix it with the raw input table

Crawler settings control how catalog definitions change. Values only change through the transformation and its resulting output.

Review structure and values together

Compare locations and definitions. Tables with the same name may differ in paths, columns, types and partitions.

Check automatic update scope. For incremental crawls or configurations that suppress schema updates, decide which tables and files need separate verification.

Report values that cannot be converted. Track counts and reasons. Successful conversions do not establish success for the entire input.

Check the worked example

Choose an option to see the result and explanation.

The crawler finds a string. Has it converted the values to numbers?
Read the answer and reason

Answer: Check stored data and ETL output separately · The crawler detects metadata.

How should a value such as unknown be handled?
Read the answer and reason

Answer: Track isolated values, counts and reasons · Decide how to handle values that cannot be converted.

Two catalog entries have the same table name. What should you compare?
Read the answer and reason

Answer: Location, columns, types and date range · Check the mapping from the table to actual data.

Check, then explain in your own words

  1. Separate input, catalog and output.
  2. Set type and rejected-value handling.
  3. Match output counts and definitions.
Check your understanding: Does a Data Catalog type change transform strings inside the original files?

Verify the transformation and the definition that points to its output separately.

Changing metadata does not rewrite stored values.

Sources and conditions checked

The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.