'AWS Glue not picking up schema change
I was under the impression that AWS glue could pick up schema changes (say, I add a column) and automatically update the table. However, after submitting a test message with an extra column to my SQS queue, picked up by the lambda, processed, and then using Firehose to write a parquet file to S3, Glue has not changed the schema. I see the message in the table though, so I am confused why this did not add the column.
Are there any settings I should be aware of? I have already set the glue crawler to "update the table definition in the data catalog" when it detects schema changes in the data store.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
