'Typeorm postgresql migration update ignores some rows
I have a big table named assets, about 2400 rows in postgresql database. I'm trying to update it using typeorm migration. Each asset have only one field - uuid id. In words I want to do the following:
- Add a new column of json type. It should have default value or be nullable, not matter.
- In a loop get all assets in groups of 100 items. I do it so to prevent memory overloading, because table can have more than 2400 rows. Please correct me here if I do it wrong.
- For each of group I make inner cycle and get each of assets one by one to check if it exists. If not exists - I throw an error
- For each of assets I run a query to update asset's new field content with json data.
- Then I again get current asset by id, check if it exists and check it's content was updated properly. If no - I throw an error. Also on this step I print one of json fields in the console.
The problem: the migration doesn't affect about 70% of rows. After that rows contain default value or NULL, depending of new column definition. No errors thrown during all process. I tried different ways of how to do it, firstly I just tried to do a one update query for each of group. At last I split it in a few small steps, but result is still the same.
Help me please with this stuff, I really got stuck. From first, second and next perspectives I do everything correct, only one thing is in doubt: I'm not sure I should do update query in migration.
The migration code below.
Thanks.
export class AssetsAddContentColumnNotNull1647776724189 implements MigrationInterface {
public async up(queryRunner: QueryRunner): Promise<void> {
const date = new Date();
await queryRunner.addColumns('assets', [
new TableColumn({
name: 'content',
type: 'json',
isNullable: true,
default: `'${JSON.stringify([{
body: '',
updatedAt: date
}])}'`
})
]);
const limit = 100;
let offset = 0;
let hasAssets = true;
while (hasAssets) {
const assets = await queryRunner.query(`SELECT id FROM assets LIMIT ${limit} OFFSET ${offset}`);
if (!assets.length) {
hasAssets = false;
} else {
for (let i = 0; i < assets.length; ++i) {
const content = JSON.stringify([{
body: 'New test content',
other_field: 'Other test field',
updatedAt: date
}]);
let assetResults = await queryRunner.query(`SELECT id FROM assets WHERE id='${assets[i].id}'`);
let assetSingle = assetResults[0];
// Here I'm checking if asset exists
if (!assetSingle) {
throw new Error(`Asset not found: ${assets[i].id}`)
}
// Here I'm trying to update
await queryRunner.query(`UPDATE assets SET "content"= '${content}' WHERE id='${assetSingle.id}'`);
// Here I'm selecting current asset again and checking if it exists
assetResults = await queryRunner.query(`SELECT id, content FROM assets WHERE id='${assetSingle.id}'`);
assetSingle = assetResults[0];
if (!assetSingle) {
throw new Error(`Asset not found after insert: ${assetSingle.id}`)
}
// Here I'm checking if asset was updated properly
const insertedContent = assetSingle.content[0];
console.log('insertedContent.other_field', insertedContent.other_field);
if (!insertedContent.other_field) {
throw new Error(`Wrong content in asset: ${assetSingle.id}`);
}
}
offset = offset + limit;
}
}
}
public async down(queryRunner: QueryRunner): Promise<void> {
}
}
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
