'Typeorm postgresql migration update ignores some rows

I have a big table named assets, about 2400 rows in postgresql database. I'm trying to update it using typeorm migration. Each asset have only one field - uuid id. In words I want to do the following:

  1. Add a new column of json type. It should have default value or be nullable, not matter.
  2. In a loop get all assets in groups of 100 items. I do it so to prevent memory overloading, because table can have more than 2400 rows. Please correct me here if I do it wrong.
  3. For each of group I make inner cycle and get each of assets one by one to check if it exists. If not exists - I throw an error
  4. For each of assets I run a query to update asset's new field content with json data.
  5. Then I again get current asset by id, check if it exists and check it's content was updated properly. If no - I throw an error. Also on this step I print one of json fields in the console.

The problem: the migration doesn't affect about 70% of rows. After that rows contain default value or NULL, depending of new column definition. No errors thrown during all process. I tried different ways of how to do it, firstly I just tried to do a one update query for each of group. At last I split it in a few small steps, but result is still the same.

Help me please with this stuff, I really got stuck. From first, second and next perspectives I do everything correct, only one thing is in doubt: I'm not sure I should do update query in migration.

The migration code below.

Thanks.

export class AssetsAddContentColumnNotNull1647776724189 implements MigrationInterface {
  
  public async up(queryRunner: QueryRunner): Promise<void> {
    const date = new Date();
    
    await queryRunner.addColumns('assets', [
      new TableColumn({
        name: 'content',
        type: 'json',
        isNullable: true,
        default: `'${JSON.stringify([{
          body: '',
          updatedAt: date
        }])}'`
      })
    ]);
  
    const limit = 100;
    let offset = 0;
    let hasAssets = true;
  
    while (hasAssets) {
      const assets = await queryRunner.query(`SELECT id FROM assets LIMIT ${limit} OFFSET ${offset}`);
    
      if (!assets.length) {
        hasAssets = false;
      } else {
        for (let i = 0; i < assets.length; ++i) {
          const content = JSON.stringify([{
            body: 'New test content',
            other_field: 'Other test field',
            updatedAt: date
          }]);
        
          let assetResults = await queryRunner.query(`SELECT id FROM assets WHERE id='${assets[i].id}'`);
          let assetSingle = assetResults[0];
          
          // Here I'm checking if asset exists
          if (!assetSingle) {
            throw new Error(`Asset not found: ${assets[i].id}`)
          }
  
          // Here I'm trying to update
          await queryRunner.query(`UPDATE assets SET "content"= '${content}' WHERE id='${assetSingle.id}'`);
  
          // Here I'm selecting current asset again and checking if it exists
          assetResults = await queryRunner.query(`SELECT id, content FROM assets WHERE id='${assetSingle.id}'`);
          assetSingle = assetResults[0];
          if (!assetSingle) {
            throw new Error(`Asset not found after insert: ${assetSingle.id}`)
          }
          
          // Here I'm checking if asset was updated properly
          const insertedContent = assetSingle.content[0];
          console.log('insertedContent.other_field', insertedContent.other_field);
          if (!insertedContent.other_field) {
            throw new Error(`Wrong content in asset: ${assetSingle.id}`);
          }
        }
      
        offset = offset + limit;
      }
    }
  }
  
  public async down(queryRunner: QueryRunner): Promise<void> {
  
  }
  
}


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source