'Can I build a blob in Postgres?
I have a scenario where a db has a few hundred million rows, keeping a history of a few weeks only.
In this database are different products (tradable instruments) and each has about 72k rows of data per hour and there is roughly 30 products.
The data is always requested by blocks of 1h, aligned on 1h. For example "I want data for X from 2pm to 3pm.
This data is processed by several tools and the requests are very demanding for the database.
Each tool does its own disk caching, building a binary blob for each hour.
But I was wondering if it would be possible to build these directly in Postgres? Data is indexed by timestamp and is written in a linear fashion as the writes represent live data. So it would be possible to detect with a trigger that we just crossed an hour.
Would it be possible when we detect this to get all this data, build a binary blob out of it and save it in it own table? the data is simply all the columns one after another in binary forms. They're all numbers, no strings, etc so the alignment / format is very simple and rigid.
In practice the rows are like this:
instrument VARCHAR NOT NULL,
ts TIMESTAMP WITHOUT TIME ZONE NOT NULL,
quantity FLOAT8 NOT NULL,
price FLOAT8 NOT NULL,
direction INTEGER NOT NULL
and I would like, at the end of each hour to build a byte array that's like this:
0 4 12 20 24
|ts|quantity|price|direction|ts|quantity|price|direction...
with every row of the hour. Build one blob per instrument and write it in a table like this:
instrument VARCHAR
ts TIMESTAMP
blob BYTEA
My questions are:
- is this possible? or would be it be very inefficient to get 30 (products) * 72k rows each to aggregate and save every hour in an efficient way?
- is there anywhere I could find an example leading me toward this?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
