'Improve performance of SQL query
I have a slow performing query which I am at a loss on how I could make more efficient. SQL isn't my speciality, but currently this query is taking about 3 seconds on average to run. I've run an execution plan on it in SMMS, and it recommended an index which I created. 3 seconds isn't awful currently, however I only have my own test data in (about 9,000 transactions, 22,000 transfers...). When I have a lot more people using this this will grow massively, and I don't want the execution time for this query to balloon substantially...
The intention of the query, is to return any transfers (and some information on their related transaction), which are of either type 0 or 1, which do not have a price stored for them in token prices (a price stored for anything within 3 hours would be a match, outside of 3 hours isn't a match - its 3 hours either side), where the number of attempts stored in the TokenPriceAttempts table is either 0, or <5 AND most recent is at least a week ago. tl;dr: return transfers, type 0 or 1, which don't have a price yet, and haven't had a recent (within the last week) attempt stored, or have no recent attempts, and have fewer than 5 total attempts.
Here is my query:
SELECT
[t1].[Id],
[t1].[TransactionId],
[t1].[From],
[t1].[To],
[t1].[Value],
[t1].[Type],
[t1].[ContractAddress],
[t1].[TokenId],
[t2].[Hash],
[t2].[Timestamp]
FROM
-- Now, this pre-query is left-joined to token price attempts
-- so ALL Transfers of type < 2 are considered
(
SELECT
[t1].[Id],
coalesce( COUNT([tpa].[Created]), 0 ) [Attempts],
MAX([tpa].[Created]) [MaxCreated]
FROM
[dbo].[Transfers] [t1]
LEFT JOIN [dbo].[TokenPriceAttempts] [tpa]
ON [t1].[Id] = [tpa].[TransferId]
WHERE
[t1].[Type] < 2
GROUP BY
[t1].[Id]
) [PQ]
-- Now, we can just directly join to transfers for the rest
JOIN [dbo].[Transfers] [t1]
ON [PQ].[Id] = [t1].[Id]
-- and the rest from the WITH CTE construct
LEFT JOIN [dbo].[Transactions] [t2]
ON [t1].[TransactionId] = [t2].[Id]
LEFT JOIN [dbo].[TokenPrices] [tp]
ON [t1].[ContractAddress] = [tp].[ContractAddress]
AND [tp].[Timestamp] >= DATEADD(HOUR, - 3, [t2].[Timestamp])
AND [tp].[Timestamp] <= DATEADD(HOUR, 3, [t2].[Timestamp])
WHERE
ABS( DATEDIFF( SECOND, [tp].[Timestamp], [t2].[Timestamp] )) IS NULL
AND
(
[PQ].[Attempts] = 0
OR
(
[PQ].[Attempts] < 5
AND DATEDIFF(DAY, [PQ].[MaxCreated], CURRENT_TIMESTAMP ) >= 7
)
)
And my execution plan (Not sure the best way to post this?):
EDIT 1: There is a paste of the execution plan XML at the bottom.

Finally, each of the referenced tables...
dbo.Transfers
CREATE TABLE [dbo].[Transfers]
(
[Id] INT NOT NULL PRIMARY KEY IDENTITY,
[TransactionId] INT NOT NULL,
[From] NVARCHAR(42) NOT NULL,
[To] NVARCHAR(42) NOT NULL,
[Value] NVARCHAR(256) NOT NULL,
[ValueDecimals] INT NOT NULL,
[Type] TINYINT NOT NULL,
[ContractAddress] NVARCHAR(42) NOT NULL,
[TokenId] NVARCHAR(256) NULL,
CONSTRAINT [FK_Transfers_Transactions] FOREIGN KEY ([TransactionId]) REFERENCES [Transactions]([Id])
)
GO
CREATE INDEX [IX_Transfers_Type] ON [dbo].[Transfers] ([Type])
dbo.TokenPriceAttempts
CREATE TABLE [dbo].[TokenPriceAttempts]
(
[Id] INT NOT NULL PRIMARY KEY IDENTITY,
[TransferId] INT NOT NULL,
[Created] DATETIME2 NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT [FK_TokenPriceAttempts_Transfers] FOREIGN KEY ([TransferId]) REFERENCES [Transfers]([Id])
)
dbo.Transactions
CREATE TABLE [dbo].[Transactions]
(
[Id] INT NOT NULL PRIMARY KEY IDENTITY,
[Hash] NVARCHAR(66) NOT NULL,
[BlockNumber] INT NOT NULL,
[Timestamp] DATETIME2 NOT NULL,
[From] NVARCHAR(42) NOT NULL,
[To] NVARCHAR(42) NULL,
[GasPaid] NVARCHAR(256) NOT NULL,
CONSTRAINT [AK_Transactions_Hash] UNIQUE ([Hash])
)
GO
CREATE INDEX [IX_Transactions_BlockNumber] ON [dbo].[Transactions] ([BlockNumber])
dbo.TokenPrices
CREATE TABLE [dbo].[TokenPrices]
(
[Id] INT NOT NULL PRIMARY KEY IDENTITY,
[ContractAddress] NVARCHAR(42) NOT NULL,
[Timestamp] DATETIME2 NOT NULL,
[PriceEth] DECIMAL(38,18) NOT NULL,
[PriceUsd] DECIMAL(38,18) NOT NULL,
[PriceGbp] DECIMAL(38,18) NOT NULL,
[PriceEur] DECIMAL(38,18) NOT NULL,
[PriceCad] DECIMAL(38,18) NOT NULL,
[PriceAud] DECIMAL(38,18) NOT NULL,
[PriceJpy] DECIMAL(38,18) NOT NULL,
[PriceNzd] DECIMAL(38,18) NOT NULL,
[PriceSgd] DECIMAL(38,18) NOT NULL,
[PriceNok] DECIMAL(38,18) NOT NULL,
[PriceSek] DECIMAL(38,18) NOT NULL,
[PriceZar] DECIMAL(38,18) NOT NULL,
[Created] DATETIME2 NOT NULL DEFAULT CURRENT_TIMESTAMP
)
--CA - CAD - Canada
--US - USD - United States
--AU - AUD - Australia
--JP - JPY - Japan
--NZ - NZD - New Zealand
--SG - SGD - Singapore
--AT - EUR - Austria
--BE - EUR - Belgium
--DE - EUR - Germany
--FI - EUR - Finland
--FR - EUR - France
--GR - EUR - Greece
--IT - EUR - Italy
--NL - EUR - Netherlands
--NO - NOK - Norway
--PT - EUR - Portugal
--ES - EUR - Spain
--CH - SEK - Switzerland
--GB - GBP - United Kingdom
--ZA - ZAR - South Africa
EDIT 1:
Here is the plan XML: https://pastebin.pl/view/d82f6aef
EDIT 2: So I think based on the comments I have gotten a little performance back - Thank you. I am currently looking at the following query:
SELECT
[t1].[Id],
[t1].[TransactionId],
[t1].[From],
[t1].[To],
[t1].[Value],
[t1].[Type],
[t1].[ContractAddress],
[t1].[TokenId],
[t2].[Hash],
[t2].[Timestamp],
[PQ].[Attempts],
[PQ].[MaxCreated]
FROM
-- Now, this pre-query is left-joined to token price attempts
-- so ALL Transfers of type < 2 are considered
(
SELECT
[t1].[Id],
coalesce( COUNT([tpa].[Created]), 0 ) [Attempts],
MAX([tpa].[Created]) [MaxCreated]
FROM
[dbo].[Transfers] [t1]
LEFT JOIN [dbo].[TokenPriceAttempts] [tpa]
ON [t1].[Id] = [tpa].[TransferId]
WHERE
[t1].[Type] < 2
GROUP BY
[t1].[Id]
) [PQ]
-- Now, we can just directly join to transfers for the rest
JOIN [dbo].[Transfers] [t1]
ON [PQ].[Id] = [t1].[Id]
-- and the rest from the WITH CTE construct
LEFT JOIN [dbo].[Transactions] [t2]
ON [t1].[TransactionId] = [t2].[Id]
WHERE
NOT EXISTS
(
SELECT *
FROM [dbo].[TokenPrices] [tp]
WHERE
[t1].[ContractAddress] = [tp].[ContractAddress]
AND [tp].[Timestamp] >= DATEADD(HOUR, - 3, [t2].[Timestamp])
AND [tp].[Timestamp] <= DATEADD(HOUR, 3, [t2].[Timestamp])
)
AND
(
[PQ].[Attempts] = 0
OR
(
[PQ].[Attempts] < 5
AND DATEDIFF(DAY, [PQ].[MaxCreated], CURRENT_TIMESTAMP ) >= 7
)
)
Its taking around a second now. Much better than before, but wonder if there's any more improvements I can make? I have tried looking at all comments and feel like there's more there that I could do, I just am at a loss. Have tried moving the final part of the final WHERE clause up inside the [PQ], but I had to put it as a HAVING clause there and the end result was such that it was a little slower than it currently is.
Also, new execution plan:
Was again unable to successfully upload the XML, for some reason, sorry. So have another paste here: https://pastebin.pl/view/f268d663
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
