'ADF: Export Parquet Files with Column names with spaces and where clause
I have a lookup that retrieves a few records from a SQL Server table containing server, database, schema, table name and a where clause with values. These values are passed to a copy data (within a ForEach) In the forEach flow I've also insterted a lookup in order to remove spaces and any special characters in the column names, in order to save to parquet file. My issue is that I can't get the "where" clause to work.
VALUES IN TABLE:
| SRC_SERVERNAME | SRC_DATABASE | SRC_SCHEMANAME | SRC_TABLENAME | SRC_WHERE_DATE_CLAUSE |
|---|---|---|---|---|
| SQ01 | NAV | dbo | Company$Sales Invoice Header | [Posting Date] >= '2021-01-01' |
SOURCE SETUP:
ERROR MSG:
A database operation failed with the following error: 'Incorrect syntax near '2021'. Must declare the scalar variable "@wherename".' Incorrect syntax near '2021'. Must declare the scalar variable "@wherename"., SqlErrorNumber=102,Class=15,State=1,
QUERY
@concat('
DECLARE @s VARCHAR(MAX)
DECLARE @servername VARCHAR(500) = ''', item().SRC_SERVERNAME ,'''
DECLARE @databasename VARCHAR(500) = ''', item().SRC_DATABASENAME ,'''
DECLARE @schemaname VARCHAR(500) = ''', item().SRC_SCHEMANAME ,'''
DECLARE @tablename VARCHAR(500) = ''', item().SRC_TABLENAME ,'''
DECLARE @wherename VARCHAR(500) = ''', item().SRC_WHERE_DATE_CLAUSE ,'''
SELECT @s = ISNULL(@s + '', '','''') + ''['' + c.name + '']'' + '' as [''+replace(replace(replace(replace(replace(replace(replace(c.name,'' '',''''),''ö'',''o''), ''ä'',''a''),''å'',''a''),''('',''_''),'')'',''''),''-'',''_'') +''] ''
FROM sys.all_columns c join sys.tables t ON c.object_id = t.object_id
WHERE t.name = @tablename
SELECT ''select '' + @s + '' from '' + ''['' + @servername + '']'' + ''.'' + ''['' + @databasename + '']'' + ''.'' + ''['' + @schemaname + '']'' + ''.'' + ''['' + @tablename + '']'' + '' where '' + @wherename as Query')
Solution 1:[1]
I've solved it, ADF handles characters differently.
Had to update the where clause and add extra characters.
So instead of:
Input
INSERT INTO [dbo].[ADF_DL_ONPREMSQL_CONFIGURATION] ([SRC_SERVERNAME], [SRC_DATABASENAME], [SRC_SCHEMANAME], [SRC_TABLENAME], [SRC_WHERE_DATE_CLAUSE])
VALUES ('SQ01', 'Company', 'dbo', 'Company$Sales Invoice Header', 'where [Posting Date] >= ''2021-01-01''')
Output
where [Posting Date] >= '2021-01-01'
I updated with:
INSERT INTO [dbo].[ADF_DL_ONPREMSQL_CONFIGURATION] ([SRC_SERVERNAME], [SRC_DATABASENAME], [SRC_SCHEMANAME], [SRC_TABLENAME], [SRC_WHERE_DATE_CLAUSE])
VALUES ('SQ01', 'Company', 'dbo', 'Company$Sales Invoice Header', 'where [Posting Date] >= ''''2021-01-01''''')
Output
where [Posting Date] >= ''2021-01-01''
Final Query
@concat('
DECLARE @s VARCHAR(MAX)
DECLARE @servername VARCHAR(500) = ''', item().SRC_SERVERNAME ,'''
DECLARE @databasename VARCHAR(500) = ''', item().SRC_DATABASENAME ,'''
DECLARE @schemaname VARCHAR(500) = ''', item().SRC_SCHEMANAME ,'''
DECLARE @tablename VARCHAR(500) = ''', item().SRC_TABLENAME ,'''
DECLARE @wherename VARCHAR(500) = ''', string(item().SRC_WHERE_DATE_CLAUSE) ,'''
SELECT @s = ISNULL(@s + '', '','''') + ''['' + c.name + '']'' + '' as [''+replace(replace(replace(replace(replace(replace(replace(c.name,'' '',''''),''ö'',''o''), ''ä'',''a''),''å'',''a''),''('',''_''),'')'',''''),''-'',''_'') +''] ''
FROM sys.all_columns c join sys.tables t
ON c.object_id = t.object_id
WHERE t.name = @tablename
SELECT ''select '' + @s + '' from '' + ''['' + @servername + '']'' + ''.'' + ''['' + @databasename + '']'' + ''.'' + ''['' + @schemaname + '']'' + ''.'' + ''['' + @tablename + '']'' + '' '' + @wherename as Query')
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | user3052850 |

