How can i convert a dataset which contains 250K rows into a fusion backed dataset in Palantir Foundry?

Question

I am currently in the process of template creation process for a project which needs the datasets contents to be included (The step to go from test to Pre-PROD/PROD). For the contents to be included of a dataset along with template, the dataset needs to be fusion backed. When i created a fusion sheet using the existing dataset, the rows are truncated to 25k rows syncing only these. But the existing dataset contains ~250K rows. I am unable to locate in the documentation; how can i efficiently include all the datasets contents as fusion backed. Any help is appreciated. Thanks

Answer 1

It is to be noted that, without having the dataset fusion backed, the dataset will be empty in PROD

You should not use fusion for this! I would assume that while using a regular dataset you probably forgot to set the dataset to include the contents, you can do that by selecting a dataset and setting includeDatasetContents to true.

{
  "writeSchemaOnUpdate": true,
  "defaultWriteSchemaOnUpdate": false,
  "includeDatasetContents": true, <------ this must be set to true
  "buildDatasetPostDeploy": true
}

In the image below you can find a screenshot of the template creation and updating process of the Foundry Training Resources (the same project we use in the videos on our youtube channel).

As a side thought, not wanting to criticise your design, just sharing some tips to improve it. Using a fusion sheet to hold data for transfer sounds like a very bad idea, and I would recommend you to refactor that. Fusion is a spreadsheet tool that users use manually like any other spreadsheet tool, if the data has 250k rows, it sounds like it can be error prone, hard to detect and fix should someone open it and edit anything incorrectly. There are better tools in foundry for data manipulation at these sizes i.e.: contour, transforms, code workbooks, etc...