Through the Import Data page, you can upload datasets or select datasets from sources that are stored on connected datastores. From the Datasets page, click Import Data.
Figure: Import Data page
To import new data:
NOTE: For file-based sources, Cloud Dataprep by TRIFACTA® expects that each row of data in the import file is terminated with a consistent newline character, including the last one in the file.
- For single files lacking this final newline character, the final record may be dropped.
- For multi-file imports lacking a newline in the final record of a file, this final record may be merged with the first one in the next file and then dropped depending on your running environment.
NOTE: For file-based external datastores, the following limitations apply:
- Only the first 10,000 files can be retrieved.
- The first sample is pulled from a maximum of the first 100 files in the directory. If the size of these 100 files is less than 10MB, the Transformer page indicates that this represents the full dataset.
- When imported, the size computation reported in the Flow View page is over the first 10,000 files.
- Jobs are run across all files in the directory, even if there are more than 10,000 files.
Connect to the source of your data:
Upload: Cloud Dataprep by TRIFACTA® can also load files from your local file system.
Tip: You can drag and drop files from your desktop to to upload them.To change your upload location, click Edit. Navigate to the preferred Google Cloud Storage location.
Google Cloud Storage: Browse or search your Google Cloud Storage files to select content for import. See Google Cloud Storage Browser.
BigQuery: Browse your BigQuery instance for tables to import as datasets.
If you have read permissions for multiple projects, you can read from tables that are part of other projects. See BigQuery Browser.
NOTE: When working with datasets sourced from Avro files, lineage information and the SOURCEROWNUMBER function are not supported.
For more information on the supported input formats, see Supported File Formats.
- Add datasets:
When you have found your source directory or file, click the Plus icon next to its name to add it as a dataset.
Tip: You can import multiple datasets at the same time. See below.
Excel files: Click the Plus icon next to the parent workbook to add all of the worksheets as a single dataset, or you can add individual sheets as individual datasets.
Tip: If you experience issues uploading XLS/XLSX files that are larger than 35MB, you can convert the files to CSV files and then upload them.
See Import Excel Data.
- When a dataset has been selected, the following fields appear on the right side of the screen. Modify as needed:
- Dataset Name: This name appears in the interface.
Dataset Description: You may add an optional description that provides additional detail about the dataset. This information is visible in some areas of the interface.
Tip: Click the Eye icon to inspect the contents of the dataset prior to importing.
You can select a single dataset or multiple datasets for import.
You can modify settings used during import for individual files. In the card for an individual dataset, click Edit Settings.
Per-file encoding: By default, Cloud Dataprep by TRIFACTA attempts to interpret the encoding used in the file. In some cases, the data preview panel may contain garbled data, due to a mismatch in encodings. In the Data Preview dialog, you can select a different encoding for the file. When the correct encoding is selected, the preview displays the data as expected.
- Detect structure: By default, Cloud Dataprep by
TRIFACTA attempts to interpret the structure of your data during import. This structuring attempts to apply an initial tabular structure to the dataset.
- Unless you have specific problems with the initial structure, you should leave the Detect structure setting enabled. Recipes created from these imported datasets automatically include the structuring as the first, hidden steps. These steps are not available for editing, although you can remove them through the Recipe panel. See Recipe Panel.
- When detecting structure is disabled, imported datasets whose schema has not been detected are labeled, raw datasets. When recipes are created for these raw datasets, the structuring datasets are added into the recipe and can be edited as needed.
- For more information, see Initial Parsing Steps.
Column data type inference: You can choose whether or not to apply Cloud Dataprep type inference to your individual dataset.
In the preview panel, you can see the data type that is to be applied after the dataset is imported. This data type may change depending on whether column data type inference is enabled or disabled for the dataset.
To enable Cloud Dataprep type inference, select the Column Data Type Inference checkbox.
Tip: To see the effects of Cloud Dataprep type inference, you can toggle the checkbox and review data type listed at the top of individual columns. To override an individual column's data type, click the data type name and select a new value.
If you have selected a single dataset for import:
- To immediately wrangle it, click Import & Wrangle. The dataset is imported. A recipe is created for it, added to a flow, and loaded in the Transformer page for wrangling. See Transformer Page.
- To import the dataset, click Import. The imported dataset is created. You can add it to a flow and create a recipe for it later. See Datasets Page.
- If you have selected multiple datasets for import:
- To import the selected datasets, click Import Datasets. The imported datasets are created. You can begin working with these imported datasets now or at a later time.
- To import the selected datasets and add them to a flow:
- Click the Add Dataset to a Flow checkbox.
- Click the textbox to see the available flows, or start typing a new name.
- Click Import & Add to Flow.
- The datasets are imported, and the associated recipes are created. These datasets and recipes are added to the selected flow.
- For any dataset that has been added to a flow, you can review and perform actions on it. See Flow View Page.
- If you are not wrangling the datasets immediately, the datasets you just imported are listed at the top of the Datasets page. See Datasets Page.
Import Multiple Datasets
You can import multiple datasets from multiple sources at the same time. In the Import Data page, continue selecting sources from the same or different connections, and additional dataset cards are added to the right panel.
NOTE: If you are importing from multiple files at the same time, the files are not necessarily read in a regular or predictable order. Avoid using functions such as SOURCEROWNUMBER, which relies on original row numbers. See SOURCEROWNUMBER Function.
In the right panel, you can see a preview of each dataset and make changes as needed.
Figure: Import Multiple Datasets
- To remove a dataset from import, click the X in the dataset card.
- To add the datasets to a flow, click the checkbox. Then, select an existing flow or enter the name of a new flow to contain your datasets.
- To import the datasets, click Import or Import & Add to Flow.