Edit Pipeline Data Source Schema
Last updated
Last updated
After creating a pipeline, you can edit the Data Source Schema to define the format of the data to be extracted from the data source or the segment for exporting.
To edit a Pipeline, perform the following steps:
Step 1: From the left navigation panel, select Pipelines.
Step 2: Click the pipeline you want to edit.
Step 3: Click Data Source Schema.
Step 4: Under Schema, the following fields are available:
Schema Fields
Additional Fields
Filters
Step 5: The Schema Fields tab is selected by default.
Step 5.1: Click the Edit button, which is located to the right of the Schema Fields section.
Step 5.2: Scroll down and click Add New Field.
Note: You can add any number of Schema Fields to the Data Source Schema.
Step 5.3: Perform the following actions:
Step 5.4: Key.
If the pipeline type is Segments, select from the Key dropdown menu.
If the pipeline type is Data Source, type a key.
Step 5.5: The String Data Type is the default data type for the Segments pipeline. Select one of the following data types for the Data Source pipleline:
NUMBER - Numeric values
STRING - Text
BOOLEAN - True/False
JSON - Structured data
JSON STRING - JSON stored as a string
Date - Calendar date
TIMESTAMP - Date with time
Step 5.6: Select the Default Value from the following:
Undefined - No value assigned
Null - Explicitly empty value
Custom Value - User-specified default value
Step 5.7: By default, the Optional checkbox is selected. Unselect it to make it a mandatory field.
Step 5.8: Array: When the data type is String, Boolean, JSON, or JSON String, ensure that the Array checkbox is checked.
Step 5.9: Click the Settings icon to add the following validations.
Minimum and maximum length - Defines the allowed length range for the input.
Pattern - Ensures the input matches a specific format using a regex pattern.
Allowed Values - Restricts input to predefined values.
Step 5.10: Click Confirm to save the validations.
Step 5.11: Click the Transformation icon to add transformations to the Data Source Schema.
Type
Function
Select the appropriate one.
Step 5.11.2: Under Type, the following options are available:
Pre-Defined - Uses standard validation rules.
Custom - Allows user-defined validation criteria.
Step 5.11.3: The pre-defined functions are as follows:
Format Data - Formats data into a specific structure.
To Bool - Converts a value to a boolean (true or false)
To Number - Converts a value to a numeric format.
URL Encode - Encodes a URL to ensure safe transmission.
Base64 Encode - Converts data into a Base64-encoded string.
SHA265 Hash - Generates a secure SHA256 hash of the input.
Base64 Decode - Decodes a Base64-encoded string back to its original form.
Step 5.11.4: In the case of Custom Functions, denote the code.
Step 5.11.5: Click Confirm.
Step 5.11.6: Step Result - Data source is created successfully.
Note: You can add any number of transformations in the Schema Fields. To update a transformation, click the settings icon next to it. To delete a transformation, click the delete icon next to the transformation function.
Step 6: Click Additional Fields.
Step 6.1: Enter the appropriate value in the Key field.
Step 6.2: Under Data Type, the following options are available. Select the appropriate one:
NUMBER - Represents numeric values.
STRING - Represents text or alphanumeric characters.
BOOLEAN - Represents true or false values.
DATE - Represents a calendar date without a time component.
Step 6.3: Under the Value field, the following options are available. Select the appropriate one:
Source Field - Uses a value from an existing field.
Function - Applies a predefined function to transform the value.
JSON Path - Extracts data from a JSON structure using a path expression.
URL Param - Retrieves a value from a URL parameter.
Step 6.4: By default, the Optional checkbox is selected. Unselect it to make it a mandatory field.
Step 6.5: Opens the Transformation panel on the right side and click Settings. Under the Type dropdown, two options will be available:
Pre-defined - Uses built-in transformation functions for common data modifications.
Custom - Allows users to define their own transformation logic based on specific requirements.
Select the appropriate one.
Step 6.5.1: The pre-defined functions are as follows:
Format Data - Structures data into a specific format.
To Bool - Converts a value to a boolean (true or false).
To Number - Converts a value to a numeric format.
URL Encode - Encodes a URL to ensure safe transmission.
Base64 Encode - Converts data into a Base64-encoded string.
SHA265 Hash - Generates a secure SHA256 hash of the input.
Base64 Decode - Decodes a Base64-encoded string back to its original form.
Step 6.5.2 In the case of Custom Functions, denote the code.
Step 6.5.3: Click Confirm.
Step Result: Data source is created successfully.
Step 7: Click Filters
Step 7.1: When you check the Enable Filter box, the Function tab opens.
Step 7.2: Under the Type dropdown, the following options are available:
Pre-Defined Fn - Uses built-in filter functions to process data automatically.
Custom Fn - Allows users to define their own filtering logic based on specific conditions
Select the appropriate one.
Follow these steps 4.9.2 and 4.9.3 for Pre-Defined Fn and Custom Fn to proceed further.
Step 5.11.1: Opens the Transformation panel on the right side and click Settings. Under the Info, two options will be available:
