Creating a New Data Pipeline in Forge AI

1. Access the Data Pipelines Section

Navigate to the Data Pipelines section in the Forge AI platform under Configuration. You will find the option to create a new data pipeline in the upper right-hand corner of the page.

2. Initiate a New Data Pipeline

Click on the "New Data Pipeline" button to start the creation process.

3. Fill Out the Pipeline Parameters

You will be prompted to fill out several parameters for the new data pipeline. Ensure you complete each section as follows:

Source

• Action: Select from the drop-down menu.
• Options: The list will include various data sources available within your Forge AI environment, such as databases, APIs, and IoT devices.

Destination

• Action: Select from the drop-down menu.
• Options: Choose the target location where the data will be sent. This could be a database, data warehouse, or another storage solution supported by Forge AI.

Mode

• Options:
  • Replace data in destination table: This option will overwrite the existing data in the destination table.
  • Append data to table: This option will add new data to the existing data in the destination table.

Execution Interval

• Action: Specify the frequency at which the pipeline should run.
• Options:
  • Second
  • Minute
  • Hour
  • Day
  • Week
• Example: Enter a number and select the interval (e.g., 5 minutes).

4. Set Up the Date Range Step

Define the date range for the data to be processed within the pipeline.

From

• Mode:
  • Absolute: Select a specific start date and time.
  • After Now: Define an offset relative to the current time (e.g., 5 minutes after now).
  • Before Now: Define an offset relative to the current time (e.g., 5 minutes before now).
• Offset: Enter the number and select the unit (seconds, minutes, hours, days, weeks, years).

To

• Mode:
  • Absolute: Select a specific end date and time.
  • After Now: Define an offset relative to the current time (e.g., 5 minutes after now).
  • Before Now: Define an offset relative to the current time (e.g., 5 minutes before now).
• Offset: Enter the number and select the unit (seconds, minutes, hours, days, weeks, years).

5. Finalize and Save the Pipeline

After filling out all the required parameters, review your configurations. If everything is correct, click on the "Save" button to create your new data pipeline.

6. Monitor and Manage the Pipeline

Once the pipeline is created, it will appear in your list of data pipelines. You can monitor its execution, check logs for any issues, and make adjustments as needed.

By following these steps, you can effectively set up and manage data pipelines within the Forge AI platform, ensuring smooth data integration and processing across your systems.