Creating a New Data Pipeline in Forge AI

From FojiSoft Docs

Step-by-Step Guide

1. Access the Data Pipelines Section

Navigate to the Configurations section in the Forge AI platform. You will find the option to create a new data pipeline in the upper right-hand corner of the page when viewing the Data Pipelines tab.

2. Initiate a New Data Pipeline

Click on the "New Data Pipeline" button to start the creation process.

3. Fill Out the Pipeline Parameters

You will be prompted to fill out several parameters for the new data pipeline. Ensure you complete each section as follows:

Source

  • Action: Select from the drop-down menu.
  • Options: The list will includes the Data Sets available within your FojiSoft environment.

Destination

  • Action: Select from the drop-down menu.
  • Options: Choose the target relational data table where the data will be sent. A new table will automatically be created to match the configuration if you do not explicitly select a table.

Mode

  • Options:
    • Replace data in destination table: This option will overwrite the existing data in the destination table.
    • Append data to table: This option will add new data to the existing data in the destination table.

Execution Interval

  • Action: Specify the frequency at which the pipeline should run.
  • Options:
    • Second
    • Minute
    • Hour
    • Day
    • Week
  • Example: Enter a number and select the interval (e.g., 5 minutes).

4. Set Up the Date Range Step

Define the date range for the data to be processed within the pipeline.

From

  • Mode:
    • Absolute: Select a specific start date and time.
    • After Now: Define an offset relative to the current time (e.g., 5 minutes after now).
    • Before Now: Define an offset relative to the current time (e.g., 5 minutes before now).
  • Offset: Enter the number and select the unit (seconds, minutes, hours, days, weeks, years).

To

  • Mode:
    • Absolute: Select a specific end date and time.
    • After Now: Define an offset relative to the current time (e.g., 5 minutes after now).
    • Before Now: Define an offset relative to the current time (e.g., 5 minutes before now).
  • Offset: Enter the number and select the unit (seconds, minutes, hours, days, weeks, years).

5. Finalize and Save the Pipeline

After filling out all the required parameters, review your configurations. If everything is correct, click on the "Save" button to create your new data pipeline.

6. Monitor and Manage the Pipeline

Once the pipeline is created, it will appear in your list of data pipelines. You can monitor its execution, check logs for any issues, and make adjustments as needed.

By following these steps, you can effectively set up and manage data pipelines within the Forge AI platform, ensuring smooth data integration and processing across your systems.

Retrieved from "https://docs.foji.io/index.php?title=Creating_a_New_Data_Pipeline_in_Forge_AI&oldid=1732"