Creating a New Data Pipeline in Forge AI: Difference between revisions

From FojiSoft Docs
(Creating a new data pipeline for your AI models)
 
No edit summary
 
(One intermediate revision by the same user not shown)
Line 2: Line 2:


=== 1. Access the Data Pipelines Section ===
=== 1. Access the Data Pipelines Section ===
Navigate to the Data Pipelines section in the Forge AI platform under Configuration. You will find the option to create a new data pipeline in the upper right-hand corner of the page.
Navigate to the Configurations section in the Forge AI platform. You will find the option to create a new data pipeline in the upper right-hand corner of the page when viewing the Data Pipelines tab.


=== 2. Initiate a New Data Pipeline ===
=== 2. Initiate a New Data Pipeline ===
Line 13: Line 13:


* '''Action:''' Select from the drop-down menu.
* '''Action:''' Select from the drop-down menu.
* '''Options:''' The list will include various data sources available within your Forge AI environment, such as databases, APIs, and IoT devices.
* '''Options:''' The list will includes the Data Sets available within your FojiSoft environment.


==== '''Destination''' ====
==== '''Destination''' ====


* '''Action:''' Select from the drop-down menu.
* '''Action:''' Select from the drop-down menu.
* '''Options:''' Choose the target location where the data will be sent. This could be a database, data warehouse, or another storage solution supported by Forge AI.
* '''Options:''' Choose the target relational data table where the data will be sent. A new table will automatically be created to match the configuration if you do not explicitly select a table.


==== '''Mode''' ====
==== '''Mode''' ====
Line 63: Line 63:


By following these steps, you can effectively set up and manage data pipelines within the Forge AI platform, ensuring smooth data integration and processing across your systems.
By following these steps, you can effectively set up and manage data pipelines within the Forge AI platform, ensuring smooth data integration and processing across your systems.
[[Category:Monitor]]
[[Category:ForgeAI]]

Latest revision as of 10:04, 10 August 2024

Step-by-Step Guide

1. Access the Data Pipelines Section

Navigate to the Configurations section in the Forge AI platform. You will find the option to create a new data pipeline in the upper right-hand corner of the page when viewing the Data Pipelines tab.

2. Initiate a New Data Pipeline

Click on the "New Data Pipeline" button to start the creation process.

3. Fill Out the Pipeline Parameters

You will be prompted to fill out several parameters for the new data pipeline. Ensure you complete each section as follows:

Source

  • Action: Select from the drop-down menu.
  • Options: The list will includes the Data Sets available within your FojiSoft environment.

Destination

  • Action: Select from the drop-down menu.
  • Options: Choose the target relational data table where the data will be sent. A new table will automatically be created to match the configuration if you do not explicitly select a table.

Mode

  • Options:
    • Replace data in destination table: This option will overwrite the existing data in the destination table.
    • Append data to table: This option will add new data to the existing data in the destination table.

Execution Interval

  • Action: Specify the frequency at which the pipeline should run.
  • Options:
    • Second
    • Minute
    • Hour
    • Day
    • Week
  • Example: Enter a number and select the interval (e.g., 5 minutes).

4. Set Up the Date Range Step

Define the date range for the data to be processed within the pipeline.

From

  • Mode:
    • Absolute: Select a specific start date and time.
    • After Now: Define an offset relative to the current time (e.g., 5 minutes after now).
    • Before Now: Define an offset relative to the current time (e.g., 5 minutes before now).
  • Offset: Enter the number and select the unit (seconds, minutes, hours, days, weeks, years).

To

  • Mode:
    • Absolute: Select a specific end date and time.
    • After Now: Define an offset relative to the current time (e.g., 5 minutes after now).
    • Before Now: Define an offset relative to the current time (e.g., 5 minutes before now).
  • Offset: Enter the number and select the unit (seconds, minutes, hours, days, weeks, years).

5. Finalize and Save the Pipeline

After filling out all the required parameters, review your configurations. If everything is correct, click on the "Save" button to create your new data pipeline.

6. Monitor and Manage the Pipeline

Once the pipeline is created, it will appear in your list of data pipelines. You can monitor its execution, check logs for any issues, and make adjustments as needed.

By following these steps, you can effectively set up and manage data pipelines within the Forge AI platform, ensuring smooth data integration and processing across your systems.