Load Data Set Node in Pathways

From FojiSoft Docs

Overview

The Load Data Set node in Foji Pathways allows workflows to retrieve data from a Foji Data Set over a specified time range. Users can define dynamic start and end times using JavaScript, JavaScript Functions, or Handlebar Templates. The node is essential for automating data ingestion and processing in workflows that require historical or real-time data.

Node Properties

1. Name

  • Description: A unique identifier for the data-loading operation.
  • Usage: Provide a clear, descriptive name for the node (e.g., "Load Patient Claims Data" or "Retrieve Sensor Logs").

2. Data Set

  • Description: Specifies the Foji Data Set from which data will be retrieved.
  • Status: Required (Must be selected for the node to function).
  • Usage: Choose an available dataset relevant to the workflow (e.g., "Medical Billing Records" or "Machine Performance Logs").

3. Start Time

  • Description: Defines the earliest time from which data should be retrieved.
  • Options for Expression Type:
    1. JavaScript Statement: Write a direct JavaScript expression.
    2. JavaScript Function: Define a custom function for complex logic.
    3. Handlebars Template: Use dynamic placeholders to fetch real-time values.
  • Example Expressions:
    • JavaScript Statement: Date.now() - 900000  // Fetch data from 15 minutes ago
    • JavaScript Function: function getStartTime() {   return Date.now() - (24 * 60 * 60 * 1000); // Fetch data from 24 hours ago }
    • Handlebars Template: start_time_variable

4. End Time

  • Description: Defines the latest time up to which data should be retrieved.
  • Options: Same as Start Time (JavaScript Statement, JavaScript Function, Handlebars Template).
  • Example Expressions:
    • JavaScript Statement: Date.now()  // Fetch data up to the current time
    • JavaScript Function: function getEndTime() {   return Date.now(); }

5. Step Interval

  • Description: Determines how frequently data points are sampled.
  • Default Value: 1 Minute
  • Customization: Adjust based on workflow needs (e.g., seconds, minutes, hours, days).
  • Example Use Cases:
    • Real-time monitoring: Set Step = 1 Second
    • Hourly reports: Set Step = 1 Hour
    • Daily summaries: Set Step = 1 Day

6. Variable Name

  • Description: The name of the output variable where the retrieved data will be stored.
  • Status: Required (Must be defined for workflow continuity).
  • Usage: Choose a meaningful variable name (e.g., claims_data, sensor_readings).

How It Works

  1. Select a Data Set: Choose the dataset containing the required information.
  2. Define Start and End Times:
    • Use JavaScript, JavaScript Functions, or Handlebars to set time ranges dynamically.
  3. Set Step Interval: Controls the granularity of retrieved data points.
  4. Store Output in a Variable: The retrieved data is saved under the specified variable name.

Example Use Cases

1. Automating Medical Billing

  • Start Time: Date.now() - 86400000 (24 hours ago)
  • End Time: Date.now()
  • Step Interval: 5 Minutes
  • Variable Name: billing_data
  • Outcome: Loads patient billing records for the past 24 hours in 5-minute intervals.

2. Real-time Machine Performance Monitoring

  • Start Time: Date.now() - 3600000 (1 hour ago)
  • End Time: Date.now()
  • Step Interval: 1 Second
  • Variable Name: sensor_data
  • Outcome: Retrieves machine sensor readings every second for the past hour.

Best Practices

  • Optimize Data Retrieval: Avoid excessive data loading by selecting appropriate time ranges.
  • Choose the Right Expression Type:
    • Use JavaScript for custom logic.
    • Use Handlebars Templates for dynamic values.
  • Use Descriptive Variable Names: This ensures seamless integration with subsequent workflow steps.

The Load Data Set node is essential for integrating historical and real-time data into Pathways, enabling intelligent automation and analysis.

Retrieved from "https://docs.foji.io/index.php?title=Load_Data_Set_Node_in_Pathways&oldid=2723"