Setting up Alert Rules in FojiSoft

Setting Up Alert Rules in FojiSoft

Overview of FojiSoft Alert Rules

FojiSoft's alert rules are notably versatile, enabling the creation of alerts from various data sources. These rules stand out because they allow for:

- Integration with multiple data sources

- Transformation of data through expressions

- Setting conditions for triggering alerts

- Inclusion of images in notifications

- Multi-dimensional alerting based on a single rule

Editing and Managing Alert Rules

Users with Edit permissions have the exclusive authority to modify or delete FojiSoft-managed alert rules stored within a specific folder. Once deleted from the user interface (UI), alerting resources cannot be retrieved. To prevent loss, it's recommended to utilize file provisioning, Terraform, or the Alerting API for backup and restoration of alerting resources.

Creation Process

Naming the Alert Rule

1. Navigate to `Alerts & IRM -> Alert rules -> + New alert rule`.

2. Assign a descriptive name to identify the rule, which also serves as the `alertname` label in alert instances.

Defining the Query and Condition

1. Select your data source.

2. Choose a fixed relative time range from the options (e.g., last 24 hours).

3. Insert your query, adding additional ones if necessary.

4. Apply expressions to refine alert conditions, using instant vector and built-in functions for time series metrics to avoid extra expressions.

5. Set a recovery threshold if needed, and mark your primary query or expression as the alert condition.

Evaluation Behavior

1. Assign the alert rule to a folder and evaluation group, setting an appropriate interval.

2. Define a pending period, which determines how long a breached condition must persist before triggering an alert.

3. Optionally, pause alert notifications to avoid disruptions during alert tuning.

Annotations and Notifications

Annotations enrich alert notifications with context, such as a summary, description, Runbook URL, and dashboard links.

Notification Policies

Label alert rules to specify which notification policy should handle alerts. The default policy applies unless overridden by matching custom labels.

Multi-dimensional Alerting

To create alerts for individual series, employ expressions like Math, Reduce, or Resample, ensuring the conditions are appropriate for the multi-dimensional context.

Handling No Data and Errors

Configure alerts to handle 'No Data' or 'Error' states appropriately, setting alerts to fire only after the specified evaluation period, thereby minimizing false alarms.

Resolving Stale Alerts

Stale alert instances are auto-resolved when the related series vanish for two intervals, and the system annotates these with the reason.

Creating Alerts from Panels

Lastly, FojiSoft permits alert creation directly from dashboard panels, allowing for the reuse of existing panel queries.