Filter problem notifications with alerting profiles

Alerting profiles control the delivery of problem notifications across your organization’s alerting channels by considering predefined filters that are based on problem severity, problem duration, custom events, and tags. Alerting profiles allow you to control exactly which conditions result in problem notifications and which don’t. This includes all problem-push notifications that are sent via the DESK mobile app and displayed in the DESK web UI. Alerting profiles can also be used to set up filtered problem-notification integrations with 3rd party messaging systems like Slack, OpsGenie, and PagerDuty.

Each of your monitoring environments has a default alerting profile that defines the level of severity that must be met before an alert is sent out for a detected problem.

Your environment administrator can create new alerting profiles that provide fine-grained control over the routing of problem alerts for individual teams or for 3rd party problem notification integrations, such as Slack, HipChat, etc.

To view your environment’s alerting profiles, go to Settings > Alerting > Alerting profiles. The list of alerting profiles in the example below includes the Default profile and two customer-defined team-specific profiles (Team Detroit and Team Linz).

Note: The default profile can be modified but not deleted.

Create an alerting profile

Alerting profiles provide a powerful filtering mechanism for problem notifications. By combining filter criteria, you can create custom profiles that, for example, delay notification of problems in development environments while immediately alerting on problems detected in production environments.

To create an alerting profile

1. Go to Settings > Alerting > Alerting profiles.

2. Type a name for the new profile in the Create new alerting profile field.

3. Click the Create button.

4. On the Edit profile page, you can optionally select a management zone for an alerting profile. Through a management zone, you can filter all outgoing problem notifications. Instead of defining additional filtering rules within your alerting profiles, you also have the possibility to select one of your management zones for filtering alerts on all detected problems.

   The default selection within an alerting profile is to use All management zones, which means that no filter is applied to reduce the amount of problem alerts that are sent out. In most cases, you should select your own management zone in order to only push out alerts for problems where the affected component falls within the scope of your teams’ responsibility. Keep in mind that management zones can overlap. If a problem is detected on a service that is defined within multiple management zones, multiple filters will be applied.

5. Click the Create alerting rule button to add a rule to the new profile.

   a. Select a severity level from the Problem severity level drop list.

   b. Type in a number of minutes to define how long a problem duration should last before an alert is sent out. This enables you to avoid alerts for low-severity problems that don’t affect customer experience and therefore don’t require immediate attention.

   c. You can also filter problems further down to specific monitored entities in your environment by selecting an existing tag from the Select tag field. The Filter problems by tag drop list enables you to define a logical operator for the tags you select. Only include entities that have all tags (i.e., AND) enables you to create combined tag rules. Include entities that have at least one tag (i.e., OR) enables you to create either/or rules for tags.

   d. Click Save.

6. Create an event filter based on either specific built-in event types or on string-filtered title or description messages by selecting an option from the Filter problems by any event of source drop list:

   • Predefined
   • Custom Then click Save.

   

   Particularly for auto-remediation use cases, it’s helpful to trigger specific actions based on detailed information that’s captured during abnormal situations. An example is triggering alerts in cases where problems are related to process crashes. The example below shows how to define a process crash filter within an alerting profile.

   

   Also, every event string filter can be inverted if you select the negate option.

   Note:
   Event filters are evaluated with a logical OR. If an event meets the criteria for a single event filter, it will be included in the profile. For this reason, you can only include one event filter with the negate option in an alerting profile. Having two negate filters will allow all events to be shown There is only one negate rule supported when using multiple two negate rules will result in essentially no filter at all.

7. Click Done in the upper right corner.

Alerting profile example

Say you want to receive immediate alerts on availability problems that involve production entities, but you only want to receive alerts on development availability problems that are open longer than 15 minutes. To model this example, you would create two availability filtering rules in your alerting profile, as shown below.

Third-party notification integrations

All third-party problem notification integrations support alerting profiles. The seamless use of alerting profiles for third-party problem integrations allows efficient filtering regardless of the alerting channel your team uses (web UI, DESK mobile app, Slack, HipChat, ServiceNow, etc.).

For example, to check this for Slack

1. Go to Settings > Integration > Problem notifications.
2. Click Set up notifications.
3. Select Slack.
4. You can find on this page, an Alerting profile list box (see image below).