Monitor Synapse Notebooks from a Microsoft Teams Channel

Working with Spark Notebooks in Azure Synapse Spark can be challenging, especially when it comes to detecting errors in various types of for loops. While you may aim for a successful notebook execution, it’s often necessary to identify and address the functions that failed. One way to efficiently monitor notebook execution without failing the entire notebook or Synapse Pipeline is by sending these errors to a Teams channel. In this blog, we’ll explore how you can set up error monitoring in Azure Synapse Spark using Teams, allowing you to detect errors more effectively.

In my scenario, this approach proves to be useful as my notebook involves processing parquet files into delta tables, with most of the tasks running concurrently. While it’s acceptable for one or two entities to fail during this process, I still want my pipeline to proceed without halting, even if not all entities were processed successfully.

Receiving notifications in a Teams Channel proves to be very beneficial in this case!

Photo by Mikechie Esparagoza

How to create a webhook in MS Teams

To begin with, we need to create an incoming webhook on our desired Teams Channel.

1. Open Microsoft Teams and navigate to the channel where you want to create the incoming webhook.
2. Click on the three dots next to the channel name and select “Connectors” from the dropdown menu.
3. In the Connectors screen, search for “Incoming Webhook” and select it.
4. Give the incoming webhook a name and (optionally) an icon, then click “Create”.
5. You’ll be taken to a screen with a unique URL for the incoming webhook. Copy this URL, as you’ll need it later.
6. Click “Save” to save the settings and activate the incoming webhook.

You can now use the webhook URL to send notifications or messages to the channel.

Microsoft documentation: Create an Incoming Webhook — Teams | Microsoft Learn

Add a Python decorator function

With the code in a decorator function, we ensure that we implement the DRY (Don’t Repeat Yourself) programming principle. We will split the code into two functions.

The first function posts our message on the Teams Channel webhook

The second function is a decorator function. In Python, a decorator is a special function that can be used to modify or extend the behavior of other functions. Decorators are essentially functions that take another function as an argument and then return a modified version of that function.

Sending messages to a Teams channel

To use the decorator we need to apply it to the function, we want to send the errors to our channel.

It is important to raise a ‘ValueError’ when using a ‘try — except’ block in your functions. If not returned by the function, the exception will not be sent to your Teams channel.

Add the decorator to all specific functions that you would like to get notified for! You could even use the ‘send_error_to_teams’ function to write different messages to your Teams channel.

See the result here!

If you want to take this to the next level, take a look at Adaptive Cards: Types of cards — Teams | Microsoft Learn

If you liked this post, please follow me for more Data Engineering stories!

Originally published at http://sidequests.blog on April 23, 2023.