Send Notifications from Zabbix to Slack

It’s surprisingly easy to setup Zabbix to send notifications to Slack including customizing which notifications go to which Slack channels and what to include on the Slack message. There is official documentation for Zabbix 4.4 available here. The official documentation is very lacking so I’m going to walk through setup start to finish and test notifications.

Slack Setup

The first step is to create an App and retrieve a Slack bot token. Slack’s API works really well but if your workplace has been in use for years it can be frustrating to find the correct page that has the information you need. They seem to have changed their API a few times and it’s not very clear which page you need to go to for which tokens, and what those tokens specifically do. I’ll try to make it clear in these instructions.

Start at https://api.slack.com/apps. You will need to login to your Workplace if you are not logged in already. When logged in, the page should look like this:

Click on Create an App. Name the app Zabbix Alerts or something similar:

It should take you to the Basic Information for your new app:

Click on “Bots” on the bottom left. Now you are directed to App Home (which is not the Bots page):

If you click on the “bot token” link you won’t be directed to the Bot Tokens, you will be brought to the documentation on what a Bot user is. The most important thing to know from that page is:

Bot user token strings begin with xoxb-

Alright so now you are probably starting to see what I meant with Slacks API sites being frustrating.

Make sure you are on the App Home page shown above, then click on the green box that says “Review Scopes to Add”. You should be directed to the OAuth & Permissions page:

Under Bot Token Scopes add an OAuth Scope of “chat:write”:

Now that the Bot Token has a scope, the “Install App in Workplace” button located on the same page (OAuth & Permissions) should be green. Click on it and it should redirect you to a page, then be redirected back to the OAuth & Permissions page, but now you have a Bot User OAuth Access Token:

The last step is to create a test Slack channel and add the app. Open Slack and create a channel named zabbix-test or whatever you want to name it:

Since Slack is so intuitive you have to click the (i) Details button in the top right corner of the channel box, then click More, then click Add apps. From there you can select your Zabbix Alerts app:

When you click Add it will say something about Zabbix Alerts in the main chat area:

That should be all that is required for Slack setup. Everything else is on Zabbix now.

Zabbix Setup

I highly, highly recommend considering updating to Zabbix 5.0+ for the sole reason that it allows for secret macros. The official documentation for Zabbix 4.4 Slack integration allows any user with access to global macros to see the bot token in plain text.

The first step is to navigate to Administration > Media Types and use the Import button in the top right corner to upload this media_slack.xml file:

You will probably run into the problem where you downloaded media_slack.xml for the latest version of Zabbix 5.0 and Zabbix GIT makes you register to view a previous version. Make sure you use a “release” version or it probably won’t work right. Here is a direct download to a version for release Zabbix 4.4 that should work :

If you are getting an error like shown below it means your media_slack.xml file is not the correct version. There is some JavaScript that is slightly tweaked between versions for everything to mesh correctly.

Once you upload that, there should be a green bar at the top of the screen that says success, then navigate back to the Media types page and select Slack. You should be presented with many fields:

The official documentation says to store the token in bot_token but I set it as a global macro:

Remember to scroll to the bottom and click Update when you make any updates.

Update those macros now, navigate to Administration > General and in the top right corner a drop down box will take you to Macros:

Set your macros for {$ZABBIX.URL} and {$SLACKTOKEN}:

{$ZABBIX.URL} just tells the media where your Zabbix server is. The notifications sent to Slack will not direct users to the correct URL if you set this to 127.0.0.1. I’m not sure why they recommend that in the official documentation.

Copy your bot token directly from Slack into the macro value. If you are using Zabbix 5.0+ you can set this macro to be secret and it’s much safer.

Now just navigate to Adminstration > Users. You will need to create a user for every channel or direct message that is used. For example, I want to send notifications to #zabbix-test channel, so I create a user specifically for that. If I wanted to send notifications to #zabbix-alerts also, I would have to create a second user. So to start I am creating Alias “Slack_zabbix-test” and it is added to the Group “No access to the frontend”:

Go to the Media tab and add the Type “Slack” and Send to should be your channel name, so for my case it is #zabbix-test:

Leave all the severity levels default, we will define what triggers cause notifications with Actions. Add the Media and add the user:

This sets up all of the routing needed to get notifications to Slack, now we just need to setup triggers to cause notifications. Navigate to Configuration > Actions and in the top right corner click Create action:

Set the Name to “Report to #zabbix-test Slack” or whatever you named your channel. This is where you clearly define what triggers will cause notifications to go to Slack. The easiest way to keep your Zabbix system under control is to use Tags here. I’ll set mine to “Tag value” and have the name be “#slack_zabbix-test” “equals” “1”. This means any tag with the name of #slack_zabbix-test and the value of 1 will be sent to the Slack channel #zabbix-test.

Make sure you click the Add button under the New Condition, otherwise it wont take.

Before this is completed, you need to open Operations and on the Operations section click New. Next to Send to Users click Add and click the Slack_zabbix-test user that was created to communicate with the Slack channel.

Now click Recovery operations and set Operation Details > Operation type to “Notify all involved”:

Double check that you clicked Add on every field. It’s really easy to mess this up… it’s not very intuitive.

Everything should be setup at this point. Just click Add and it should immediately become active. We still need to setup a trigger with the correct tag, so let’s do that and see if it works.

Testing

I’m going to test my system with the Ashly amplifier template that was covered in this post. I went to Configuration > Templates > Template Ashly Amplifier > Discovery rules > Trigger prototypes and selected Channel {#CHANNELINDEX} is too hot. This will be the same for any trigger on Zabbix though.

Click on Tags:

Create a tag “#slack_zabbix-test” or whatever you set your condition for in the Action we created. I set the value for 1 since that is required for the condition to work.

Now just click Update. At this point everything should be working. Let’s force a problem by making the Ashly template macro excessively low. Navigate to Configuration > Templates > Template Ashly Amplifier > Macros. Let’s set {$TEMPERATURE_THRESHOLD} to a value below the current temperature of the amplifier (you can find this in Monitoring > Latest Data). In my case I will do 30:

Click Update then head over to Monitoring > Problems and make sure a Ashly amplifier host is selected. After a minute or two you should see the problems start. If you look under Actions you will see it also sent a notification to Slack.

It look’s like everything is working as expected. Now set the macro back to something reasonable and watch the problems resolve in Zabbix and Slack.

Customization

Everything that is displayed in the Slack messages can be fully customized at Administration > Media types > Slack then select Script.

Many things are customized here including the colors used for the severity levels and even the structure of the message. I am very bad at JavaScript but someone with limited knowledge can do a lot of modifications here to include text from Inventory. I’ll show an example with Building, Region, Opdata, Trigger description, Host, Event Time, Severity, and Event tags.

Start by going back to Administration > Media types > Slack and adding some parameters:

Name	Value
building	{INVENTORY.LOCATION}
building_address	{INVENTORY.SITE.ADDRESS.A}
building_code	{INVENTORY.SITE.CITY}
territory	{INVENTORY.SITE.STATE}
region	{INVENTORY.SITE.COUNTRY}

Also update the Script with the code from custom_slack above the “try { var params” statement at the bottom (don’t modify that):

Now edit the host and go to Inventory. Set it for Manual and fill out the fields Location, Site Address A, Site City, Site State, Site Country:

You can later see this data from the Inventory tab:

Your Slack notifications should have a new structure and pull data from Inventory now:

This provides quite a bit of information in Slack and gives users direct links to the problem and device. Now I will set the macro back to 60 and the triggers should go low and Slack will show the notifications as green.

Conclusion

Any trigger generated in Zabbix can be specifically routed to any Slack channel as a message with details about the host experiencing the trigger. It is not difficult to setup and it provides users with simple notifications that don’t require checking Zabbix.