It’s surprisingly easy to setup Zabbix to send notifications to Slack including customizing which notifications go to which Slack channels and what to include on the Slack message. There is official documentation for Zabbix 4.4 available here. The official documentation is very lacking so I’m going to walk through setup start to finish and test notifications.

Slack Setup

The first step is to create an App and retrieve a Slack bot token. Slack’s API works really well but if your workplace has been in use for years it can be frustrating to find the correct page that has the information you need. They seem to have changed their API a few times and it’s not very clear which page you need to go to for which tokens, and what those tokens specifically do. I’ll try to make it clear in these instructions.

Start at https://api.slack.com/apps. You will need to login to your Workplace if you are not logged in already. When logged in, the page should look like this:

Click on Create an App. Name the app Zabbix Alerts or something similar:

It should take you to the Basic Information for your new app:

Click on “Bots” on the bottom left. Now you are directed to App Home (which is not the Bots page):

If you click on the “bot token” link you won’t be directed to the Bot Tokens, you will be brought to the documentation on what a Bot user is. The most important thing to know from that page is:

Bot user token strings begin with xoxb-

Alright so now you are probably starting to see what I meant with Slacks API sites being frustrating.

Make sure you are on the App Home page shown above, then click on the green box that says “Review Scopes to Add”. You should be directed to the OAuth & Permissions page:

Under Bot Token Scopes add an OAuth Scope of “chat:write”:

Now that the Bot Token has a scope, the “Install App in Workplace” button located on the same page (OAuth & Permissions) should be green. Click on it and it should redirect you to a page, then be redirected back to the OAuth & Permissions page, but now you have a Bot User OAuth Access Token:

The last step is to create a test Slack channel and add the app. Open Slack and create a channel named zabbix-test or whatever you want to name it:

Since Slack is so intuitive you have to click the (i) Details button in the top right corner of the channel box, then click More, then click Add apps. From there you can select your Zabbix Alerts app:

When you click Add it will say something about Zabbix Alerts in the main chat area:

That should be all that is required for Slack setup. Everything else is on Zabbix now.

Zabbix Setup

I highly, highly recommend considering updating to Zabbix 5.0+ for the sole reason that it allows for secret macros. The official documentation for Zabbix 4.4 Slack integration allows any user with access to global macros to see the bot token in plain text.

The first step is to navigate to Administration > Media Types and use the Import button in the top right corner to upload this media_slack.xml file:

You will probably run into the problem where you downloaded media_slack.xml for the latest version of Zabbix 5.0 and Zabbix GIT makes you register to view a previous version. Make sure you use a “release” version or it probably won’t work right. Here is a direct download to a version for release Zabbix 4.4 that should work :

If you are getting an error like shown below it means your media_slack.xml file is not the correct version. There is some JavaScript that is slightly tweaked between versions for everything to mesh correctly.

Error when media_slack.xml is wrong version for Zabbix

Once you upload that, there should be a green bar at the top of the screen that says success, then navigate back to the Media types page and select Slack. You should be presented with many fields:

Many fields

The official documentation says to store the token in bot_token but I set it as a global macro:

bot_token is set for {$SLACKTOKEN}

Remember to scroll to the bottom and click Update when you make any updates.

Update those macros now, navigate to Administration > General and in the top right corner a drop down box will take you to Macros:

Set your macros for {$ZABBIX.URL} and {$SLACKTOKEN}:

{$ZABBIX.URL} just tells the media where your Zabbix server is. The notifications sent to Slack will not direct users to the correct URL if you set this to 127.0.0.1. I’m not sure why they recommend that in the official documentation.

Copy your bot token directly from Slack into the macro value. If you are using Zabbix 5.0+ you can set this macro to be secret and it’s much safer.

Now just navigate to Adminstration > Users. You will need to create a user for every channel or direct message that is used. For example, I want to send notifications to #zabbix-test channel, so I create a user specifically for that. If I wanted to send notifications to #zabbix-alerts also, I would have to create a second user. So to start I am creating Alias “Slack_zabbix-test” and it is added to the Group “No access to the frontend”:

Go to the Media tab and add the Type “Slack” and Send to should be your channel name, so for my case it is #zabbix-test:

Leave all the severity levels default, we will define what triggers cause notifications with Actions. Add the Media and add the user:

This sets up all of the routing needed to get notifications to Slack, now we just need to setup triggers to cause notifications. Navigate to Configuration > Actions and in the top right corner click Create action:

Set the Name to “Report to #zabbix-test Slack” or whatever you named your channel. This is where you clearly define what triggers will cause notifications to go to Slack. The easiest way to keep your Zabbix system under control is to use Tags here. I’ll set mine to “Tag value” and have the name be “#slack_zabbix-test” “equals” “1”. This means any tag with the name of #slack_zabbix-test and the value of 1 will be sent to the Slack channel #zabbix-test.

Make sure you click the Add button under the New Condition, otherwise it wont take.

Before this is completed, you need to open Operations and on the Operations section click New. Next to Send to Users click Add and click the Slack_zabbix-test user that was created to communicate with the Slack channel.

Make sure you click the Add button under the Operation details, otherwise it wont take.

Now click Recovery operations and set Operation Details > Operation type to “Notify all involved”:

Double check that you clicked Add on every field. It’s really easy to mess this up… it’s not very intuitive.

Everything should be setup at this point. Just click Add and it should immediately become active. We still need to setup a trigger with the correct tag, so let’s do that and see if it works.

Testing

I’m going to test my system with the Ashly amplifier template that was covered in this post. I went to Configuration > Templates > Template Ashly Amplifier > Discovery rules > Trigger prototypes and selected Channel {#CHANNELINDEX} is too hot. This will be the same for any trigger on Zabbix though.

Click on Tags:

Create a tag “#slack_zabbix-test” or whatever you set your condition for in the Action we created. I set the value for 1 since that is required for the condition to work.

Now just click Update. At this point everything should be working. Let’s force a problem by making the Ashly template macro excessively low. Navigate to Configuration > Templates > Template Ashly Amplifier > Macros. Let’s set {$TEMPERATURE_THRESHOLD} to a value below the current temperature of the amplifier (you can find this in Monitoring > Latest Data). In my case I will do 30:

Click Update then head over to Monitoring > Problems and make sure a Ashly amplifier host is selected. After a minute or two you should see the problems start. If you look under Actions you will see it also sent a notification to Slack.

It look’s like everything is working as expected. Now set the macro back to something reasonable and watch the problems resolve in Zabbix and Slack.

Customization

Everything that is displayed in the Slack messages can be fully customized at Administration > Media types > Slack then select Script.

Many things are customized here including the colors used for the severity levels and even the structure of the message. I am very bad at JavaScript but someone with limited knowledge can do a lot of modifications here to include text from Inventory. I’ll show an example with Building, Region, Opdata, Trigger description, Host, Event Time, Severity, and Event tags.

Start by going back to Administration > Media types > Slack and adding some parameters:

NameValue
building{INVENTORY.LOCATION}
building_address{INVENTORY.SITE.ADDRESS.A}
building_code{INVENTORY.SITE.CITY}
territory{INVENTORY.SITE.STATE}
region{INVENTORY.SITE.COUNTRY}

Also update the Script with the code from custom_slack above the “try { var params” statement at the bottom (don’t modify that):

Now edit the host and go to Inventory. Set it for Manual and fill out the fields Location, Site Address A, Site City, Site State, Site Country:

You can later see this data from the Inventory tab:

Your Slack notifications should have a new structure and pull data from Inventory now:

This provides quite a bit of information in Slack and gives users direct links to the problem and device. Now I will set the macro back to 60 and the triggers should go low and Slack will show the notifications as green.

Conclusion

Any trigger generated in Zabbix can be specifically routed to any Slack channel as a message with details about the host experiencing the trigger. It is not difficult to setup and it provides users with simple notifications that don’t require checking Zabbix.

Comments

  1. Doug

    Thanks for this guide!
    I had issues with actions not triggering at all, It turns out that I also needed to make sure that the Slack user had access to all the hosts via User Groups. Just adding them to “No access to the frontend” wasn’t enough on its own.

  2. Trix

    The problem resolution message and the Slk notification failed : invalid_auth error do not arrive , but the problem messages do. When sending a message about the solution, the Slack notification failed : channel_not_found error appears

    1. Jacob Robinson Article Author

      Interesting error. It sounds like it is attempting to reach another Slack channel. Have you checked the “Recovery operations” tab of the Action and confirmed it is set to the proper Default subject and Default message as well as set to “Notify all involved” with Default message checked?

      1. Trix

        Everything is checked by default, one problem is solved “invalid auth” , but the problem with resolved is not solved, and no messages are received.

  3. Lewis

    Details Media type test failed.
    Slack notification failed : Field “event_id” is not a number

    Seem to be getting this when trying to Test the alert via the Media types page.

  4. User

    Hello, can you change naming used to zabbix-alerts or similar, not zabbix-test as it’s not intuitive to change that easily later on and its not easy to follow with different naming.
    Other than that it’s great tutorial!

    1. Jacob Robinson Article Author

      In “Media” you can change “Send to” to any Slack channel you would like. It just needs the # prefix, so it would be #zabbix-alerts for your case. You can also create multiple Users with different Media settings to send to multiple Slack channels with the same Action. The Zabbix app you create in Slack also needs to be added to all channels you would like Zabbix to send alerts to, unless you give the app access to all channels (set in Slack).

Leave a Reply to Jacob Robinson Cancel reply

Your email address will not be published.