Monitoring Ashly Protea Amplifiers with Zabbix

Ashly makes their own software to operate their Protea amplifiers. For testing, I bought two of them on eBay for much less than MSRP. Since Protea does not provide any sort of alerting, let’s use Zabbix to monitor the amplifiers so we know if a problem ever occurs.

First, let’s take a look at what the Ashly Protea software offers. It’s free to download here. Once you get it installed you will need to add your amplifiers. If your computer is on the same subnet as the amplifiers they will automatically be discovered, otherwise you will need to manually add the amplifier by right clicking “Ashly Network” and click “Add Device”. Then you can add the IP address and it will be added if the device is detected.

In my case the amplifier was automatically detected so I can see it in the software:

If you open the amplifier you can modify settings and see some stats on the device:

We will want to see these settings in Zabbix, so how do we do that? Luckily Ashly provides their IP communication standards available here. Feel free to read through that entire document if you need some help sleeping. The instructions I provide will keep this whole process easy.

Prep Work

There is a little bit of prep work necessary for these scripts to run. If you do not have Python3 installed yet, go ahead and start doing that:

Install Python3 :

yum install python3

Install CentOS Development Tools:

yum -y install gcc

Install Python3 Development Tools:

yum python3-devel

Install setproctitle:

pip3 install setproctitle

That should be all that is necessary for these scripts to run.

Externalscripts Folder

First you will need to get the Python and bash scripts. I have them available on my Github here. These will need to be placed in the Zabbix externalscripts folder which by default is located at /usr/lib/zabbix/. The easiest way to load these files is with an SCP client like WinSCP which is usually enabled on most Linux systems that will be running your Zabbix server. Connect to the server and try to copy the files over:

This error will most likely occur so you will need to adjust permissions on the externalscripts folder on the Zabbix server. Start by checking what the current permissions are:

Since root owns these files they can write to them, so you can connect via WinSCP using the root account and add the files, or you could assign the folder to a group that the user is in, or you could allow others to write. Either method works since this folder is not critical to anything besides Zabbix external scripts. (I’m going to hear about this from someone)

This is what your folder should look like (_pycache_ will create on its own):

Everything should be loaded on the Zabbix server but there still will be some permission issues. If you left it as-is you would end up with this error in Zabbix:

The reason this failed is that the scripts are not executable by anyone based on the current permissions. Let’s try to fix that, run this from /usr/lib/zabbix:

Let’s take care of our global MACROs now so that we don’t have to worry about that later. Navigate to Administration > General then on the right side of the screen there is a drop down box. Select Macros from there:

Make sure you have the following macros setup. You can modify the values for whatever you would like:

Macro	Value
{$HISTORY_STORAGE_PERIOD}	90d
{$NORMAL_UPDATE_INTERVAL}	1m
{$SNMP_COMMUNITY}	public
{$TREND_STORAGE_PERIOD}	365d
{$NORMAL_DISCOVERY_INTERVAL}	3600
{$NORMAL_KEEP_LOST_TIME}	30d

Note that Zabbix 5.0+ has secret macros now, so its much safer to store SNMP communities and passwords to be used.

Now we just need to create a template and tell it what to do with the scripts.

Template Configuration

Start by creating the template (name it anything you want):

Link “Template ICMP Ping” so ping is also monitored. Create some MACROs with this template that we will use later:

{$CURRENT_THRESHOLD} = 40
{$TEMPERATURE_THRESHOLD} = 60

Macros for Zabbix work top-down, as in global macros are automatically applied but the same macro set on the template level will override the global macro for all hosts on that template. Macros can also be set on the host level which will override the template macro for that host.

Add the template, open the template you just created, click on “Items” and add an item. This will be the most important item as it will connect with the external script and retrieve all information. Once this is created we will make other items that are dependent on this item.

You may notice that I’m using {$NORMAL_UPDATE_INTERVAL} for the Update interval and {$HISTORY_STORAGE_PERIOD} for the History storage period. This allows you to create global MACROs that update all hosts with one input. This is very convenient if you ever want to modify the values, you can even create multiple MACROs so you can assign tiers to templates.

Remember to add a Description so that the user has more information about the trigger. This information can also be delivered to external systems, such as the Slack notification system covered here.

What is going on?

Now we can create the items that actually define what is being monitored. The “Ashly device” item retrieves all of the information that is collected on the amplifier. HOST.CONN is the IP address of the host to be monitored, it is being sent as a variable in the Python scripts, which is executed and the results are brought back to Zabbix. The items we create now are parsing the data to get specific information.

So let’s create a few items of specifics to monitor. All items listed below will have a Type of Dependent Item. Use the Key listed below to populate the item. All of these items have a “Master item” of the item we have already created (Ashly device in this case) and the Type of Information is Numberic(unsigned) except for OPT_DEVICE_NAME.

Name	Key	Description
Channel protect	opt_amp_meter_protect_0	Detects if a channel goes into protect mode
Channel thermal	opt_amp_meter_protect_1	Detects if a channel is running hot
Fuse protect	opt_amp_meter_protect_2	Detects if a fuse is blown
OPT_DEVICE_NAME	opt_device_name	Retrieves the device name
OPT_POWER_STATUS	opt_power_status	Retrieves the device power status
OPT_STANDBY	opt_standby	Detects if the device is in standby
Rail Fault	opt_amp_meter_protect_3	Detects if a channel has a rail fault
Supply Fault	opt_amp_meter_protect_4	Detects if a channel has a power supply fault

Preprocessing

Some preprocessing is necessary since the “Ashly device” will be returning a chunk of data at once. This will just parse the data so it can be defined clearly. Follow the table below to configure:

All preprocessing step names will be “JSONPath”:

Name	Parameter
Channel protect	$.OPT_AMP_METER_PROTECT_0
Channel thermal	$.OPT_AMP_METER_PROTECT_1
Fuse protect	$.OPT_AMP_METER_PROTECT_2
OPT_DEVICE_NAME	$.OPT_DEVICE_NAME
OPT_POWER_STATUS	$.OPT_POWER_STATUS
OPT_STANDBY	$.OPT_STANDBY
Rail Fault	$.OPT_AMP_METER_PROTECT_3
Supply Fault	$.OPT_AMP_METER_PROTECT_4

Mass Update

So I messed up and forgot to add the Global MACRO settings to the items. Instead of manually updating each item individually, let’s take advantage of the Mass update feature within Zabbix. Start by checking the items to be updated:

Now click Mass update and select the items to be updated. I updated “History storage period” to Storage period > {$HISTORY_STORAGE_PERIOD} and “Trend storage period” to Storage period > {$TREND_STORAGE_PERIOD}

Select Update and now the changes will be applied to those items:

Testing 1-2-3

Create a host for an amplifier to test the template as it is currently setup (make sure you apply the Ashly template we just created):

Now go to Monitoring > Latest Data and select the newly created host. It may take a minute or two but you should start to see data populate.

If you don’t see anything populating here then go back to Configuration > Hosts and select the host you created then go to Items and on the right side of the table there should be an error you can hover over to get details.

Discovery

Up to this point we are monitoring the basic stats of the amplifier, but we are not monitoring information per channel. Ashly makes a few different models ranging from 2 to 8 channels (from what I know). It would be more work to setup a template for each type, so what if we could build the system that automatically detects how many channels exist and monitor them? Luckily Zabbix has this feature, and it’s not that difficult to implement.

Go back to the template and click Discovery rules:

Now in the top right corner click Create discovery rule. Fill in the forms with the following:

Item	Value
Name	Channel_discovery
Type	External check
Key	ashlydiscovery[“{HOST.CONN}”]
Update Interval	{$NORMAL_DISCOVERY_INTERVAL}
Keep lost resources period	{$NORMAL_KEEP_LOST_TIME}

This will discover how many channels exist but we still need to create some items to go along with it. Click Item prototypes and Create item prototype to create the first item. This is a similar setup to how the standard items are polled, there will be one external check and other items are dependent.

Field	Value
Name	Channel {#CHANNELINDEX}
Type	External Check
Key	ashlychannel[“{HOST.CONN}”,”{#CHANNELINDEX}”]
Type of Information	Character
Update Interval	{$NORMAL_UPDATE_INTERVAL}
New application prototype	Channel {#CHANNELINDEX}

You might notice we are actually using applications now, specifically to divide the information up per channel. Applications can be customized however you would like and I don’t want to pretend to understand what data is important to you but for this application it makes sense to split up the channel data.

Below is a list of the dependent items. Remember to set these for:
Type – Dependent item
History storage period – {$HISTORY_STORAGE_PERIOD}
Trend storage period – {$TREND_STORAGE_PERIOD}
Application prototype – Channel (#CHANNELINDEX)

Name	Key	Type of information	Preprocessing Name	Preprocesing Parameters
OPT_CHANNEL_PROTECT channel {#CHANNELINDEX}	opt_channel_protect[{#CHANNELINDEX}]	Numeric (unsigned)	JSONPath	$.OPT_CHANNEL_PROTECT
OPT_METER_CURRENT channel {#CHANNELINDEX}	opt_meter_current[{#CHANNELINDEX}]	Numeric (unsigned)	JSONPath	$.OPT_METER_CURRENT
OPT_METER_OUTPUT channel {#CHANNELINDEX}	opt_meter_output[{#CHANNELINDEX}]	Numeric (unsigned)	JSONPath	$.OPT_METER_OUTPUT
OPT_METER_TEMP channel {#CHANNELINDEX}	opt_meter_temp[{#CHANNELINDEX}]	Numeric (unsigned)	JSONPath	$.OPT_METER_TEMP
OPT_MUTE_INPUT channel {#CHANNELINDEX}	opt_mute_input[{#CHANNELINDEX}]	Numeric (unsigned)	JSONPath	$.OPT_MUTE_INPUT
OPT_MUTE_OUTPUT channel {#CHANNELINDEX}	opt_mute_output[{#CHANNELINDEX}]	Numeric (unsigned)	JSONPath	$.OPT_MUTE_OUTPUT
Temperature channel {#CHANNELINDEX}	temperature[{#CHANNELINDEX}]	Numeric (float)	See below	See below

Temperature channel is a little weird. Set the preprocessing for:
1. JSONPath – $.OPT_METER_TEMP
2. JavaScript –

value=(875-value)*0.0928
return value

Everything should be all set now. Your Items prototype should look similar to this:

Now navigate over to Monitoring > Lastest Data and select the test amplifier. {$NORMAL_DISCOVERY_INTERVAL} is set for 3600 but you can lower that macro temporarily to make it discover the channels more frequently. It is better to leave it a higher interval rate to reduce processing required, but it is safe to leave it low if you have resources.

It looks like it is working on the test amp for my system. I can see channels 0 through 7 which are all 8 channels of the amp:

Let’s compare that with what Ashly’s software says:

It looks like a very close match for temperature. Note that no formula was provided by Ashly so I had to make that one up on my own. It seems to always be within 1 degree C which is perfectly reasonable for this application.

Triggers

Now that all of the data is being collected, triggers can be created. Since we are using Discovery rules for individual channel data, we will need to create the triggers for the individual channels there. First, lets create some basic triggers for the main items. Head over to the template you’ve created for the Ashly amplifiers and click Triggers:

You can see I already have 3 triggers from the ICMP template that is currently linked. Let’s create 3 more…

First let’s do 2 triggers that are really simple, then we will do one with a dependency. In the top right click Create trigger. Let’s name the first one “Amplifier has major failure”. Make the severity whatever you would like, I am going to use Average.

Now we will define what makes the trigger happen, on the Expression field click the Add block to the right. Click Select and the name of whatever the item you named for key opt_amp_meter_protect_3. The default function of last() should be used, leave Last of (T) and Time shift blank, and set the Result to <> 0. Now you should have the first part of the expression entered. Click on the field and add ” or ” to the end of the expression, then do the Add process again with the item you named for opt_amp_meter_protect_4. The final function should read similar to:

{Template Ashly Amplifier:opt_amp_meter_protect_4.last()}<>0 or {Template Ashly Amplifier:opt_amp_meter_protect_3.last()}<>0

Now just Add the trigger and it will immediately start working. Let’s do the next one for Amplifier is in protect mode. Follow the instructions above except for this expression use opt_amp_meter_protect_0 and opt_amp_meter_protect_2. The expression should be similar to:

{Template Ashly Amplifier:opt_amp_meter_protect_0.last()}<>0 or {Template Ashly Amplifier:opt_amp_meter_protect_2.last()}<>0

Add this trigger and now we will do one with a dependency. You will need to make sure the Template Module ICMP Ping is linked to the Ashly template to do this.

Create a trigger and name it “Amplifier is shut off”. Set the severity for Warning. Use the expression creator to make opt_power_status = 0. It should look like:

{Template Ashly Amplifier:opt_power_status.last()}=0

Now click on Dependencies and click Add. Select “Unavailable by ICMP ping”. Add the trigger. It should look like this now:

This will make it so that you do not get a trigger of Amplifier is shut off just because the device is unreachable and the result is 0.

Discovery Triggers

Triggers that are specific to the channel of the amplifier can also be created. These will need to be created within the Discovery rules of the template, so navigate there and click Trigger prototypes. Note that you will need the macros for {$CURRENT_THRESHOLD} and {$TEMPERATURE_THRESHOLD} as we created before.

Let’s make 3 basic triggers: channel current is too high, channel is in protect mode, and channel is too hot. We will design these so that the trigger tells you specifically which channel is having the trigger occur.

Create the first trigger prototype and name it “Channel {#CHANNELINDEX} current is too high”. Set the severity to Average. Click Add next to the expression field and click Select prototype. Select the item with key opt_meter_current[{#CHANNELINDEX}] and set it to >= {$CURRENT_THRESHOLD}. The result should be similar to:

{Template Ashly Amplifier:opt_meter_current[{#CHANNELINDEX}].last()}>={$CURRENT_THRESHOLD}

Add this trigger. Now create a new trigger prototype named “Channel {#CHANNELINDEX} is in protect mode”. Set the severity for Average and click Add > Select prototype on the expression field then select the item for key opt_channel_protect[{#CHANNELINDEX}]. Set the Result for <> 0 and add the expression. The expression should be similar to:

{Template Ashly Amplifier:opt_channel_protect[{#CHANNELINDEX}].last()}<>0

Now for the last trigger. Create a trigger prototype named “Channel {#CHANNELINDEX} is too hot”. Set the severity for Average and click Add > Select prototype on the expression field then select the item for key temperature[{#CHANNELINDEX}]. Set the Result for >= {$TEMPERATURE_THRESHOLD}. The expression should be similar to:

{Template Ashly Amplifier:temperature[{#CHANNELINDEX}].last()}>={$TEMPERATURE_THRESHOLD}

Add this trigger and everything should be set.

Final Test

As a final test, go back to your template macros and set the thresholds to a very low number such as 1. This should test the current and temperature thresholds.

Go to Monitoring > Problems and select the Ashly host to see if it worked. It will take 3 polling intervals (or whatever you set the custom last value to) for the problem to be shown. They should all pop up within a few minutes:

Success. The amplifiers are now being monitored and have triggers set up. Set your macros back and the problems will go away on their own.

You can add more details to these triggers such as tags and set them up to notify you via E-mail, SMS, Slack, or many other messaging services whenever they go off. I’ll make another post on how to do that.