SMS Alerting from Sensu using playSMS

chef, monitoring, ruby

We’ve been using Multitech iSMS modems for several years to send alerts from our Sensu monitoring system. In late 2015 we noticed that AT&T had announced they were shutting down 2G access so we had to come up with a new way to send out alerts. We don’t have the budget to invest in something like PagerDuty so we built our own solution using playSMS and some Raspberry Pis. The playSMS playnet plugin allows us to cluster 2 or more Pi devices so that a loss of one Pi doesn’t stop alerts from going out.

Basic Design

We have two Raspberry Pi devices running as playSMS slaves (one is a Pi2 and one is a Pi3). Each Pi device has 2 Huawei E353 modems attached to it.

In our main datacenter, we have playSMS installed on a CentOS VM, setup as the playSMS master.

The Pi devices are located in a secondary data center on our campus where they can get a cell signal. They could essentially be placed in any number of remote locations. All they need is an HTTP connection back to the playSMS master.

This design currently meets our needs but could be scaled out to as many Pi devices and SMS modems as needed to support higher message throughputs.

Sensu is configured to send SMS alerts to the playSMS master. The playSMS slaves poll the master for outbound messages and when one is found they send it out.

Deploying playSMS to the Raspberry Pis with Chef

We believe in “Infrastructure as Code” here, so in order to deploy playSMS to our Pis we first needed to install Chef Client. Since there is no omnibus package yet for the Raspberry Pi it takes some manual work to get Chef installed and the node bootstrapped so we can start running cookbooks on it.

  1. Deploy Raspbian to the Pi device
  2. Update packages and install git

    sudo apt-get update && sudo apt-get upgrade && sudo apt-get install git

  3. Install Ruby. I used this bash script Github gist by blacktm to install Ruby 2.3 with rbenv.

    bash <(curl -s https://gist.githubusercontent.com/blacktm/8302741/raw/199fe5d97990dad0be1b89f0b5c7eaa66d69920c/install_ruby_rpi.sh)

    source ~/.bashrc

  4. Install Chef. Once the base chef gem is installed, I have a cookbook recipe that keeps Chef Client updated when I update it in the rest of my environment. I may make a separate blog post about that in the future.

    gem install chef

  5. Bootstrap the Pi device into your Chef environment.

Now that Chef is installed we can deploy playSMS using our recipes/cookbooks.

SMS Modem Setup

Probably the hardest task was getting a USB SMS modem to work with playSMS on the Pis. We tried a couple and found the Huawei E353 worked well for us. There’s a decent walkthrough at HowToForge on installing SMSTools3 and setting up the modems.

One difference is we used modemmanager instead of usb_modeswitch to get the E353 working. I’ve posted a gist of the Chef recipe that sets most of it up for us here.

playSMS and playnet Plugin Setup

There is not much documentation available for the playnet plugin. I eventually found a playSMS forum post, written in Indonesian, that described the setup. I used some Google Translate Fu to figure it out. Take a look at my reply to my original question in the playSMS forum for the setup details in English.

If multiple modems are connected to the Pi device and you want the Pi to send messages out of both modems, login to playSMS and go to “Settings > Route outgoing SMS”. Setup a route for each modem and prefix. playSMS will essentially round robin outgoing SMS messages between the modems.

For example…

UserPrefixSMSCDestination name
*1modem1all
*1modem2all

Sending Sensu Alerts to playSMS

Once the playSMS systems are ready, Sensu can be configured. We wrote a playSMS ruby gem and added a playSMS Sensu handler to the Sensu SMS plugin to facilitate alerting. Usage details can be found in the Sensu SMS handler README, but the gist is…

  1. Create a Sensu user on the playSMS master.
  2. Get the Webservices token from the Sensu user configuration page on the playSMS master and add the hostname, username, and token to your playSMS handler JSON file in Sensu.
  3. Create users and groups in the “phonebook” of the Sensu user account on the playSMS master.
  4. Add playSMS recipients to your Sensu checks. Recipients can be one or more phone numbers, users addressed as @username, or groups addressed as #groupname.

Conclusion

We just converted to the new alerting model with playSMS in mid-December, so we’ll see how it goes, but we expect it will perform better than the old iSMS systems. Also, the changeover has allowed us to eliminate code pointing to a legacy qpage binary running on a 15+ year old Solaris SPARC server. That’s a big win in my book.

I hope this helps someone and am always looking for feedback. Again, if I missed something important or need to clarify something let me know. I’m usually around the Chef Community Slack channel or you can send me an email or message me on Twitter.

2 thoughts on “SMS Alerting from Sensu using playSMS

  1. HI Matt,

    I’ve fllowed your git project sensu-sms-plugin but i can figure out how to send sms to a phone number? what shall i do with FREE-SMS json and play-sms json ?

    Kind Regards,

    Adrian Andrei

Leave a Reply

Your email address will not be published. Required fields are marked *