| Home > My Projects > Home Automation > Protocols and Standards |
The home automation world is full of thousands of standards. And I'm automatically discounting nearly all of them because they are not industry standard, widely used and open protocols. Which unfortunately is yet another reason why most commercial "smart" products are not usable for me (with some exceptions). I have thus standardized on only a few protocols, trying to limit all my devices to those. They are primarily MQTT and SNMP. Any device that communicates on an open and documented HTTP API is also acceptable.
Using open protocols means that in 30 years, my devices will be just as accessible as they are today regardless of the state of the manufacturer, or even if that manufacturer exists. It also does not limit me to using only compatible control systems or methods since virtually every tool can handle these protocols, from command line scripts to full blown automation platforms like Home Assistant. Limiting protocols also brings a certain stability as the control system does not need to juggle multiple integrations. Though Home Assistant supports thousands of different integrations, the true standards remain stable and always working. This is unlike, for example, the Tuya "standard" used by many Chinese manufacture devices. Home Assistant supports Tuya of course but that isn't to say the manufacturer won't decide to change the protocol and break the integration. Which has happend. Multiple times.
You may be familiar with some or all of these protocols. It wasnt my aim here to provide a detailed description of how they work down to the network traffic level. Just enough information for a basic understanding.
MQTT |
MQTT is the primary protocol I use to control devices within my automation system.
If you, for example, Google up "Arduino Ethernet Relay Control" it is almost certain that within the first few hits you will find a reference to MQTT. MQTT expands to Message Queuing Telemetry Transport which is a verbose way of saying that MQTT is a lightweight messaging protocol designed for low power, low bandwidth devices. It is an easy to understand, reliable and simple protocol perfectly suited to implementation in microcontrollers for transmission over a network. With MQTT, all devices communicate with a designated "broker" system that is responsible for receiving messages, managing them, and sending them out to the devices which have subscribed to (elected to receive) them. The broker can be instructed to retain certain messages, so even if the devices which have subscribed to the message don't have their connection when it is sent, they still receive it when they reconnect. Overall, a robust system.
MQTT handles messages in a hierarchical topic system that is separated with forward slashes. For example a message may be sent in the topic of "shop/temperature" . This designates the topic is "shop", with a subtopic of "temperature". A temperature reading coming from the shop building. The message data itself is the "payload". In this case, the payload would be a temperature reading, say 26, to represent 26 Celsius. Any device which has subscribed to "shop/temperature" by informing the broker will receive the message as the broker will push it out. Optionally the message can be set "retained" which simply means the broker keeps track of the payload and if a device with a subscription to that topic connects, the last payload is pushed out. MQTT allows subscriptions with wildcards, so subscribing to "shop/#" would instruct the broker to send you every message under the topic of "shop/". And subscribing to "#" will give you a subscription to every topic...potentially a lot of messages!
MQTT is extremely easy to implement in microcontrollers using the Arduino framework thanks to Nick O'Leary's excellent library PubSubClient. PubSubClient allows you to subscribe to and publish messages via MQTT over a network connection (Ethernet, WiFi, any connection supporting the Arduino's Ethernet stream class) on microcontrollers such as Atmega (classical Arduino), ESP8266/ESP32, STM32 and others.
Many commercial devices support MQTT as well if the manufacturer is committed to providing a solution, not a subscription.
There are many MQTT brokers available with plenty of open source options. As the protocol itself is lightweight, the broker has very low requirements and can easily run on a RasberryPi or equivalent. Hardware requirements primarily depend on the traffic the broker has to handle; the number of messages it is asked to pass and the number of connections it must maintain.
My installation uses the Eclipse Mosquitto MQTT broker running on the same host as Home Assistant. Under most Linux distributions it is a single apt-get install and will work without any further configuration. I believe the only configuration settings I have changed from the default are logging location and level.
The MQTT Explorer client is a necessary tool if you are working with MQTT. It allows you to subscribe to and observe topics/messages in a hierarchical tree view with message histories, graphing, various formatted and decoded views, etc.
SNMP |
Anyone who has worked in IT is undoubtabley familiar with SNMP, or Simple Network Management Protocol. SNMP is a near universal protocol originally designed for management of network devices yet is also ideally suited for automation control.
SNMP has an "agent" (the device) and a "manager" (the system requesting data from the device) architecture. The manager either requests data from the device via a "GET", or receives data sent by the device via a "trap". Data is exchanged in SNMP represented by Object IDs (OIDs) which are a hierarchical list of dotted integers. For example, the OID .1.3.6.1.2.1.1.1.1 "sysDescr", which is simply a text description of the device. The OID .1.3.6.1.2.1.1.3 returns sysUpTime which as you might be able to guess, is the uptime of the system (incidentally represented as hundredths of a second since the system was powered on).
SNMP capable systems will typically implement a set of standard OIDs that provide known query points for system information (like CPU use, network information, device name and location, etc.).
A collection of OIDs is referred to as a "MIB", or Management Information Block. It is a list of OIDs, their meaning, and the values or value ranges they will provide. A device will typically provide OIDs as defined in a MIB. For example, if you have a printer, it is likely going to provide SNMP OIDs as defined in the standard MIBs of HOST-RESOURCES for basic host resources (name, location, etc.), Printer-MIB for OIDs relating to printer functions (toner level, jam status, media information) and RFC12130-MIB for OIDs relating to interfaces and TCP/IP (number of network interfaces, IP address info, network errors, etc.). Thus you can be fairly certain you know what OIDs to query for the information you want to obtain. Additionally manufacturers may provide their own OIDs for information specific to their device, providing their own MIBs so you know how to query it.
A "TRAP" is a communication sent from an SNMP enabled device (agent) to a system listening (the manager). Traps are typically unsolicited. The device has something it needs to report, and will thus send a trap without waiting for query for that info from the manager. For example, a SNMP enabled temperature sensor has an alert threshold, so it sends a trap with the OID and value of the sensor reading to the manager when that threshold is reached.
SNMP supports "SET" through which the manager contacts the agent then sends an OID and a value it wishes to set it to. Thus, the device itself can be controlled.
OIDs are managed by IANA in much the same way IP addresses are centrally managed. So for non-standard OIDs, manufacturers have requested their own OIDs and these can be looked up in the PEN (Private Enterprise Number) registry.
SNMP is mostly supported as a control method on enterprise or industrial hardware, not consumer grade. It's rare to find an off the shelf consumer smart switch, for example, that includes an SNMP agent. But most network equipment, even consumer level stuff like home routers and printers, can be managed via SNMP.
SNMP for me is less about controlling devices and more about pulling data from them. For example, my Netopia router supplies a list of DHCP clients (including MAC, time left on lease, IP, etc.) under the OID table 1.3.6.1.4.1.304.1.3.1.8.2.2 so it is easy to determine how many dynamic devices are currently on the network. And it also provides connection status in 1.3.6.1.4.1.304.1.3.1.12.18.1 , with the WAN IP address in the standard MIB ipAddrTable ( .1.3.6.1.2.1.4.20). With those two I have SNMP sensors which determine my Internet connection status and IP address without the need to query 3rd party services.
HTTP |
Many, or rather most, provide access via HTTP (HyperText Transfer Protocol). The same standard that your browser used to speak to my web server and request this page. HTTP is a client-server protocol where the client either requests via "GET" or sends via "POST", then the server responds back. Both these communications are formatted into a HTTP standard header which may contain various fields including any data the client may want to send to the server, or any status the server may send back to the client.
There are many ways the client may send data or make a request to the server. GET requests can be used to special URLs like GET "http://device/output1/on" to turn output 1 on. Or that may be supplied as URL parameters (query string) like GET "http://device/output?1=on". In either case, hopefully the device supplies a HTTP response containing a confirmation of the change. Another option is to POST the data to the device where the parameters are actually in the body of the HTTP request.
Sometimes this is in the form of a RESTful API.
Commands can be in the form of JSON, XML or really any format the device has been designed to understand.
Some devices may support higher level continuous communication using AJAX in which there is a continuous communication between the two hosts where JSON strings are used to exchange information.
Regardless of the method used, HTTP is an acceptable method of communications with a device having an expected lifespan of decades. As long as the requests and responses are well documented and in the clear (non-HTTPS), HTTP is a protocol not going away any time soon.
However, SSL secured HTTP (HTTPS) is something I am cautious about because certificate expiry can break communications. If the device only accepts requests via HTTPS, then at some point its root certificate store will be out of date and if the device doesn't trust self-signed certs or have some way to update its certificate store, communications could fail due to its inability to validate certificates.
Others |
As long as a protocol is open, documented and cannot be remotely depricated by a manufacturer, it is acceptable.
For example, I have some no-name Chinese relay boards that respond simply to a TelNet connection to a specific port and then a set of comma separated values telling it which relays to switch. This isn't any sort of standard protocol, but documentation came with the device and there is no way for the manufacturer to disable it. It will continue to work in that same way for as long as it continues to function.