A Comprehensive Guide to MQTT — Part 2: Core Concepts of MQTT

Welcome to Part 2 of our MQTT journey! Last part, we covered the basics of MQTT, including its architecture, publish/subscribe model, and how it powers lightweight, efficient communication for IoT systems. This week, we’re stepping into more advanced territory: Quality of Service (QoS) levels, retained messages, Last Will and Testament (LWT), and session management. These features make MQTT incredibly versatile for real-world applications. Plus, we’ll wrap up with a hands-on experiment to solidify your understanding.

This story is free for everyone: Free Link

Pre-requisites:

Basic networks knowledge (stuff like TCP/UDP, Ports etc.)
Programming basics (any language will do)

Quality of Service (QoS) Levels:

MQTT offers three QoS levels to balance reliability and performance based on your application’s needs. Each level defines how messages are delivered between the publisher, broker, and subscriber.

QoS 0: At Most Once

How it works: The message is sent once, with no acknowledgment or retry mechanism. It’s a “fire and forget” approach — fast but not guaranteed to arrive.

When to use it: Ideal for scenarios where losing a message isn’t critical, and low latency is key. Think of frequent sensor updates (e.g., temperature readings every second) where missing one reading won’t disrupt the system.

Example: A weather station publishing non-critical data like humidity updates every 10 seconds. A update can be lost without much impact on the end product.

QoS 1: At Least Once

How it works: The sender ensures the message is delivered at least once by requiring an acknowledgment (PUBACK) from the receiver. If no acknowledgment is received, the message is resent, which could lead to duplicates.

When to use it: Use this when you need guaranteed delivery, but duplicates are tolerable. For instance, a smart home system sending a “turn on the lights” command — repeating it won’t cause harm.

Example: A logistics tracker updating package status.

QoS 2: Exactly Once

How it works: The most reliable level, using a four-step handshake (PUBREC, PUBREL, PUBCOMP) to ensure the message is delivered exactly once, with no duplicates. It’s the slowest but most robust option.

When to use it: Critical applications where duplicates or message loss could cause real problems, like financial transactions or medical device alerts.

Example: A hospital system sending a patient alert to a nurse’s station.

The QOS 2 Four-Step Handshake:

How QOS 2 works is pretty interesting and might even help you in designing better reliable distributed systems.

PUBLISH (Publisher to Broker):

The publisher sends the message to the broker with a unique Packet Identifier (Packet ID), a 16-bit number that tags this specific message. The QoS level is set to 2, signaling the start of the process.

The publisher stores the message in memory (not necessarily a database — more on that later) until it’s confident the broker has it.

PUBREC (Broker to Publisher):

The broker receives the message, stores it temporarily, and responds with a PUBREC (Publish Received) packet, echoing the Packet ID. This tells the publisher, “I’ve got your message, but hold on.”

The broker now takes responsibility for ensuring delivery to subscribers.

PUBREL (Publisher to Broker):

The publisher, upon receiving PUBREC, sends a PUBREL (Publish Release) packet with the same Packet ID. This is the publisher saying, “Okay, you’ve got it — now finish the job.”

At this point, the publisher can clear the message from its memory, as the broker has acknowledged receipt.

PUBCOMP (Broker to Publisher):

The broker responds with a PUBCOMP (Publish Complete) packet, confirming, “All done on my end.” This completes the publisher-to-broker leg of the journey.

Meanwhile, the broker follows a similar process to deliver the message to subscribers (if any), ensuring they also receive it exactly once.

Delivery to Subscribers:

The broker repeats a QoS 2-like process with each subscriber

It sends the message with the Packet ID.

The subscriber responds with PUBREC, then PUBREL after processing, and the broker finalizes with PUBCOMP.

This ensures the subscriber gets the message exactly once, even if the network hiccups.

How QOS 2 Prevents Loss and Duplication?

If any step fails (e.g., PUBREC isn’t received due to a network drop), the sender (publisher or broker) retransmits the last packet with a “DUP” (duplicate) flag until acknowledged. The Packet ID ensures the receiver recognizes it’s not a new message.

The Packet ID is key here. Both the broker and subscriber track Packet IDs they’ve processed. If a duplicate arrives (e.g., due to retransmission), they discard it after acknowledging it, ensuring it’s not processed twice.

Retained Messages

How Do Retained Messages Work?

When a publisher sends a message with the “retained” flag set, the broker stores the last retained message for that topic. Any new subscriber to that topic immediately receives this retained message, even if they connect after it was originally published.

If a publisher sends a new retained message, it replaces the old one. If a publisher sends a null retained message, the retained message is deleted.

For example, for sharing static or semi-static data, like a system’s operating mode, with all subscribers or An IoT dashboard showing the latest state of a factory machine.

Use retained messages sparingly, as they persist on the broker until overwritten or cleared, consuming broker’s memory.

Last Will and Testament (LWT)

LWT is like an emergency broadcast. When a client connects to a broker, it can specify an LWT message and topic. If the client disconnects unexpectedly (e.g., due to a network failure or crash), the broker publishes the LWT message to the designated topic on behalf of the client.

Setting Up LWT

During the MQTT CONNECT phase, the client specifies:

LWT Topic: Where the message will be sent.
LWT Message: The content (e.g., “Device offline”).
QoS Level: For delivery assurance.

The broker monitors the client’s connection (via keep-alive pings).
If the client drops without a proper disconnect, the broker sends the LWT.

Use Case:

Notify other clients when a device goes offline unexpectedly or trigger fallback actions (e.g., switch to a backup device) when a critical component fails.

Clean Session vs. Persistent Session

When a client connects with clean_session=True, the broker discards any prior session data (subscriptions, queued messages) and starts fresh. Once the client disconnects, the session is terminated.

This is useful for temporary clients or applications where historical data isn’t needed, like a one-off diagnostic tool.

With clean_session=False, the broker retains the client’s session (subscriptions and undelivered QoS 1/2 messages) even after disconnection. When the client reconnects with the same Client ID, it resumes where it left off.

This can be pretty useful for long-lived IoT devices that need to pick up missed messages, like a smart meter tracking power usage.

Example: A weather station with a persistent session ensures it receives any missed alerts (e.g., storm warnings) when it reconnects after a power outage.

Hands-On 002: Smart Warehouse Temperature Monitoring

Let’s build a real-world example: a temperature monitoring system for a smart warehouse. We’ll use Go with the paho.mqtt.golang library and EMQX as our MQTT broker, but you can use any language and any broker (like Mosquito, HiveMQ etc). The system will:

Publish temperature readings with different QoS levels.
Use retained messages to share the latest temperature.
Set an LWT to alert when a sensor fails.
Demonstrate clean vs. persistent sessions.

Please note that the paho.mqtt.golang library supports only MQTT 3.1.1 which is enough for our hands-on for this part.

Prerequisites

Before getting started, if you are following along with Go, make sure to

Have Go install in your computer
Have EMQX installed in your computer (or) you can use the publicly available EMQX broker.
Initalize the Go project and download the MQTT library

go get github.com/eclipse/paho.mqtt.golang

Project Structure

We’ll create two Go files:

sensor.go: Simulates a temperature sensor publishing data.

monitor.go: Subscribes to sensor data and alerts.

Temperature Sensor (sensor.go)

This simulates a warehouse temperature sensor. It publishes temperature data to an MQTT topic, sets an LWT message, and uses retained messages for critical updates.

package main

import (
    "fmt"
    "math/rand"
    "os"
    "os/signal"
    "time"

    mqtt "github.com/eclipse/paho.mqtt.golang"
)

func main() {
    // MQTT broker connection options
    opts := mqtt.NewClientOptions().
        AddBroker("tcp://localhost:1883").
        SetClientID("sensor_001").
        SetCleanSession(false) // Persistent session to queue messages

    // Set Last Will and Testament
    opts.SetWill("warehouse/sensor/status", "Sensor 001 offline", 1, false)

    // Connect to EMQX
    client := mqtt.NewClient(opts)
    if token := client.Connect(); token.Wait() && token.Error() != nil {
        panic(token.Error())
    }
    defer client.Disconnect(250)
    fmt.Println("Sensor connected to EMQX")

    // Simulate temperature readings
    go func() {
        for i := 0; ; i++ {
            temp := 20 + rand.Float32()*10 // Random temp between 20-30°C
            topic := "warehouse/temperature"

            // Publish with QoS 0 (non-critical frequent updates)
            if i%5 != 0 {
                client.Publish(topic, 0, false, fmt.Sprintf("%.2f", temp))
                fmt.Printf("Published QoS 0: %.2f°C\n", temp)
            } else {
                // Every 5th message is critical (retained, QoS 2)
                client.Publish(topic, 2, true, fmt.Sprintf("%.2f", temp))
                fmt.Printf("Published QoS 2 (retained): %.2f°C\n", temp)
            }
            time.Sleep(2 * time.Second)
        }
    }()

    // Graceful shutdown
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, os.Interrupt)
    <-sigChan
    fmt.Println("Shutting down sensor...")
}

The program connects to the broker and starts publishing temperature readings every two seconds. Most messages use QoS 0 (fast but unreliable), while every fifth message is sent with QoS 2 (ensuring exactly one delivery) and is marked as “retained” so new subscribers get the latest reading immediately. If the sensor crashes, the broker sends the LWT message to inform other clients.

Warehouse Monitor (monitor.go)

This monitor subscribes to temperature data and the sensor’s status, reacting to critical temperatures and LWT messages.

package main

import (
    "fmt"
    "os"
    "os/signal"
    "strconv"

    mqtt "github.com/eclipse/paho.mqtt.golang"
)

func main() {
    // MQTT broker connection options
    opts := mqtt.NewClientOptions().
        AddBroker("tcp://localhost:1883").
        SetClientID("monitor_001").
        SetCleanSession(true) // Clean session for simplicity

    // Callback for incoming messages
    opts.SetDefaultPublishHandler(func(client mqtt.Client, msg mqtt.Message) {
        payload := string(msg.Payload())
        switch msg.Topic() {
        case "warehouse/temperature":
            temp, _ := strconv.ParseFloat(payload, 32)
            fmt.Printf("Received temp: %.2f°C (QoS: %d)\n", temp, msg.Qos())
            if temp > 28 {
                fmt.Println("ALERT: Temperature exceeds 28°C!")
            }
        case "warehouse/sensor/status":
            fmt.Printf("Status update: %s\n", payload)
        }
    })

    // Connect to EMQX
    client := mqtt.NewClient(opts)
    if token := client.Connect(); token.Wait() && token.Error() != nil {
        panic(token.Error())
    }
    defer client.Disconnect(250)
    fmt.Println("Monitor connected to EMQX")

    // Subscribe to topics
    client.Subscribe("warehouse/temperature", 2, nil)
    client.Subscribe("warehouse/sensor/status", 1, nil)
    fmt.Println("Subscribed to temperature and status topics")

    // Keep running until interrupted
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, os.Interrupt)
    <-sigChan
    fmt.Println("Shutting down monitor...")
}

In monitor.go, the program subscribes to the temperature and status topics, printing received messages and raising an alert if the temperature exceeds 28°C. It also listens for the LWT message, alerting when the sensor goes offline.

Running the Example

Start EMQX (if not already running).
In one terminal, run the monitor:

go run monitor.go

3. In another terminal, run the sensor

go run sensor.go

Observations:

The monitor receives frequent QoS 0 updates (may miss some if disconnected) and guaranteed QoS 2 retained messages every 5th update.

Stopping and restarting the monitor — it instantly gets the last retained temperature (QoS 2 message).

Killing the sensor with Ctrl+C (simulating a crash) — the monitor receives the “Sensor 001 offline” message.

The sensor uses a persistent session (clean_session=false), so if you stop and restart it, it resumes publishing without losing its state. The monitor uses a clean session, starting fresh each time.

The Big Picture

In Part 2, we’ve looked at MQTT’s core concepts — QoS levels, retained messages, LWT, and session management — and seen them in action with a Go-based warehouse monitoring system using EMQX. These features make MQTT adaptable to diverse IoT needs, from casual updates to mission-critical alerts. Experiment with the code: tweak QoS levels, simulate network drops, or add more sensors

Next up, we’ll explore MQTT security

“As time goes on, you’ll understand. What lasts, lasts; what doesn’t, doesn’t. Time solves most things. And what time can’t solve, you have to solve yourself.”
― Haruki Murakami, Dance Dance Dance

SpaceTime