Telegraf GNMI Input Plugin Panic On Empty Updates With Deletes A Deep Dive

Jul 31, 2025 by ADMIN 75 views

Telegraf gNMI Input Plugin Panic with Empty Updates and Deletes

Hi guys! Today, we're diving into a tricky issue encountered with the Telegraf gNMI input plugin. Specifically, we're talking about a scenario where Telegraf panics when it receives an empty update coupled with deletes. This can be a real headache, especially when you're relying on Telegraf for critical monitoring tasks. Let's break down the problem, understand the root cause, and explore potential solutions.

Understanding the Issue

The core problem revolves around how the Telegraf gNMI input plugin handles delete operations, especially when an update doesn't contain any data but includes delete instructions. In simpler terms, imagine Telegraf is watching a network device for changes. If a monitored interface is deleted, the device might send an update that says, "Hey, this interface is gone," but doesn't include any specific data about it. This empty update combined with delete instructions is what triggers the panic.

The Configuration

To illustrate the issue, let's look at a sample Telegraf configuration (telegraf.conf) that triggers this behavior:

[[inputs.gnmi]]
  addresses     = ["192.168.10.1"]
  redial        = "10s"

  [[inputs.gnmi.subscription]]
    name = "interface_state_change"
    path = "huawei-ifm:ifm/interfaces/interface/dynamic/oper-status"
    subscription_mode = "on_change"

This configuration tells Telegraf to connect to a gNMI-enabled device at 192.168.10.1 and subscribe to changes in the oper-status of network interfaces using the on_change subscription mode. This means Telegraf will receive updates whenever the operational status of an interface changes. The redial setting ensures that Telegraf will attempt to reconnect every 10 seconds if the connection is lost.

The Panic Logs

When the issue occurs, Telegraf logs a panic message, providing valuable clues about the root cause. Here’s an example of the panic logs:

2025-07-31T07:17:56Z I! Loading config: /etc/telegraf/telegraf-on-change.conf
2025-07-31T07:17:56Z I! Config watcher started for /etc/telegraf/telegraf-on-change.conf
{"time":"2025-07-31T07:17:56.759021369Z","level":"INFO","msg":"Loading config: /etc/telegraf/telegraf-on-change.conf"}
{"time":"2025-07-31T07:17:56.76488263Z","level":"INFO","msg":"Config watcher started for /etc/telegraf/telegraf-on-change.conf"}
{"time":"2025-07-31T07:17:56.764994381Z","level":"INFO","msg":"Starting Telegraf 0.0.1 (customized) brought to you by InfluxData the makers of InfluxDB"}
{"time":"2025-07-31T07:17:56.765002467Z","level":"INFO","msg":"Available plugins: 7 inputs, 1 aggregators, 5 processors, 1 parsers, 3 outputs, 0 secret-stores"}
{"time":"2025-07-31T07:17:56.765007576Z","level":"INFO","msg":"Loaded inputs: gnmi internal"}
{"time":"2025-07-31T07:17:56.76500994Z","level":"INFO","msg":"Loaded aggregators:"}
{"time":"2025-07-31T07:17:56.765012128Z","level":"INFO","msg":"Loaded processors:"}
{"time":"2025-07-31T07:17:56.765014672Z","level":"INFO","msg":"Loaded secretstores:"}
{"time":"2025-07-31T07:17:56.765018553Z","level":"INFO","msg":"Loaded outputs: health kafka (2x)"}
{"time":"2025-07-31T07:17:56.765021494Z","level":"INFO","msg":"Tags enabled: host=huawei-gnmi-collector-ne8000-m14-6798613e-7b4c5c5fb7-9rvc7"}
{"time":"2025-07-31T07:17:56.765033648Z","level":"INFO","msg":"[agent] Config: Interval:5m0s, Quiet:false, Hostname:"huawei-gnmi-collector", Flush Interval:10s"}
{"time":"2025-07-31T07:17:56.767599316Z","level":"INFO","msg":"Listening on http://[::]:8082","plugin":"health","category":"outputs"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xdc0116]

goroutine 85 [running]:
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*pathInfo).append(0xc000a12050, {0xc0005c3940, 0x1, 0x6?})
        github.com/influxdata/telegraf/plugins/inputs/gnmi/path.go:134 +0x3d6
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*handler).handleSubscribeResponseUpdate(0xc0005c3d80, {0x1943900, 0xc000278d40}, 0xc0006d2030, {0x0, 0x0, 0x5?})
        github.com/influxdata/telegraf/plugins/inputs/gnmi/handler.go:179 +0x59d
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*handler).subscribeGNMI(0xc0005c3d80, {0x1934558, 0xc00033cd50}, {0x1943900, 0xc000278d40}, 0x1?, 0xc000309810)
        github.com/influxdata/telegraf/plugins/inputs/gnmi/handler.go:123 +0xc45
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start.func1({0xc0005224f8, 0x11})
        github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:313 +0x785
created by github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start in goroutine 1
        github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:280 +0x359

The key part of the log is the panic: runtime error: invalid memory address or nil pointer dereference message. This indicates that the code is trying to access a memory location that is either invalid or doesn't exist, leading to the crash. The stack trace provides further details, pinpointing the issue to the github.com/influxdata/telegraf/plugins/inputs/gnmi package, specifically in the path.go and handler.go files. This helps us narrow down the area of the codebase where the problem lies.

Steps to Reproduce

To consistently reproduce this issue, follow these steps:

Run Telegraf with the provided configuration.
Delete an interface from the Huawei device that Telegraf is monitoring.
Observe that Telegraf crashes with the panic message.

This sequence of actions reliably triggers the bug, making it easier to test potential fixes.

Expected vs. Actual Behavior

The expected behavior is that Telegraf should gracefully handle empty updates, especially those associated with delete operations. It should ignore these updates and continue to function without crashing. The actual behavior, however, is that Telegraf crashes due to a panic, disrupting the monitoring process.

Root Cause Analysis

The panic occurs because the Telegraf gNMI input plugin isn't correctly handling empty updates that include delete operations. When an interface is deleted on the device, the gNMI server sends an update to Telegraf. This update might not contain any data (i.e., it's empty) but includes instructions to delete the corresponding entry in Telegraf's internal data structures. The plugin's code attempts to process this empty update, leading to a nil pointer dereference when it tries to access data that doesn't exist. This is akin to trying to read a value from an empty box – there's nothing there, and the attempt to access it causes a crash.

Potential Bug in Device

It's also worth noting that this behavior might indicate a bug in the device's gNMI implementation. Sending an empty update for a delete operation might not be the standard or expected behavior. Ideally, a device should either send a complete update with the new state or not send an update at all for a delete operation. However, regardless of the device's behavior, Telegraf should be resilient and handle such scenarios gracefully.

Solutions and Workarounds

While the ideal solution is to fix the bug in the Telegraf gNMI input plugin, there are a few workarounds you can use in the meantime to mitigate the issue.

1. Code-Level Fix

The most effective solution is to modify the Telegraf gNMI input plugin code to handle empty updates correctly. This involves adding a check for empty updates before attempting to process them. If an update is empty and contains delete operations, the plugin should skip processing it and log a warning or debug message. This prevents the nil pointer dereference and allows Telegraf to continue running.

The fix would likely involve modifying the handleSubscribeResponseUpdate function in handler.go and the append method in path.go to check for nil values before accessing them. Here’s a simplified example of how the fix might look in handler.go:

func (h *handler) handleSubscribeResponseUpdate( /* ... */ ) {
    if update == nil || len(update.Deletes) > 0 && len(update.Updates) == 0 {
        log.Warn("Received empty update with deletes, skipping processing")
        return
    }
    // ... rest of the processing logic
}

This code snippet checks if the update is nil or if it contains deletes but no actual updates. If either condition is true, it logs a warning and returns, effectively skipping the problematic processing logic.

2. Device Configuration (If Possible)

If the issue is indeed caused by the device sending incorrect updates, you might be able to configure the device to avoid sending empty updates for delete operations. This might involve tweaking the gNMI server settings on the device or updating the device's firmware. However, this workaround depends on the device's capabilities and might not always be feasible.

3. Filtering Updates

Another workaround is to introduce a filtering mechanism that discards empty updates before they reach the problematic code. This can be done by adding a processor plugin in Telegraf that inspects the updates and drops those that are empty. While this adds complexity to the configuration, it can prevent the crashes.

4. Temporary Restart Script

As a last resort, you can implement a script that automatically restarts Telegraf when it crashes. This doesn't solve the underlying issue, but it minimizes the downtime caused by the crashes. However, this should be considered a temporary measure until a proper fix is implemented.

Conclusion

The panic in the Telegraf gNMI input plugin when handling empty updates with deletes is a significant issue that can disrupt monitoring operations. Understanding the root cause and implementing the appropriate solution or workaround is crucial for maintaining a stable monitoring environment. Whether it's fixing the code, configuring the device, or implementing a temporary workaround, addressing this issue ensures that Telegraf can reliably collect and process data from your network devices. So, keep an eye out for updates and fixes, and until then, consider the workarounds to keep your Telegraf instances running smoothly!