# High system CPU usage on CC100 with PREEMPT\_RT kernel – HRTIMER / NET\_RX softirq storm
**URL:** https://www.wago.community/t/high-system-cpu-usage-on-cc100-with-preempt-rt-kernel-hrtimer-net-rx-softirq-storm/2496
**Category:** Hardware and Linux
**Tags:** cc100
**Created:** [January 9, 2026, 4:52pm UTC](https://www.wago.community/t/high-system-cpu-usage-on-cc100-with-preempt-rt-kernel-hrtimer-net-rx-softirq-storm/2496 "2026-01-09T16:52:32Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@bernardms](https://www.wago.community/u/bernardms)
#### Post date: [January 9, 2026, 4:52pm UTC](https://www.wago.community/t/high-system-cpu-usage-on-cc100-with-preempt-rt-kernel-hrtimer-net-rx-softirq-storm/2496/1 "2026-01-09T16:52:32Z")
Hello everyone,
I’m looking for insights regarding a high system CPU usage issue on a CC100 controller.
System details
- Device: WAGO CC100
- Kernel:
Linux 5.15.107-rt62-w04.02.03 #1 SMP PREEMPT\_RT
armv7l
- Single-core CPU
- CODESYS: disabled
- User application: SCADA, written in Python
- Read/communication cycle: every 10 seconds
- No busy loops or high-frequency polling
Despite the low-frequency application logic, the system shows ~100% CPU usage, with about 70–80% attributed to kernel/system time (top).
Clarification
- The issue persists even when the SCADA application is stopped
- The same behavior is observed with and without Docker
- Disabling observability / monitoring tools (metrics, exporters, etc.) does not change CPU usage
This further suggests the problem is **not related to containers, monitoring, or user-space tooling**.
Observations
/proc/interrupts (excerpt)
- Very high interrupt count on:
- arch\_timer (~1.5 billion)
- eth0 (~57 million)
- IRQ work interrupts (IPI5)
/proc/softirqs
HRTIMER: 1216929071
TIMER: 191172500
NET\_RX: 221283404
This strongly suggests a timer + network softirq storm, not user-space load.
CPU idle behavior
From repeated /proc/stat checks:
- The CPU can enter idle
- But the idle counter barely increases
- CPU is constantly being woken up by timers and softirqs
Boot parameters
cat /proc/cmdline
No idle=poll, nohlt, or other parameters forcing busy idle.
Important clarification
The SCADA application:
- Runs every 10 seconds
- Uses blocking I/O
- Is not CPU-intensive
- Reducing or stopping it does not significantly change system CPU usage
This points to a kernel / driver / RT configuration issue, not an application bug.
My understanding so far
It seems this is related to:
- PREEMPT\_RT kernel behavior on a single-core system
- Very frequent HRTIMER activity
- High NET\_RX softirq rate (even with modest network usage)
While this may be expected behavior for an RT kernel, it significantly reduces idle time and margin for real-time workloads.
1. Is this level of HRTIMER / NET\_RX activity considered normal on CC100 with PREEMPT\_RT?
2. Has anyone successfully reduced CPU usage via:
- Network driver tuning (interrupt coalescing, NAPI behavior)?
- Kernel configuration (HZ, timer settings)?
- Alternative firmware / kernel variants (PREEMPT instead of RT)?
3. Are there known recommendations from WAGO for SCADA-style workloads on CC100 where hard real-time is not required?
Any shared experience, tuning advice, or confirmation would be greatly appreciated.
Thanks in advance!
---
### Author:  [@bernardms](https://www.wago.community/u/bernardms)
#### Post date: [January 12, 2026, 4:26pm UTC](https://www.wago.community/t/high-system-cpu-usage-on-cc100-with-preempt-rt-kernel-hrtimer-net-rx-softirq-storm/2496/2 "2026-01-12T16:26:10Z")
\*\* SOLVED \*\*
Root cause identified.
The issue was caused by the ipwatchd service interacting badly with Docker-created network bridges.
ipwatchd was repeatedly trying to monitor Docker bridge interfaces (br-xxxx), generating continuous log spam like:
“Could not retrieve IP address of the device br-xxxx”
This resulted in:
- Constant wakeups
- High HRTIMER and NET\_RX softirq activity
- Near 100% system CPU usage on a single-core PREEMPT\_RT system
Stopping ipwatchd immediately resolves the issue:
/etc/init.d/ipwatchd stop
No network disruption was observed:
- VPN connection remained active
- Main Ethernet (br0 / eth0) continued working normally
- SCADA application unaffected
Inspection of /etc/ipwatchd.conf shows that Docker bridges are automatically added to the configuration (even as passive), and the file is auto-generated and overwritten, making selective exclusion unreliable.
Hope this helps anyone facing similar high system CPU usage on CC100.