Unknown stop error and block of PFC200 (750-8213)

CODESYS
1

Hello guys ,
I have two questions about my issue:

  1. Is it possible to find out which or what makes that PLC to stop responding , and how to do this research ?
  2. Is there a sequence of which module to be placed at a special place after main module? I mean mandatory one or good practice …
    So the system is maybe ordinary : we use this PLC as a charging controller for EVs. We have 1 main module (750-8213) , next is 750-652 (as RS485 communication to Frequency Converter Rexroth and Isolation Meter device - Bender), next is 750-658 (as communication module via CAN to Phoenix Contact HPC charging cable), next is another one 750-658 (for ChaDeMo communication), 750-430 → 2 pcs for various on/off sensing , 750-530 → 1pcs for on/off switching, 750-514 → on/off for ChaDeMo switching, 750-461 → Analog Inputs 4-20mA for Voltage measurements.Of course end module 750-600 is installed too. To X4 CAN of main module are connected power converters. We use Modbus over TCP to communicate with HMI interface, pc-based, MQTT for communication with a separate pcb , thru which connects to the EV (in wbm is set MQTT broker ). Almost independently for the past 3 months the PLC stops responding at “almost random” time, usually at night. When this happens one red LED (A) is lit on the 750-652 , (B) & (E) are steady green, and on 750-658 (A) & (B) are steady green , but (H) is red . Starting “top” or “htop” commands while working show “normal” duty at CPU at 56% up to 80% for all threads running (Tasks: 42 , Threads: 68, 1 running ). It is interesting that other almost “same” configurations of charger controllers does not have the same behaviour , except no Frequency Converter is communicating, but another device Energy Meter (Carlo Cavazzi) instead. I found frequently message into Diagnostic tool box , saying :

Thu Jan 02 2025 10:45:04.638181 eventmsg: Event reset:Unexpected stop of runtime
Thu Jan 02 2025 10:45:09.739038 init: System Booted
Thu Jan 02 2025 10:45:11.458482 CODESYS_3: KBus is booting
Thu Jan 02 2025 10:45:13.623451 CODESYS_3: KBus has booted.
Thu Jan 02 2025 10:45:13.628232 CODESYS_3: KBus is operational
Thu Jan 02 2025 10:45:13.686293 CODESYS_3: The PROFIBUS DP device has not been opened
Thu Jan 02 2025 10:45:13.690001 CODESYS_3: The PROFIBUS DP device has not been opened
Thu Jan 02 2025 10:45:13.755689 CODESYS_3: The PROFIBUS DP device has been closed
Thu Jan 02 2025 10:45:14.614892 CODESYS_3: PLC is online and no program is loaded
Thu Jan 02 2025 10:45:16.000812 CODESYS_3: Event reset:Common System Error
Thu Jan 02 2025 10:45:16.005773 CODESYS_3: CAN device has not been opened
Thu Jan 02 2025 10:45:16.080792 CODESYS_3: Event reset:CAN device has not been opened
Thu Jan 02 2025 10:45:16.087580 CODESYS_3: The CAN device has been opened
Thu Jan 02 2025 10:45:16.119918 CODESYS_3: Event reset:The CAN device has been opened
Thu Jan 02 2025 10:45:16.122512 CODESYS_3: CAN Operational
Thu Jan 02 2025 10:45:16.148823 CODESYS_3: Application stopped
Thu Jan 02 2025 10:45:16.463178 CODESYS_3: Application is running\

I have read many topics here but did not found anything similar to my case . Thanks in advance !

2

Hi again guys ,
here is what I see when “top” is sent to the wago PLC . But I hope to understand what means “zombie” thread running and which is it , like here :

root@PFC200V3-4F0B1B:~ top

top - 16:01:05 up 23:48, 1 user, load average: 1,13, 1,20, 1,22
Tasks: 122 total, 1 running, 120 sleeping, 0 stopped, 1 zombie
%Cpu(s): 36,2 us, 42,0 sy, 0,0 ni, 15,9 id, 0,0 wa, 0,0 hi, 5,8 si, 0,0 st
MiB Mem : 490,090 total, 299,547 free, 96,871 used, 93,672 buff/cache
MiB Swap: 0,000 total, 0,000 free, 0,000 used. 347,605 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1548 root 20 0 109948 98624 34164 S 41,7 19,7 782:47.88 codesys3
863 root 20 0 10812 5316 4352 S 10,4 1,1 189:53.73 syslog-ng
12779 root 20 0 7824 2488 2032 R 8,3 0,5 0:00.09 top
3640 root -51 0 0 0 0 S 6,2 0,0 79:29.92 irq/40-can0
12 root 20 0 0 0 0 S 2,1 0,0 15:41.83 ksoftirqd/0
85 root -87 0 0 0 0 S 2,1 0,0 36:22.25 spi0
1 root 20 0 2808 1464 1392 S 0,0 0,3 0:04.17 init
2 root 20 0 0 0 0 S 0,0 0,0 0:00.30 kthreadd
3 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 rcu_par_gp
6 root 0 -20 0 0 0 I 0,0 0,0 0:00.10 kworker/0:0H-mmc_complete
8 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 mm_percpu_wq
9 root 20 0 0 0 0 S 0,0 0,0 0:00.00 rcu_tasks_kthre
10 root 20 0 0 0 0 S 0,0 0,0 0:00.00 rcu_tasks_rude_
11 root 20 0 0 0 0 S 0,0 0,0 0:00.00 rcu_tasks_trace
13 root -2 0 0 0 0 I 0,0 0,0 5:25.37 rcu_preempt
14 root -2 0 0 0 0 S 0,0 0,0 0:00.00 rcub/0
15 root -2 0 0 0 0 S 0,0 0,0 8:29.63 rcuc/0
16 root 20 0 0 0 0 S 0,0 0,0 0:00.00 kdevtmpfs
17 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 netns
19 root 20 0 0 0 0 S 0,0 0,0 0:00.00 oom_reaper
20 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 writeback
66 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 kintegrityd
67 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 kblockd
68 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 blkcg_punt_bio
69 root -51 0 0 0 0 S 0,0 0,0 0:00.00 watchdogd
71 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 rpciod
72 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 kworker/u3:0
73 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 xprtiod
74 root 20 0 0 0 0 S 0,0 0,0 0:00.00 kswapd0
75 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 nfsiod
77 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 kthrotld
78 root -51 0 0 0 0 S 0,0 0,0 0:00.25 irq/47-49000000
79 root -51 0 0 0 0 S 0,0 0,0 0:00.00 irq/49-49000000
80 root -51 0 0 0 0 S 0,0 0,0 0:00.00 irq/42-48310000
81 root 20 0 0 0 0 S 0,0 0,0 0:00.00 hwrng
83 root -51 0 0 0 0 S 0,0 0,0 0:00.00 irq/35-48080000
84 root -87 0 0 0 0 S 0,0 0,0 0:00.00 irq/27-48030000
87 root -51 0 0 0 0 S 0,0 0,0 16:48.64 irq/44-4a100000
88 root -51 0 0 0 0 S 0,0 0,0 2:22.24 irq/45-4a100000
89 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 uas
90 root -51 0 0 0 0 S 0,0 0,0 0:00.00 53500000.aes-en
96 root -51 0 0 0 0 S 0,0 0,0 0:00.00 irq/53-53100000
97 root -51 0 0 0 0 S 0,0 0,0 0:00.00 53100000.sham-e
106 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 dsa_ordered
107 root 20 0 0 0 0 S 0,0 0,0 0:00.00 pr/tty0
108 root -51 0 0 0 0 S 0,0 0,0 0:00.00 irq/19-gpmc
109 root -87 0 0 0 0 S 0,0 0,0 6:11.48 irq/20-44e07000
111 root -51 0 0 0 0 S 0,0 0,0 0:00.69 irq/22-44e0b000
113 root -51 0 0 0 0 S 0,0 0,0 0:00.00 irq/33-4804c000
114 root -51 0 0 0 0 S 0,0 0,0 0:00.00 irq/38-481ac000
115 root -51 0 0 0 0 S 0,0 0,0 0:00.00 irq/39-481ae000
116 root -87 0 0 0 0 S 0,0 0,0 3:26.82 irq/55-kbus
121 root -51 0 0 0 0 S 0,0 0,0 0:00.00 irq/57-tps65218
122 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 sdhci
123 root 20 0 0 0 0 I 0,0 0,0 0:00.00 kworker/u2:2-ext4-rsv-conversion
124 root -91 0 0 0 0 S 0,0 0,0 0:00.00 kworker/gpiokey

3

You probably have an exception somewhere in your code, related to a index overflow or a division by 0…

You might want to use POU for Implicit Checks.

In your CODESYS program, right click on your Application / Add Object / POU for Implicit Checks…

Check all the available functions.

image
image346×500 7.21 KB

After uploading your program on the PLC, and before starting it, go in each function and put a breakpoint (F9) in the error statement of the function (IF divisor = 0 … IF value < lower, IF value > higher…)

image
image354×147 5.15 KB

image
image344×119 2.55 KB

Start you program. It should end to a breakpoint. Then you can Step over breakpoint (F10) in order to see in which part of the program there is an issue.

1 Like
4

Hello quenorha ,
Thanks for your reply !
I forgot to mention that we still use e!Cockpit and latest firmware available for it. The main reason for this is that we have and some additional wago PLCs (750-8203) that are not supported by next level of Codesys, but they still work satisfying with almost the same functioning. Just the CAN protocol differs for another kind of power converters and CAN instead of MQTT too for EV communication.
So migrating in between will be difficult according to me and my colleagues , who actually write and configure the app for the wago plc.
Maybe there is similar tool in e!Cockpit ?

Regards ,
Anatoliy

5

Hello guys ,
Let me just share an update - one of the power converters on the next PLC (750-8213) was not responding via integrated CAN.
Obviously, but not so often and not all the time , CAN LED was changing colors green/red. But CAN 750-658 & RS485 750-652 were working fine at that moment. I notified this at the most critical moment , when tried to raise the power of charger to the next step.
I would say we have to improve the “bubble” function of how to setup which converters to work at right moment.
Our thinking about interruptions of CPU, Memory allocated , loading time and so on was not so useful at this moment.

Thank you all guys !
Regards,
Anatoliy

2 Likes