CC100 Docker Daemon Issues after Reboot

Hardware and Linux
, ,
1

Dear Community

We use Docker on our Compact Controller 100 installations. Docker was installed and activated via WBM > Configuration > Docker. Docker uses the SD card as storage as defined in /etc/docker/daemon.json with the “data-root” configuration key, and we use the internal flash as the boot device.

Over the past few months, we have repeatedly noticed that when restarting (e.g. via SSH command ‘sudo /sbin/reboot’ or via the reboot button in the WBM), the Docker Deamon sometimes does not start cleanly. Sometimes the Deamon still starts, but the containers and images are missing (‘docker ps -a’ returns nothing, ‘docker images ls’ and ‘docker container ls’ also return empty results).

Have other users also experienced this phenomenon and were they able to find a solution to this problem?

Normally, a second reboot is sufficient so that the Docker Deamon runs again as usual when restarted and the containers/images are also output again with the corresponding commands. According to the entry in /etc/rc.d/S99_docker which came with the WBM service installation / activation, the Deamon should only be started at the very end, which is why I do not suspect that relevant service dependencies would be missing.

A further note from our side: In /etc/docker/daemon.json we also have another setting active: “shutdown-timeout”: 180, as our Docker containers need a little more time for their cleanup when shutting down.

If we want to gracefully reboot the system with enough time for the container to stop, we use our own bash script:
/usr/bin/docker stop && PID=$! & wait $PID && /sbin/reboot

We would now like to know how we can find out why the Docker Deamon sometimes fails to find containers/images after a reboot. Are there any specific logs or other clues that could help us with this?

Product: WAGO 751-9301 Compact Controller 100
Firmware revision: 04.01.10(23)

Thanks in advance

2

Hello :wave:

Unfortunately, I am not able to provide a solution to your problem, but I can assure you that you’re not the only one who has faced this issue - I am running a similar setup, taken the same steps as you have described.

The only other thing I have found out was that looking at kernel logs, I could find that an SD card has failed to load properly. These logs can be accessed from WBM. I will update this post with a picture of them next time it happens, but maybe it will help another user to point out a potential issue :slight_smile:

1 Like
3

Hello,

Did you investigate the root cause for this issue ?
I’m also struggling with exactly the same situation with need of double reboot of Docker, as it looses images and containers.
Is that the sd card loading issue ?

4

I had a problem like this on a couple of CC100s and fixed it by doing a quick and dirty SLEEP workaround in the docker startup script. I think docker ends up in a limbo mode if the boot isn’t clean and everything else on the CC100 tries to start up at the same time.

I modified /etc/rc.d/S99_docker and added sleep 180 before it starts the daemon.

function do_docker_start()
{
    echo "configuring network..."
    firewall_hook_activate
    echo -n "starting docker daemon..."
    sleep 180
    start-stop-daemon -S --background --quiet --oknodo --pidfile $DOCKER_PIDFILE --exec /usr/bin/env PATH="/"$DOCKER_IPTABLES_PATH":$PATH"  /usr/bin/dockerd --             #manipulate iptables-binary path!
}

After this I’ve never experienced the same problem again. The 180 seconds is just a random number I selected to be sure that everything else is done starting up.

3 Likes
5

Hello,
regarding the SD-Card issue I recommend to increase the timeout from 10 to 20 in the function acquire_lock of file /usr/local/bin/wago-automount

Best regards
Juri

3 Likes
6

Hey guys. We had the same problem and noticed that the SD Card had mount errors. Using the fdisk command, an error was displayed in the card’s partition table. We tried changing the scripts but it wasn’t enough. Our solution was to exchange common SD Cards for industrial pSLC SD Cards.

7

yes commonly sold SD card don’t have some safety feature that industrial one have.

1 Like