View a large number of files on an SD card

Hardware and Linux
1

I had many files on my SD card, more than 6000 thousand. Through visualization or through filezilla I was unable to access the directory. I had to take out the card and read it through a card reader to understand what files were there. Are there any restrictions on viewing files? Maybe somehow you need to limit the program recording so that a similar situation does not happen again?

2

Which controller do you use? In linux controllers like PFC100/200/TP600/EDGE/CC100 there is /etc/pure-ftpd.conf, but there are no limits defined by default.

but.. request ā€˜full directory listingā€™ allocate memory for file names for listing, so its good to limit ~1000 files per folder, the best is to create logs in hierarchy /year/month/log_date
When creating files you can use WagoAppString.Format1(ā€˜/media/sd/Logs/%{%Y/%M/Log_%D}t.csvā€™,ADR(typWagoTimeComponents)) function to name new files.

So you can create new files based on date:

PROGRAM PLC_PRG
VAR
	FileTest : WagoSysFileDir.FbSysFile;
	typWagoTimeComponents : WagoAppTime.typWagoTimeComponents;
	eAccMode: WagoSysFileDir.eFileAccessMode := WagoSysFileDir.eFileAccessMode.FAM_Append;
	eSyncMode: WagoSysFileDir.eFileSyncMode;
	xOpen: BOOL;
END_VAR

IF xOpen THEN
	xOpen := FALSE;

	typWagoTimeComponents := FuGetLocalTimeComponents();
	FileTest.Open(sName:= WagoAppString.Format1('/media/sd/Logs/%{%Y/%M/Log_%D_%h-%m}t.csv',ADR(typWagoTimeComponents)), eAccMode:= eAccMode, eSyncMode:= eSyncMode, xExclusive:= FALSE);
	FileTest.Write(pTxBuffer:= ADR('test'),4);
	FileTest.Close();
END_IF
3

I missed the message I have PFC200 (8202/8212)
Do you propose to sort simply by date within the program? Or the second option is to expand the range (from 1000 to 5000, for example)?

4

Itā€™s not limited, so its not a problem, but so many files in 1 folder make problems. Thatā€™s why I prefer to use YEAR/MONTH folder hierarchy, you can have less files in one :file_folder:.

I had similar problem, so I created organize.py python script which you can run with docker (so not with PFC100/PFC200 G1 750-8202, but all new can do this). This script takes modification time of the file and move them to YEAR/MONTH/DAY folder hierarchy.

  1. Connect to the controller using SSH console (via PuTTY)
  2. Create python script on the controller - inside folder whish you want to organize.
nano organize.py
  1. Paste this script
import os
import shutil
from datetime import datetime

# Get current working directory
SOURCE_DIR = os.getcwd()

def get_creation_time(path):
    """
    Returns creation time of file.
    On Linux, falls back to last metadata change time.
    """
    if os.name == 'nt':  # Windows
        return os.path.getctime(path)
    else:  # Linux / Unix
        stat = os.stat(path)
        try:
            return stat.st_birthtime  # macOS
        except AttributeError:
            return stat.st_mtime  # fallback (modification time)

def organize_files():
    for filename in os.listdir(SOURCE_DIR):
        file_path = os.path.join(SOURCE_DIR, filename)

        # Skip directories and this script itself
        if not os.path.isfile(file_path):
            continue

        # Get file time
        timestamp = get_creation_time(file_path)
        dt = datetime.fromtimestamp(timestamp)

        year = str(dt.year)
        month = f"{dt.month:02d}"
        day = f"{dt.day:02d}"

        # Create target directory
        target_dir = os.path.join(SOURCE_DIR, year, month, day)
        os.makedirs(target_dir, exist_ok=True)

        # Move file
        target_path = os.path.join(target_dir, filename)

        # Handle duplicate filenames
        if os.path.exists(target_path):
            base, ext = os.path.splitext(filename)
            counter = 1
            while True:
                new_name = f"{base}_{counter}{ext}"
                new_path = os.path.join(target_dir, new_name)
                if not os.path.exists(new_path):
                    target_path = new_path
                    break
                counter += 1

        shutil.move(file_path, target_path)
        print(f"Moved: {filename} -> {target_dir}")

if __name__ == "__main__":
    organize_files()
  1. Turn on Docker (on Web-Based Management) or call:
/etc/config-tools/config_docker install && /etc/config-tools/config_docker activate
  1. Now you can call organize.py script from the folder
cd /folder/where/you/placed/organize.py
docker run --rm -it -v $(pwd):/app -w /app python:3-alpine python organize.py

As a result files will be moved to folders according to the modification date:

image
image1039Ɨ111 3.85 KB

5

Can this be done without Docker if you install Python and just run the script?

6

The YEAR/MONTH suggestion by @WagoDamRud is by far the best practice to prevent a situation like this. However if you already have the single directory with a large number of files and you attempt to use FileZilla to browse the directory through ftp then it may take a long time to get the full list of files depending not only on the number of files but also on the speed of the connection to the controller. In other words: listing of files will time out and no files will be shown because it simply takes longer than a preset time to transfer the full list of files in the directory through ftp.

There is a setting in Filezilla to disable the timeout. Menu Edit ā†’ Settingsā€¦, then under Connection the first setting is ā€œTimeout in secondsā€. Enter zero here and Filezilla will wait as long as it takes to read the full list of files. Just be patient and your files will show.

This is only to be used as a workaround in case the files are already there. Once again I strongly suggest you update the creation of files on the SD card using the YEAR/MONTH suggestion. The proposed WagoAppString.Format1() function is one way to go about this.