Persistent Circular Buffers: Implementing FIFO/LIFO Logic on Top of SPIFFS

In the world of IoT, a stable network connection is often a luxury, not a guarantee. Whether it’s a cold-chain sensor traveling through a rural highway or an industrial logger in a basement with thick concrete walls, “Offline-First” is a survival requirement, not just a design choice. While RAM-based circular buffers are the standard for high-speed data handling, they are inherently volatile. If the power cycles during a network outage, your data is gone.

Standard file logging, on the other hand, often lacks the sophisticated pointer management needed for long-running systems. Simply appending to a file leads to storage exhaustion, and scanning a massive log file for the “oldest” entry after a reboot is a CPU-intensive operation that slows down system startup. This article explores an architectural alternative: implementing a Persistent Circular Buffer (PCB) directly on a filesystem like SPIFFS or LittleFS.

The Architectural Challenge: Why SPIFFS?

SPIFFS (SPI Flash File System) is designed for wear-leveling on small SPI flash chips. It doesn't use a traditional directory table, which makes it resilient against power loss during writes but also introduces specific constraints—notably, its linear performance characteristics and fragmentation overhead. To build a circular buffer on top of it, we need to simulate circularity on a linear file space without incurring the cost of constant file re-scans.

‍1. The Header-Driven Persistence

The core of our implementation is a fixed-width header at the very beginning of the file. This 32-byte string acts as the “Source of Truth” for the buffer's state. By using a fixed-width format (%7d,%7d,%7d,%7d,%d\n), we ensure that we can overwrite the header in-place at the exact byte offset 0 without shifting the rest of the file content.

Example Header Structure:

[Stream Init],[Stream Finish],[Data Count],[Max Size],[Mode]
"1024,         2048,           15,          8192,       0\n"

‍

This header stores:

- stream_init (Read Pointer): The byte offset where the oldest valid data begins.

- stream_finish (Write Pointer): The byte offset where the next data entry should be written.

- data_count: The number of discrete entries currently stored.

- max_file_size: The soft limit for the buffer to prevent partition overflow.

- mode: Defines the overflow behavior (FIFO vs. Overwrite-LIFO).

On boot, the system only needs to read these 32 bytes to know exactly where to start reading and where to continue writing, providing instantaneous recovery regardless of file size.

2. Implementing Circularity: The "Wrap-Around" Problem

In a RAM buffer, we use the modulo operator. In a filesystem, we must manually manage file pointers using fseek() and ftell(). When a "Push" operation reaches the max_file_size, we have two choices based on the mode:

FIFO (Queue Mode):

In CIRCULAR_BUFFER_MODE_QUEUE, the buffer returns an error (ESP_ERR_NO_MEM) if full. This is crucial for applications where data integrity is paramount and old data cannot be discarded under any circumstances.

Overwrite (Stack Mode):

In CIRCULAR_BUFFER_MODE_STACK, the logic is more complex. If there's no space between the write pointer (stream_f) and the read pointer (stream_i), we must “consume” the oldest entry to make room. This involves advancing stream_i to the next newline character (\n). This simple “search for next newline” logic allows us to effectively delete the oldest record and maintain circularity.

Logic for advancing the read pointer in Overwrite mode:

fseek(f, stream_i, SEEK_SET);
int aux_c;
while ((aux_c = fgetc(f)) != '\n' && aux_c != EOF) {
    stream_i++;
}
if (aux_c == '\n') {
    stream_i++; // Point to start of the next entry
    data_count--;
}

‍

3. Concurrency and Recursive Semaphores

SPIFFS is not inherently thread-safe across multiple file descriptors for the same file. In a FreeRTOS environment where multiple tasks (e.g., GPS, telemetry, and system logs) might attempt to write simultaneously, protection is mandatory.

We chose a recursive semaphore (spiffsGatekeeper) over a standard mutex. Standard mutexes can lead to deadlocks if a function calling a PCB operation is already holding a lock on the filesystem for another purpose. The recursive nature allows the same task to re-acquire the lock, which is vital when complex logging routines call nested filesystem utility functions. A 5000ms timeout ensures the system handles worst-case flash write latencies without permanent deadlocks.

if (!(xSemaphoreTakeRecursive(spiffsGatekeeper, 5000 / portTICK_PERIOD_MS))) {
    return ESP_FAIL; // Handle concurrency timeout
}
// ... Perform Atomic File Operations ...
xSemaphoreGiveRecursive(spiffsGatekeeper);

‍

4. The Truncation Debate: ftruncate vs. truncate

A critical decision in this implementation was how to handle file size management when the buffer wraps around or is emptied. There was a technical debate regarding the use of ftruncate(fd, size) versus truncate(path, size).

In our production code, we utilized ftruncate() during the “Push” wrap-around. Since the file is already open, ftruncate() is more efficient as it operates directly on the file descriptor, avoiding the overhead of re-resolving the file path through the VFS (Virtual File System) layer. This is crucial for maintaining atomicity during the milliseconds where the flash is being modified.

However, we used truncate() in the “Pop” operation when the data_count reaches zero. Since the file is closed at that point, truncate() allows for a clean “hard reset” of the file size to the header size plus one byte. This practice helps SPIFFS manage its internal block allocation and reduces the time required for subsequent fseek() operations, which can degrade as file fragmentation increases.

5. Edge Case: Power Failure During Header Update

The most vulnerable moment for any PCB is the header update. If power fails while writing the 32-byte header, the system could reboot to a corrupted state. To mitigate this, our implementation relies on the fact that SPIFFS writes are page-aligned. Since our header is exactly 32 bytes (well within a standard 256-byte page), the underlying flash controller is more likely to complete the write atomically or not at all. For even higher reliability, one could implement a “shadow header” or a simple CRC check at the end of the header string to validate its integrity upon boot.

Conclusion: RAM Savings vs. Flash Wear

The Persistent Circular Buffer is a trade-off. By moving the buffer to Flash, we save precious heap RAM—often the most constrained resource in microcontrollers like the ESP32. However, we introduce flash wear. By using fixed-width headers and minimal seeks, we mitigate this wear, but it remains a “heavy” operation compared to RAM.

For engineers building remote industrial loggers or critical fleet management systems, the peace of mind that comes from knowing every telemetry point is safely on non-volatile storage far outweighs the millisecond-scale latency of a flash write. This blueprint provides a deterministic, fault-tolerant path for data persistence in the most challenging IoT environments.