Skip to content

README.md

Latest commit

 

History

History
282 lines (232 loc) · 16.7 KB

README.md

File metadata and controls

282 lines (232 loc) · 16.7 KB
Raw

Repository documentation

Start here when working on _core, the shared firmware and tooling repository for Zeptocore, Zeptoboard, Ectocore, and Ezeptocore. Run the commands below from the repository root. Source and artifact paths written in the development notes also refer to the repository root; Markdown links are relative to each document.

The short orientation note is readme-starthere.md.

SDK upgrade and validation: Pico SDK 2.3.1 and connected hardware validation.

Current performance controls: Zeptocore A/B knob mappings.

Reading order

  1. Use the repository map and build instructions below to locate the relevant code.
  2. Read seek-map design for allocation maps, persistence, memory limits, and filesystem ownership.
  3. Read implementation status and hardware results for the original validation record, together with the subsequent review notes below.
  4. Use diagnostics when investigating runtime behavior, then follow the topic-specific documents in the index.
  5. Consult the implementation brief for the original requirements and acceptance criteria. It is a historical implementation brief, including session-specific hardware and completion statements.

Repository map

Location Role
main.c, lib/includes.h, lib/globals.h Firmware startup, module integration, and shared state.
lib/ Device wrappers, controls, audio rendering, DSP, media lifecycle, and drivers. Many implementations live in headers.
lib/audio_callback.h, lib/realtime_stretch.h Normal playback and time-stretch rendering.
lib/audio_seek_map.c, lib/audio_media.h, lib/audio_media_control.h, lib/audio_media_owner.c Persistent map/cache management, playback file lifecycle, foreground integration, and cross-core filesystem ownership.
lib/sdio/ Bundled FatFs and SD support; allocation validation and mapped seeking are implemented in ff.c.
lib/my_pico_audio/, lib/my_pico_audio_i2s/ Audio buffers, format conversion, I2S/DMA output, and restart queue integration.
core/ Go application and sample-preparation/web-server tooling.
dev/ Development utilities, audio/header generators, and device tools.
scripts/ Host-side build comparisons, diagnostics, hardware workloads, and capture analysis.
test/ Native firmware tests and Python protocol tests. Older standalone module tests also live in lib/test/.
Makefile, CMakeLists.txt, lib/cmake *_compile_definitions*.cmake Build targets, feature switches, and device/buffer configurations.
schematics/ Hardware schematics.
docs/ Development notes and legacy Hugo site sources/assets. The site's public/ output is generated; top-level development notes are ordinary Markdown.

Build and test

The standard Zeptocore build is:

make zeptocore

It produces zeptocore.uf2 and build/_core.elf, using the 441-frame device configuration. make zeptocore_256 selects the supported 256-frame configuration. Other device targets are listed in the Makefile. These targets share build/ and lib/cmake/target_compile_definitions.cmake; use separate CMake build directories and CORE_COMPILE_DEFINITIONS for simultaneous configurations, as shown in the diagnostics build instructions.

The Makefile downloads Pico SDK 2.3.1 and Pico Extras sdk-2.3.1, creates a stable Python 3.14 .venv with uv, and generates DSP/audio headers when absent. Host tools include CMake, Make, ARM GCC with newlib, Git, uv, clang-format, Go, and SoX. The firmware CI workflow documents its Linux setup, including the TinyUSB patch used by CI. Building firmware does not flash it.

To set up the Python tools directly:

uv self update
uv venv .venv --python 3.14.8
uv pip install --python .venv/bin/python -r requirements.txt
source .venv/bin/activate

The pinned dependencies also support Python 3.11 for the firmware CI jobs. Use a current uv so its download catalog includes stable Python 3.14 builds. If uv was installed through a package manager, update it through that manager instead of uv self update. The exact patch version excludes early 3.14 alpha builds that a bare -p 3.14 can reuse; those builds cannot load the current NumPy wheels. Check .venv/bin/python -VV for the interpreter version. Recreate an environment using an alpha, beta, or release candidate with uv venv .venv --python 3.14.8 --clear, then install the requirements again. Reinstalling packages alone does not replace its interpreter. For an existing stable Python 3.14 environment, rerun the install command to update the packages.

Make checks existing SDK/extras checkouts against the pinned tags and initialized submodules. It reports a mismatch without overwriting local changes. For an older installation, move the old SDK, extras, and build directories into a uniquely named backup under artifacts/, then run make zeptocore to fetch the pinned dependencies and configure a fresh build. Direct CMake builds can select other SDK/extras paths for comparisons; use a separate build directory for each pair.

Run the relevant native suites after building the generated headers:

.venv/bin/python test/fatfs_seek/run.py
.venv/bin/python test/seek_diagnostics/run.py
.venv/bin/python test/audio_source/run.py
.venv/bin/python test/dsp_loops/run.py
.venv/bin/python test/midi/run.py
.venv/bin/python test/rack/run.py --sanitize address,undefined

These runners invoke host cc/c++ with sanitizers. The FatFs runner uses GNU linker --wrap=calloc, which Apple's linker does not support; the diagnostics runner also encounters Mach-O section/Clang compatibility issues on macOS. A successful macOS firmware build does not establish that these native suites pass unchanged. The earlier review used temporary host-portability adaptations for some tests; those adaptations are not part of the repository.

On 2026-09-16, make zeptocore passed on macOS at c5f884b (PR #830 merged), with three conversion warnings in lib/mcp23017/mcp23017_lib.c. No hardware flashing or playback validation was performed during that build.

Automatic server updates and cleanup

On each Linux server checkout, add this line to root's crontab (sudo crontab -e):

* * * * * /www/ezeptocore/dev/auto_update.py

Replace /www/ezeptocore with that checkout's absolute path. The executable auto_update.py locates the repository from its own path, so cron needs no cd or arguments. The repository folder name selects the service: ezeptocore restarts zns.ezeptocore.service, ectocore restarts zns.ectocore.service, and zeptocore restarts zns.zeptocore.service.

For a persistent log, use this entry instead and include the log in the server's normal log rotation:

* * * * * /www/ezeptocore/dev/auto_update.py >> /var/log/ezeptocore-update.log 2>&1

The cron account needs noninteractive Git access, write access to the checkout, and permission to restart its existing systemd service. Install Python 3.8+, Git, Make, Go, Hugo, uv, and the server's existing build dependencies first. The cleanup utility requires Python 3.11+ and its inline click and rich dependencies, which uv manages. The script extends cron's PATH with standard system directories, /usr/local/go/bin, and the account's .local/bin, bin, go/bin, node/bin, and .cargo/bin. It does not load interactive shell configuration; custom tool locations must be included in cron's PATH.

Each invocation fetches the checked-out branch's configured upstream and fast-forwards to the fetched commit. Detached checkouts, missing upstreams, local commits ahead of upstream, divergent history, and conflicting local edits stop deployment with an error. Git preserves nonconflicting local edits; those edits remain part of the build. No reset or stash is performed. Git hooks are disabled for the script's commands so older post-merge hooks cannot trigger duplicate builds or restarts.

After an update, the script runs make core_server, including the Hugo docs build, then restarts the service and checks systemctl is-active. A failed build does not trigger a restart. Build and restart failures stay pending and retry on the next invocation, even if Git is already up to date. Deployment requires a successful fetch. The first invocation assumes an existing core_server already matches the current commit; a missing binary triggers a build. A successful immediate service check does not monitor later crashes or provide automatic rollback.

Cleanup runs on the first invocation and then once every 24 hours. It calls delete_old_folders.py with --delete --yes for:

Directory relative to the checkout Retention
drum_separation_model_output/modelo_final 30 days
storage 60 days

The existing utility deletes immediate, nonhidden child folders only when their newest file anywhere inside is older than the retention period. Empty folders, folders with no files, and files directly inside the target directory are left alone. Missing target directories are skipped. Both cleanup commands are attempted even if one fails, and a deployment error does not prevent cleanup. The daily timestamp records an attempt, including failed or interrupted scans, so another attempt waits 24 hours.

The lock and state live under the checkout's Git metadata directory, normally .git/auto-update/: lock, deployment.json, and cleanup-attempt.json. Deployment records the last successful commit and whether work remains pending; state files are replaced atomically. Overlapping invocations exit quietly. Unchanged checks are quiet; actions and command output have timestamps, and command failures return a nonzero exit status. The script never installs cron entries itself. Use dev/auto_update.py --help to read usage without running it.

Run the isolated regression tests with:

python3 -m unittest discover -s test/auto_update -v

Tests use temporary Git repositories and stub build/service commands. If uv is available, they also exercise the real cleanup utility on temporary data.

Current seek/audio behavior

The fast-seeking work was merged in PR #830. Maps describe contiguous runs of filesystem clusters; they do not cache audio data. Startup validates saved allocation chains, rebuilds missing/changed maps, and persists recoverable index generations. Every open must attach a current-mount map because FatFs clears FIL.cltbl when opening a file.

  • Limits are 256 indexed physical WAVs, two RAM map entries, and 64 DWORDs (31 contiguous runs) per map. Excess files or fragmentation use ordinary seeks.
  • File duration alone does not determine map size. Allocation validation still walks filesystem metadata on each mount, so long files can increase startup work.
  • The live file's map is pinned. A runtime cache miss uses ordinary seeking and queues work until playback is stopped/muted, effect tails are disabled, bass voices are idle, and filesystem ownership has been acknowledged.
  • The audio core owns filesystem access during playback. Foreground filesystem work uses the shared ownership handoff; optional mapping does not justify an uncoordinated SD operation during rendering.
  • Normal builds enable seek-map attachment. Diagnostics, detailed timing, test controls/fixtures, next-file preparation, and extra output buffering default off.
  • Clock restart correction and wake default on for the Ectocore/Ezeptocore path; immediate wake is limited to the 256-frame core-1 worker configuration.
  • Variable splicing uses each selected slice's duration on the 192 PPQN timer, including between external 2 PPQN clock edges. Clock input supplies tempo and restart synchronization; the fixed splice grid does not determine variable slice timing. TAP+MODE preserves its nearest-clock-edge reset behavior. The shared-engine tests cover uneven slices, source formats, resets and clock loss.
  • The same series also fixes the pitch/retrigger source-buffer overflow and optimizes delay/filter processing. See the documents below for each change.

The constants and feature flags are defined in audio_seek_map.h, CMakeLists.txt, and audio_restart.h.

Subsequent review notes

The following findings from the follow-up review remain present at c5f884b. They qualify the original documents' completion claims; no fixes were applied during this documentation consolidation.

  • In audio_seek_map.c, identity() hashes fs->bitbase for every filesystem, although FatFs initializes that field for exFAT. The native FAT remount test reproduced unnecessary map rebuilds/index writes when the unused field differed. Zero-initialized startup storage can mask this.
  • In ff.c, ff_clmt_inspect() checks obj.n_frag and obj.stat without restricting the exFAT allocation-state checks to exFAT. The native benchmark reproduced rejection of an otherwise valid FAT file.
  • The optional 3784-byte map arena is allocated after reverb initialization but before later mandatory DSP/control allocations. The review identified missing heap-reserve protection as a risk; a resulting hardware failure was not reproduced.

Temporary copies with the two FAT/exFAT guards passed the FatFs suite after the host-portability adaptations mentioned above. That result does not mean the unmodified checkout passed every native suite or that hardware was retested.

Development document index

All 14 Markdown documents introduced by PR #830 are collected directly in this folder, including the relocated implementation brief. Keep detailed evidence in the topic documents and use this index for orientation.

Document Purpose
MIDI slice triggering Channel 1 note mapping, transport/mute behavior, Ableton routing, and native tests.
Implementation brief Original scope, requirements, integration checklist, and acceptance/completion record.
Seek-map design Memory policy, durable index, allocation validation, filesystem ownership, and mapped execution.
Implementation status Original integration audit, correctness coverage, hardware evidence, and limitations.
Hardware results Zeptocore map-on/off comparisons, persistence checks, memory use, and supported buffer sizes.
Diagnostics usage Build/run instructions, SWD service, captures, metrics, and interpretation.
Diagnostics plan Original measurement protocol and instrumentation-overhead acceptance gate.
Ectocore hardware Device-specific validation and follow-up measurements.
Detailed timing Callback-stage profiling and remaining starvation cases.
Freeze fix Retrigger/pitch stack overflow, bounded source buffer, and playback-rate limits.
DSP optimization Exact fixed-point multiplication and delay/filter loop optimizations.
Clock latency Measurements before the clock restart correction.
Clock restart fix Phase correction, fade, immediate renderer wake, and measured results.
Next-file experiment Staged file preparation experiment; disabled by default.
Output-buffer experiment Additional buffering and its latency tradeoff; disabled by default.

Hardware capture and firmware paths under artifacts/seek/ are historical evidence references. artifacts/ is ignored by Git and may be absent in a fresh checkout. Verify the relevant build, device, and available evidence before reusing a measurement or assuming a previously described hardware setup.

Older references include the Ectocore quickstart and Zeptocore specification. Product guides, downloads, and licensing are linked from the root README.