|
| 1 | +# OTTER: Orchestratable Transcription, Timing and Expression Runtime |
| 2 | + |
| 3 | +An analysis layer for [synthrt](https://github.com/diffscope/synthrt) that defines contracts and interpreters for inference modules producing structured annotations of audio. |
| 4 | + |
| 5 | +synthrt defines singers and inferences, and [wolf](https://github.com/diffscope/wolf) adds languages. Neither library determines the sung content of a recording, that is, the notes, their pitches and their timing. otter provides this layer. A model that reads audio and returns a structured annotation of it is an inference module whose interface is one of the otter contracts, and a package declares it in the same way as any other module. otter adds no contribution category; the analyzers belong to the built-in synthrt `inference` category, as the wolf G2P and S2P modules do. |
| 6 | + |
| 7 | +otter contains no executable. It is a library that an editor or a tool embeds alongside synthrt. |
| 8 | + |
| 9 | +## The Analysis Module |
| 10 | + |
| 11 | +A package listing an analyzer in its `desc.json`: |
| 12 | + |
| 13 | +```json |
| 14 | +{ |
| 15 | + "contributions": { |
| 16 | + "inference": [ |
| 17 | + { "id": "f0", "path": "./inferences/f0/inference.json" } |
| 18 | + ] |
| 19 | + } |
| 20 | +} |
| 21 | +``` |
| 22 | + |
| 23 | +Each entry is a path to an analysis manifest: |
| 24 | + |
| 25 | +```json |
| 26 | +{ |
| 27 | + "interface": "org.openvpi.otter.inference.F0", |
| 28 | + "level": 1, |
| 29 | + "variant": "rmvpe", |
| 30 | + "name": "RMVPE", |
| 31 | + "exports": { |
| 32 | + "sampleRate": 16000, |
| 33 | + "channelCount": 1, |
| 34 | + "interval": 0.01, |
| 35 | + "maxSegmentDuration": 60, |
| 36 | + "knobs": { |
| 37 | + "voicingThreshold": { "minimum": 0.0, "maximum": 1.0, "default": 0.03 }, |
| 38 | + "interpolateUnvoiced": { "default": true } |
| 39 | + } |
| 40 | + }, |
| 41 | + "configuration": { |
| 42 | + "model": "../../rmvpe.onnx" |
| 43 | + } |
| 44 | +} |
| 45 | +``` |
| 46 | + |
| 47 | +The two blocks follow the division defined by spec 2.4. `exports` belongs to the contract: it declares the audio format that a host must prepare and the knobs that the module supports, has the same form for every variant of the interface, and is published as JSON Schema under [docs/schemas](docs/schemas). `configuration` belongs to the variant: it specifies the model paths and parameters, and no host reads it. A variant checks the two blocks against each other at load. A package that declares a sample rate its model does not support, or an alignment path without the model that performs it, is therefore rejected before any analysis runs. |
| 48 | + |
| 49 | +Level 1 defines three contracts: |
| 50 | + |
| 51 | +| `interface` | Produces | |
| 52 | +| :-- | :-- | |
| 53 | +| `org.openvpi.otter.inference.F0` | A fundamental frequency curve and a voiced flag per frame | |
| 54 | +| `org.openvpi.otter.inference.Note` | Note intervals, optionally conditioned on notes already known | |
| 55 | +| `org.openvpi.otter.inference.Align` | Word and phoneme intervals, for lyrics the caller already knows | |
| 56 | + |
| 57 | +The name `org.openvpi.otter.inference.Transcribe` is reserved for a future text transcription contract. |
| 58 | + |
| 59 | +## Host responsibilities |
| 60 | + |
| 61 | +otter contains no audio processing code. It does not decode, resample, slice or read files, and it depends on neither ffmpeg nor libsndfile. An executive accepts one contiguous span of float PCM and the start time of that span. |
| 62 | + |
| 63 | +The host is therefore responsible for three tasks: |
| 64 | + |
| 65 | ++ **Resampling.** Each module declares the sample rate it requires. The rates differ between algorithms (RMVPE requires 16 kHz; the GAME note model and the HFA aligner require 44.1 kHz), so the host reads the declaration instead of assuming a rate. An executive rejects audio at any other rate and never resamples implicitly, because implicit resampling would change the result without notice. Channels are the exception: an executive averages them down, because the downmix has only one possible result. |
| 66 | ++ **Slicing.** The host divides the audio into spans and makes one call per span, and no span exceeds the `maxSegmentDuration` that the module declares. |
| 67 | ++ **Musical time.** Every time value that otter produces is an absolute number of seconds. Ticks, tempo and tempo curves belong to the host timeline. |
| 68 | + |
| 69 | +## Analyzer creation |
| 70 | + |
| 71 | +A host passes the otter plugin directory, `${OTTER_PLUGINS_DIR}/inferenceinterpreters`, together with the directories of dsinfer and wolf to `SynthUnit::setPluginPaths` for the inference category. The call replaces the existing list, and all directories must therefore be passed in a single call. An analyzer does not require an importing module: the host opens the package, obtains the module, and creates an analyzer through the contract header. |
| 72 | + |
| 73 | +```cpp |
| 74 | +auto spec = package.contribution(srt::InferenceCategory::NAME, "f0"); |
| 75 | +auto analyzer = otter::Api::F0::L1::createAnalyzer(*spec->as<srt::InferenceSpec>()); |
| 76 | +``` |
| 77 | + |
| 78 | +`createAnalyzer` builds the import and runtime options of the contract from the variant of the module and calls `srt::InferenceSpec::createInference`, which rejects a module of another contract before creating an inference. `createAnalyzer` then verifies that the returned executive implements the contract and rejects an executive that does not with `otter::AnalysisError::Internal`. Another module can also import an analyzer through an ordinary `imports` entry with empty `options`. |
| 79 | + |
| 80 | +## Versioning and ABI |
| 81 | + |
| 82 | +**otter provides no ABI stability guarantee before 1.0.** Nearly every type that a host uses is a |
| 83 | +value type in a public header: every payload under `Api/` is one, and the vtable of a contract |
| 84 | +interface changes when a virtual function is added. Mixing objects compiled against two versions |
| 85 | +of these headers is undefined behaviour, not a link error, and therefore produces no diagnostic. |
| 86 | +Rebuild the host whenever the otter version changes. |
| 87 | + |
| 88 | +The headers under `otter/Support` are not part of the API. They declare the manifest readers |
| 89 | +shared by the library and its plugins, are not installed, and are compiled into a private static |
| 90 | +library instead of being exported from the shared library. |
| 91 | + |
| 92 | +## Requirements |
| 93 | + |
| 94 | +CMake 3.19 or later, and a checkout of the vcpkg overlay submodule: |
| 95 | + |
| 96 | +```sh |
| 97 | +git submodule update --init |
| 98 | +``` |
| 99 | + |
| 100 | +Every dependency, synthrt included, comes from vcpkg. Ports resolve through two overlays, searched |
| 101 | +in order: `scripts/vcpkg-ports` in this repository, then the shared `scripts/vcpkg` submodule. |
| 102 | + |
| 103 | ++ [synthrt](https://github.com/diffscope/synthrt), through the in-repo `synthrt-main` port, which |
| 104 | + pins the `onnxruntime-builds-uptake` branch of the main line |
| 105 | ++ [stdcorelib](https://github.com/SineStriker/stdcorelib) |
| 106 | ++ [qmsetup](https://github.com/stdware/qmsetup) |
| 107 | + |
| 108 | +Boost.Test is needed only to build the tests. |
| 109 | + |
| 110 | +All three shipped providers run ONNX models and are built only if dsinfer is present. The `onnx` |
| 111 | +feature (`--x-feature=onnx`) adds `synthrt-main[onnx]`, which installs the dsinfer CMake package |
| 112 | +beside the synthrt package so that `find_package(dsinfer)` locates it. Without this feature the |
| 113 | +library still builds and packages still load; only analyzer execution is unavailable. synthrt |
| 114 | +behaves in the same way without its own driver. |
| 115 | + |
| 116 | +## Build |
| 117 | + |
| 118 | +```sh |
| 119 | +vcpkg install --x-manifest-root=scripts/vcpkg-manifest --x-install-root=build/vcpkg_installed \ |
| 120 | + --x-feature=onnx --x-feature=tests |
| 121 | + |
| 122 | +cmake -B build/cmake -G Ninja \ |
| 123 | + -DCMAKE_TOOLCHAIN_FILE=<vcpkg>/scripts/buildsystems/vcpkg.cmake \ |
| 124 | + -DVCPKG_INSTALLED_DIR=<install root passed above> \ |
| 125 | + -DVCPKG_MANIFEST_MODE=OFF \ |
| 126 | + -DCMAKE_BUILD_TYPE=Release \ |
| 127 | + -DOTTER_BUILD_TESTS=ON |
| 128 | + |
| 129 | +cmake --build build/cmake |
| 130 | +ctest --test-dir build/cmake |
| 131 | +``` |
| 132 | + |
| 133 | +The model tests (`test_Rmvpe`, `test_Game` and `test_Hfa`) require fixtures, which are generated |
| 134 | +instead of committed. If the build finds a Python interpreter with the `onnx` and `numpy` modules, |
| 135 | +it generates the fixtures into the build tree as a dependency of those tests; otherwise the three |
| 136 | +tests are registered as disabled and ctest reports them as such. The script can also be run |
| 137 | +manually: |
| 138 | + |
| 139 | +```sh |
| 140 | +python3 scripts/make-model-fixtures.py --output build/fixtures |
| 141 | +``` |
| 142 | + |
| 143 | +The fixtures are real ONNX graphs with the real signatures and arithmetic in place of trained |
| 144 | +weights. They verify that the providers conform to the contract; they do not verify numerical |
| 145 | +accuracy, which requires the trained models. A test that finds no fixture, no driver plugin or no |
| 146 | +ONNX Runtime at run time exits with the status that ctest reports as skipped, never as passed. |
| 147 | + |
| 148 | +## Model packages |
| 149 | + |
| 150 | +The declarations of the three shipped variants are kept under [packages](packages) without their |
| 151 | +model files. Release `models-v0.3.0.0` provides the packages `otter-rmvpe`, `otter-game` and |
| 152 | +`otter-hfa`, version 0.2.0.0, together with a `manifest.json`; `scripts/make-package.py` |
| 153 | +assembles each package from its declaration and the model files. [packages/README.md](packages/README.md) |
| 154 | +describes the assembly. |
| 155 | + |
| 156 | +Every package must pass the lint before publication: |
| 157 | + |
| 158 | +```sh |
| 159 | +python3 scripts/check-declarations.py <package directory>... |
| 160 | +``` |
| 161 | + |
| 162 | +The declarations under `packages/` are kept without their models. `--declarations-only` checks |
| 163 | +the declarations without opening the files they refer to; CI checks `packages/` in this mode. |
| 164 | + |
| 165 | +## Usage |
| 166 | + |
| 167 | +```cmake |
| 168 | +find_package(otter CONFIG REQUIRED) |
| 169 | +target_link_libraries(example otter::otter) |
| 170 | +``` |
| 171 | + |
| 172 | +The analyzers are plugins located through the inference search path. A host therefore links otter |
| 173 | +only for the contract headers and the library functions that they call; linking performs no |
| 174 | +registration. |
| 175 | + |
| 176 | +## Documentation |
| 177 | + |
| 178 | ++ [otter-design.md](docs/otter-design.md): the design, including the layering, the contract surfaces, the host responsibilities, the decision ledger, the audits and the implementation milestones. |
| 179 | ++ [lite-integration.md](docs/lite-integration.md): the integration of otter into ds-editor-lite and the functions that ds-editor-lite no longer implements itself. |
| 180 | ++ [plans/hfa-align.md](docs/plans/hfa-align.md): the implementation plan and record of the Align contract and the `hfa` variant. |
| 181 | ++ [packages/README.md](packages/README.md): the declarations of the shipped variants and the package assembly. |
0 commit comments