Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ All notable changes to cellmap-flow. The format follows [Keep a Changelog](https

### Changed
- **Help text shows one line, and the rest under an "i" beside it.** The panels' help ran to paragraphs: every hint now keeps one short line in view and the rest of what it said in a `<details class="more">` after it, opened by a click or a tap on its "i", with no script; a mouse resting on the "i" is shown the rest as a tooltip too (`css/dark.css`; `lib/dom.js` `hint()` and `setHint()` build the same for text a script writes). The Finetune, Models and Review tabs, the model advice, and the browser's Save predictions tab, connect panel, folder prompt and link notices, in both modes. A consent box keeps all it agrees to in view.
- **The dashboard's tabs stay at the top of its column as it scrolls**, and help in a `<div>` is the grey the rest is: it took Bootstrap's own form-text colour, dark on the dark page. Placeholders are a little lighter.
- **The dashboard's tabs stay at the top of its column as it scrolls**, and so do the open tab's own bars below them: the Processing tab's Apply bar and its Input normalization / Postprocessing tabs, and the Finetune tab's Annotation Crops / Training (the Apply bar slid under the tabs); help in a `<div>` is the grey the rest is: it took Bootstrap's own form-text colour, dark on the dark page. Placeholders are a little lighter.
- **A model's output activation is read from its network, not guessed from noise.** Whether a model's output needs a sigmoid (the start's sigmoid, *Use this model*'s postprocessing and threshold, the ranking's cut-offs, *Seed from Prediction*, the model advice) came from one pass on `randn * 100` at its server's start, its output range bucketed into [0, 1], [−1, 1] or unbounded; `cellmap/jrc_mus-livers_16nm_to_8nm_peroxisome` gave −0.9595 at every voxel, so it was taken for a tanh and given no sigmoid, though its network ends in a convolution, as its sibling's does. Now the inference server reads the last value-changing op of the network it runs, past reshapes, crops and casts (`portable/activation.py`; TorchScript's inlined graph, torch.export, a torch.fx trace, an ONNX file's wire format without the onnx package: `inference/graph.py`, `configs/onnx.py read_graph`): a Sigmoid or Softmax is [0, 1], a Tanh [−1, 1], a convolution or other linear layer logits; the probe decides only when the graph does not say, and an output that is one value says nothing (`classify_output_range` gives None). The browser reads the ONNX file it serves the same way, through the same rule, and the manifest's text only when the graph does not say. A training job the dashboard gave no class (a YAML's, the command line's) reads it the same way, the graph first and the probe after (`finetune/trainer/supervised.py output_class_of`), and a flat probe there reads as logits. `model_info`, the model advice and the ranking's row say where the class came from (`output_class_from`: graph, probe, or in the browser manifest; `output_op`), and the advice says "its network ends in a Sigmoid" or that the class is a guess.
- **Every finetune eases its learning rate down over the run, and the rate is typed.** The server's trainer and the browser's three (the last layer, and the whole network in 2D and in 3D) stepped at the form's rate from the first step to the last. A run now starts at it and eases down along half a cosine to a tenth of it at its last step (`LEARNING_RATE_FLOOR`, 0.1, in `finetune/options.py`, which the browser reads from `web/src/generated/python-data.json`; `trainer/loop.py learning_rate_factor` and its twin `web/src/finetune/schedule.ts learningRateAt`): big steps while the model is far from the paint, small ones once it is close, which is what a LICONN head trained at 1e-2 and then restarted at a smaller rate did by hand. The server counts a run's steps as its epochs times its optimizer steps an epoch, gradient accumulation included, and a step past that count (an OOM that halves the batch mid-run) stays at the floor; a restart builds a new trainer, and so starts the schedule again. Each run's start says it (*Learning rate: 1e-4, cosine to 1e-5 over 40 steps*), and TensorBoard's `train/lr` is the rate each step took. The browser's whole network steps an Adam of its own (`web/src/finetune/full/adam.ts`, TF.js's arithmetic), because TF.js's optimizer keeps the rate it was made with; the 3D runtime's `step` takes each step's rate, and `buildNet3d` no longer takes one. The training form's *Learning Rate* fields (the server's, the browser's last layer's and its whole network's) take any number above 0, `1e-4` or `0.0003`, with their presets beside them, and keep what was typed with the rest of the form; an empty, zero, negative or non-numeric rate is refused in the form, saying why.
- **Every finetune eases its learning rate down over the run (a box keeps it constant), and the rate is typed.** The server's trainer and the browser's three (the last layer, and the whole network in 2D and in 3D) stepped at the form's rate from the first step to the last. A run now starts at it and eases down along half a cosine to a tenth of it at its last step (`LEARNING_RATE_FLOOR`, 0.1, in `finetune/options.py`, which the browser reads from `web/src/generated/python-data.json`; `trainer/loop.py learning_rate_factor` and its twin `web/src/finetune/schedule.ts learningRateAt`): big steps while the model is far from the paint, small ones once it is close, which is what a LICONN head trained at 1e-2 and then restarted at a smaller rate did by hand. The server counts a run's steps as its epochs times its optimizer steps an epoch, gradient accumulation included, and a step past that count (an OOM that halves the batch mid-run) stays at the floor; a restart builds a new trainer, and so starts the schedule again. Each *Learning Rate* (the server's, the browser's last layer's and its whole network's) has an *Ease down to a tenth over the run* box beside it, ticked by default, saved and restored with the form and sent with submit and restart; unticked, every step of the run takes the form's rate, as before. It is the setting `learning_rate_schedule`, `cosine` or `constant`, in `TrainingOptions` like every other (the submit and restart bodies, `--learning-rate-schedule`, a restart's parameters, the browser's `FinetuneOptions.learningRateSchedule`). Each run's start says which (*Learning rate: 1e-4, cosine to 1e-5 over 40 steps*, or *1e-4, constant over 40 steps*), and TensorBoard's `train/lr` is the rate each step took. The browser's whole network steps an Adam of its own (`web/src/finetune/full/adam.ts`, TF.js's arithmetic), because TF.js's optimizer keeps the rate it was made with; the 3D runtime's `step` takes each step's rate, and `buildNet3d` no longer takes one. The training form's *Learning Rate* fields (the server's, the browser's last layer's and its whole network's) take any number above 0, `1e-4` or `0.0003`, with their presets beside them, and keep what was typed with the rest of the form; an empty, zero, negative or non-numeric rate is refused in the form, saying why.
Expand Down
20 changes: 20 additions & 0 deletions cellmap_flow/dashboard/static/css/dark.css
Original file line number Diff line number Diff line change
Expand Up @@ -1038,3 +1038,23 @@ details.more > .more-body {
z-index: 20;
background: var(--bg-base);
}

/* Under them, the open tab's own bars stay too: the Processing tab's Apply
bar and its Input normalization / Postprocessing tabs, the Finetune tab's
Annotation Crops / Training. Each stops below the bars above it, whose
heights (they wrap with the column's width) main.js keeps in --tabs-h and
--apply-h. */
#applyBar,
#processingTabs,
#finetuneTabs {
position: sticky;
z-index: 15;
background: var(--bg-base);
}
#applyBar,
#finetuneTabs {
top: var(--tabs-h, 0px);
}
#processingTabs {
top: calc(var(--tabs-h, 0px) + var(--apply-h, 0px));
}
16 changes: 16 additions & 0 deletions cellmap_flow/dashboard/static/js/dashboard/main.js
Original file line number Diff line number Diff line change
Expand Up @@ -291,6 +291,22 @@ initServerConfigModal();
initReviewTab();
// A hint's "i" shows the rest of its hint as a tooltip too (lib/dom.js).
tooltipHints();
// The bars that stay at the top of the column stack: each stops below the
// ones above it (css/dark.css), whose heights change as the column's width
// wraps them, and the Apply bar's as its notices come and go.
function stackStickyBars() {
const style = document.documentElement.style;
const tabs = document.getElementById("myTab");
const apply = document.getElementById("applyBar");
const measure = () => {
style.setProperty("--tabs-h", `${tabs.offsetHeight}px`);
style.setProperty("--apply-h", `${apply ? apply.offsetHeight : 0}px`);
};
const observer = new ResizeObserver(measure);
[tabs, apply].forEach((el) => el && observer.observe(el));
measure();
}
stackStickyBars();
// The Finetune tab starts last, so its first requests go after the other
// tabs' (the Models tab asks for the GPU queues only once the server config
// has answered): on DOMContentLoaded while the page is still being parsed,
Expand Down
2 changes: 1 addition & 1 deletion cellmap_flow/dashboard/templates/_processing_tab.html
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@
<div
class="apply-bar py-2 mb-2"
id="applyBar"
style="position: sticky; top: 0; z-index: 10; background: var(--bg-base); border-bottom: 1px solid var(--border);"
style="background: var(--bg-base); border-bottom: 1px solid var(--border);"
>
<div class="d-flex align-items-center gap-2">
<h6 class="mb-0 me-auto">Steps for all models</h6>
Expand Down
Loading