Repository navigation
Status and Updates
Sanshiro Enomoto edited this page Jun 8, 2026
·
93 revisions
- Layout is a big to-do item for the future.
- Maybe a graphical design tool for the Canvas (schematic view) will be added.
- Basic structure has been made. More will be added.
- The analysis part of the
SlowPylibrary will be largely restructured. - Data transform (data functors) is at a design stage.
- Data transform, including resampling with bin-stats (min/max/std) and table-to-graph
- Log messages, Alarms
- Table row/cell attributes (colors, links, etc.)
- User management & user roles
- Flexible layout
- Settings in a side-panel (replace the dialogs)
- Graphical display widgets
- Dashboard Canvas design tool, built-in SVG editor
- Second vertical axis
- Color-coded graph points
- Sortable tables
- Preference settings, custom styles, home layout
- Directory-less project (DB entry, on-memory, .db file, etc.), project configuration from
config - WebDAV blob storage
- time-range table (run table, phase table, ...)
- Be careful how resampling affects the visualization:
- The time-series projection histograms accumulate the data points after resampling. This can be avoided by setting the resampling threshold to a very large value (i.e., disabling resampling).
- The time-series scatter plots become incorrect if two time-series data points are not aligned. If data points are not aligned, set the resampling threshold to zero (always apply resampling).
Major changes since 2507 Nooksack are:
- Plotting improvements:
- Single value panels
- Table view of time-series data
- Customized resampling, min/max envelopes
- Optional previous point fetching
- SlowPy data features:
- streaming of various data types, including Python dataclass, dict, immutables, etc., in addition to histograms and graphs
- streaming of Matplotlib figures, with extraction of data points and plot layouts
- writing to HDF5 files
- PubSub modules in SlowPy
- Supports NATS, MQTT, RabbitMQ, Redis Pubsub, and SlowMQ (SlowDash's built-in MQ)
- All in async, implement reconnecting
- DB reconnect and rescan
- More devices: Modbus, LabJack, Nanotech Motors, Audio Input (PC mic), etc.
- In slowpy/control, the following PubSub modules are now available:
- [Async]: AsyncNATS, AsyncMQTT, AsyncRabbitMQ, AsyncRedis pubsub, and AsyncSlowMQ
- [Non Async]: MQTT, RabbitMQ, Redis pubsub
- All PubSub modules implement reconnecting
- SlowMQ: SlowDash's Pubsub fabric using the SlowDash server as a broker and JSON on WebSockets for message exchange
- See
lib/slowpy/test/test-control-async-PUBSUB.pyfor examples
- With the "update" button in the Channel List panel, reconnecting to DB and updating the channel list is attempted.
- The DB contents can be updated without restarting SlowDash.
- With a prior point, the last value will always be shown (if exists) with the "last" line type.
- This feature is particularly useful if data recording intervals are large, e.g., for set points.
- Time-series plots now have options to set various parameters:
- Resampling buckets: the number of data points after resampling
- Reducer:
last,mean,median, etc.: the value that represents each resampling bucket - Envelope checkbox: controls whether the min-max envelope is calculated or not
- Threshold: the length of the display time range above which resampling is applied
- Note:
- Calculating the envelope can be slow depending on the setup (resampling in DB query will be disabled).
- Using too many resampling buckets (or setting the threshold too high) can slow down or crash the web browsers.
- A simple example to post data values from external programs through HTTP-POST
- [May 1, 2025] Moved to
ExampleProjects/Advanced/WebPost
- The Table element on the Layout can show the raw time-series values. Select
Value Tablefor the Data Type and then choose the channel.
- Pub/Sub to RabbitMQ, and RPC are implemented
- Both async and no-async versions
- publish example:
p=rabbitmq().topic_exchange(name).publish(routing_key); await p.aio_set(value) - subscribe example:
q=rabbitmq().topic_exchange(name).queue(qname, routing_key); value = await q.aio_get()
- Writes time-series data to an HDF5 file in a wide, row-append format
- Using NumPy compound dtypes (timestamp : channels)
- A figure object of Matplotlib can be directly publish()-ed in SlowPy
- A layout configuration (slowplot-XXX.json) will be dynamically created based on the figure layout (Axes)
- Data objects (histograms, plots, etc.) are extracted from the figure and publish()-ed.
- See example in ExampleProjects/Streaming/Matplotlib
- On plot panels, an envelope showing graph's min and max values can be drawn
- Currently available only for SlowPy Trend
- See example in ExampleProjects/Streaming/Trend
- New slowplot panel to display one scalar value
- To display the last value of a time-series of scalers, or to display a publish()-ed scalar value
-
export(value, name="...")can take not only a slowpy control node, but also a slowpy graph/histogram, Python dataclass, dict, and class instance -
publish(value, name="...")can take, in addition to the values export-table, Python immutables such as str and float -
ControlSystem.export()will replace the_export()callback -
ControlSystem.publish()will replace theNode.deliver()method
- The last point can be highlighted using a different marker type, size, or color.
Major changes since 2411 Skykomish are:
- Drastic performance improvements
- ASGI interface with new Web server engine (Slowlette)
- HTTP/2 with a Nginx reverse proxy tested and recommended
- Async I/O for PostgreSQL, MySQL, and Redis
- Data Streaming using WebSockets
- Dynamic generation (by user scripting) of plot layouts and HTML contents
- Fully-featured user HTML pages with embedded SlowDash JS objects
- Define
_get_html()in UserModule to generateconfig/html-XXX.htmlfile equivalent - Define
_get_layout()in UserModule to generateconfig/slowplot-XXX.jsonfile equivalent - Multiple generations of HTML and Layouts are possible.
- More in the documentation, UserModule/AdvancedTopics
- Blob data block can be stored outside a database (database stores a link (id) to blob)
- To write (SlowPy)
blob_storage = BlobStorage_File(basedir='data/blob', names=['photos', '%y-%m', '%y%m%d-%H%M%S-%Z', ext='.jpg')
...
while in_run_loop:
blob_id = blob_storage.write(photo_image)
datastore.append(blob_id, tag='photo')
This will create a directory photos/YY-MM/ and store files named YYMMDD-HHMMSS-TZ.jpg.
Other parameter substitutions include ${timestamp}, ${md5}, $md5[N:M], and ${uuid}.
- To read (SlowdashProject.yaml)
data_source:
obj_time_series:
...
blob_storage:
type: file
base_directory: data/blob
- Channel names (tag values and fields) are obtained through InfluxDB queries; however, InfluxDB defaults do not extend back to a long range.
- Parameters to specify data time range are added to the time_series configuration:
data_source:
url:influxdb://ORG:TOKEN@HOST:PORT/BUCKET
time_series:
schema: MEASUREMENT
tags:
scan_date: 2025-01-01
scan_length: 100d
The tag values and fields are scanned for the time range scan_date - scan_length to scan_date.
Default scan_date is now() and default scan_length is 10d.
- Nginx ExampleProjects/ReverseProxy/Nginx, Apache ExampleProjects/ReverseProxy/Apache
- Jupyter with a reverse proxy: ExampeProjects/Jupyter/NginxProxy
- Async access is used by default for PostgreSQL (by
asyncpg) and MySQL (byaiomysql) - To use non-async access, use
type: PostgreSQL_NoAsync(PostgreSQL bypsycopg2) ortype: MySQL_NoAsync(MySQL bymysql-connector).
- Files under the project's
userhtmlwill be available to browsers atapi/userhtml/{filename} - The HTML files under that can use all the HTML features, such as CSS and JavaScript (embedded and external).
- SlowDash's Web API and JavaScript library are accessible using a relative URL.
- See User HTML section for details
- Now user module is a "Slowlette" (SlowDash Web API server) module
- Slowlett Apps defined in user modules will be automatically integrated using the Slowlette aggregation mechanism
- See Slowlette documentation for details
- By placing
slowplot-XXX.pyunderconfig, it will be executed instead of readingslowplot-XXX.json. - In the Python code, define a function
async def _setup(app)and return a Python dict.
- In SlowTask, ControlNode variables can now
deliver()the values (sending the values to browsers via WebSocket)
- The old (non async) version was renamed to
datasource_Redis_NoAsync, being a fallback when the server is not ASGI.
- Now the setup process creates a dedicated venv for SlowDash, with installing ASGI server etc.
- See Project Setup section for setup procedure.
- Made
SlowAPImicro web framework (underslowdash/app/server/slowapi) to be the basis of all the SlowDash web services. - ASGI should provide faster responses. Also, the "real-time display" in to-do will be built on that.
- To use ASGI, install the
uvicornpython package (pip install uvicorn). Otherwise SlowDash will fall back to WSGI.
- ❗ The Jupyter config path in the
SlowdashProject.yamlis changed:- from
extension/name=jupytertoexport/type=jupyter
- from
-
main/web/slowdash_wsgi.pyimplements the WSGI interface - Run
SLOwDASH/utils/slowdash-setup-apache.py --interface=WSGIto do the setup for Apache - See Project Setup section for details.
Major changes since 2407 Snoqualmie are:
- More SlowPy Control components:
- HTTP, Telnet, VISA
- SlowDash as a control variable
- DataStore as a control variable
- LabJack
- SlowPy-HDL: HDL-like control logic in Python
- SlowTask auto-loading
- Jupyter integration
- MySQL support
- Watchdog (SlowDash "freeze" detection and auto-restarting)
- Add the following block in
SlowdashProject.yaml:
extension:
- name: jupyter
parameters:
url: JUPYTER_URL
token: JUPYTER_TOKEN
- To use in Docker Compose, see
ExampleProjects/Jupyter.
ControlSystem.visa(visa_address). ...-
scpi()can be appended
- New type of plugin, to extend SlowDash API
- implemented in
slowpy/control/control_DataStore.py
- SlowDog sends HTTP requests to the SlowDash server periodically, and if it does not receive a reply before timeout, the dog kills the server and restarts another one.
- Use
--slowdogoption toslowdashto enable this. This is already enabled for the Docker container.
- currently supports U12 Analog In/Out and Digital In/Out
- Verilog-style scripting for SlowPy/Control
- see documentation, SlowPy HDL section
- For controlling Slowdash servers from SlowPy scripts
-
ControlSystem.http(url).slowdash()to create a node - Implements
ping(),config(),channels(),data(channels,length=3600), andconfig_file(name)subnodes
- HTTP:
ControlSystem.http(url).path(path).get(), orControlSystem.http(url).path(path).json().get() - Telnet:
ControlSystem.ethernet(host, port).telnet(prompt, line_terminator).command(cmd).get() - ❗ SCPI: previous
ControlSystem.ethernet(host, port).scpi(cmd).get()is nowControlSystem.ethernet(host, port).scpi().command(cmd).get()
- Files under
configwith a name likeslowtask-*.pywill be automatically listed (without loading). Scanning is triggered by browser reloading of the SlowHome page. - Clicking
startwill load the file. For auto-loading, the task must be listed in theSlowdashProject.yamlconfig file.
Major changes since 2402 Nisqually are:
- Controls script: Controls section
- User module threading
- Container support
-
Dockerfileand docker images on DockerHub and GHCR - File manager on browser
- Parameter substitutions in
SlowdashProject.yaml - Configuration by environmental variables
- Dash-Start
-
- see documentation, Controls section
- ❗ for example, previous
initialize()is now_initialize() - this is to avoid possible conflicts with user-defined functions
2024-05-30 Options to allow Python file uploading, program argument to select Slowdash configuration file
- Program option
--project-file=FILENAMEspecifies the Slowdash configuration file (defaultSlowdashProject.yaml). - In the configuration file, setting
system/our_security_is_perfecttotruewill allow file uploading of any type, including Python.
- Environmental variables used when
SlowdashProject.yamldoes not exist:-
SLOWDASH_INIT_DATASOURCE_URL: data source URL -
SLOWDASH_INIT_TIMESERIES_SCHEMA: time-series schema for the data source given above
-
- This will allow a docker-only quick start if a data source already exists. An example of
docker-compose.yaml:
version: '3'
services:
slowdash:
image: slowdash
ports:
- "18881:18881"
environment:
- SLOWDASH_INIT_DATASOURCE_URL=postgresql://postgres:postgres@POSTGRESQL_HOST:5432/SlowTestData
- SLOWDASH_INIT_TIMESERIES_SCHEMA=slowpy_ts[channel]@timestamp(unix)=value
-
SlowdashProject.yamlcan handle different DB host settings between docker-compose and bare-metal - Example docker-compose includes test data generation with
generate-testdata.py, using SlowPy in the SlowDash image - for PostgreSQL, SQLite, InfluxDB2, and Redis, just running
docker-compose upwill do everything
- creating, uploading, deleting, and editing files under the user project configuration directory
The syntax is basically the same as docker-compose:
-
${VARIABLE}for substitution with environmental variable-
${VARIABLE:-DEFAULT}with default when the variable is undefined or the value is empty -
${VARIABLE-DEFAULT}with default when the variable is undefined
-
-
$(COMMAND)for substitution with shell command execution result -
$$for literal$ -
environmententry to define variables, either as a dict or an array
Example:
environment:
- BASE=${HOME:-/home/slow}/project
- DB=sqlite:///${BASE}/SlowStore.db
data_source:
url: ${DB}
- If DateTime type was used for timestamps (instead of UNIX time), the query failed, and no data was returned, fixed
- Also: Implemented "retries" on connecting to SQL-DB and CouchDB, necessary for use on docker-compose etc.
- Screenshot did not include legend, fixed
To build a docker image,
git clone github.com/slowproj/slowdash.git --recurse-submodules
cd slowdash
docker build -t slowdash .
The container launches slowdash --port=18881 at the /project directory. To run this, bind the port and volume:
cd PATH/TO/SLOWDASH/PROJECT
docker run -p 18881:18881 -v $(pwd):/project slowdash
Two call-back functions in the user module are added:
-
loop()function: if this function is defined, a thread is created, and this function is called repeatedly. -
run()andhalt(): if both are defined, a thread is created, andrun()is called afterinitialize(). Whenhalt()is called, therun()function must exit as soon as possible.finalize()will be called afterward.
These two cannot be used simultaneously. For more details, see the Pages document.
- Data visualization is mostly working (except for custom layout)
- Tested back-end bindings including: PostgreSQL, SQLite, InfluxDB, Redis, CouchDB
- Controls are primitive, subject to large changes in the future releases
- All the analysis-related functions are experimental (SlowPy and Data Functors)