progress needs an API

[cgwalters]

Today we directly create progress bars for the terminal in the library code, but to actually be a library we need to have an API that allows the calling process to render.

In particular e.g. rpm-ostree is actually a DBus daemon and proxies its progress out over DBus, but in the general case obviously a client will want configurability.

Also xref bootc-dev/bootc#921 - I'd like to try to share code here.

[allisonkarlitskaya]

Thanks for filing this. I've put it in the bootc milestone.

Is there some sort of Rust state of the art for this? Should we just try to emulate the json format from bootc, or should we have a callback-based mechanism so bootc can create the json itself?

[cgwalters]

Is there some sort of Rust state of the art for this?

Not that I know of - also specifically in this domain it's not just Rust we're concerned about, that's why the "json lines" format which is a standalone thing (not that it's a complicated spec 😉 )

Should we just try to emulate the json format from bootc

I think it'd be extremely helpful for us to go in the direction of sharing more code so making another copy of that seems suboptimal from that PoV. Maybe it moves into this crate? Another crate seems overkill.

or should we have a callback-based mechanism so bootc can create the json itself?

That is how the ostree-ext code is doing it, which is pretty reasonable I think.

[allisonkarlitskaya]

I've started working on this and took a look at the linked report.

The immediate tension I see is in trying to figure out a balance between useful concepts for OCI (such as "numbers of layers") vs. wanting to share the API for things like the http and (future) ostree downloaders...

We might say that because those are separate APIs that maybe it's OK to have separate progress APIs, but I have some dream of some day unifying this under some large "just get me my updated image" API that leaves the exact network protocol as an implementation detail. Consider our plans for a composefs system-level "pulling" service to be used from flatpak, for example.

I like that the bootc JSON API has stages. I think in general we can split things up into:

• prep work (scanning the local system, downloading manifest, config)
• bulk download stage (ie. pulling layers)
• unpacking/processing work at the end (although we should try to keep this to an absolute minimum in my opinion by trying to do as much work as possible concurrent with the download, even slowing down the download if necessary if the disk can't keep up, for example). This should generally be very fast metadata-only operations, less than ~1 second or so.

In general I like the idea of splitting progress into "downloaded" and "cached" portions reported against some total. It feels like UIs might like to present that information to the user, possibly in the form of a multistage progress bar or simply as a percentage at the end. I've certainly tried to do that with indicatif (but failed).

[cgwalters]

I forgot to link bootc-dev/bootc#1016 here which has a lot of related things, but to summarize this strongly intersects with the problem domain of cross-language IPC in general. But also specifically e.g. Docker defined a JSON struct for its progress that seems reasonable superficially (though I didn't dig at the details), and LSP also has a schema (as it's json-rpc).

[cgwalters]

I think varlink would be a good match for this project. I am gaining some experience with it over in bootc-dev/bcvk#222

[cgwalters]

This is done by #295