Proposal: Custom high-performance rendering item

[rjbaucells]

Feature Description

Support custom Slint items that can perform high-performance rendering directly through the active Slint renderer.

The custom item should participate in Slint's normal rendering pipeline, including transforms, clipping, scaling and dirty-region handling, while allowing application code to perform the actual drawing using the active renderer's rendering context.

The API should remain renderer-agnostic. I am not proposing a new universal 2D drawing API; the actual drawing should use the capabilities of the active renderer (for example, a Skia canvas when using the Skia renderer).

I would like to discuss the appropriate API and architecture before implementing an upstream PR.

Product Impact

I am building a cross-platform C++ schematic editor with Slint.

The application needs a highly interactive canvas containing potentially thousands of graphical elements such as wires, symbols, pins, labels, grid elements and selection highlights. These elements need to be redrawn continuously while the user pans, zooms, selects and edits the schematic.

Using individual Slint items or paths for all of these elements would introduce unnecessary UI-item overhead. Rendering the canvas to an Image would also introduce an unnecessary intermediate rendering/copying step.

A custom rendering item would allow Slint to continue handling the UI, layout, input and item hierarchy while the application handles high-performance canvas rendering through the same renderer used by Slint.

This is important for the product rather than a nice-to-have feature. Without this capability, the application needs to maintain a separate rendering mechanism or use approaches that are not appropriate for a large, highly interactive schematic canvas.

[ogoffart]

One thing we usually recommend is to use an Image with a texture

Both existing routes have C++ examples:

• examples/opengl_texture renders into your own FBO and displays it as an Image, which participates in layout, z-order, clipping and transforms like any other element. See Image::create_from_borrowed_gl_texture.
• examples/opengl_underlay draws with raw OpenGL beneath the scene via Window::set_rendering_notifier.

So does that fit your usecase, or were you thinking something different?
Could you elaborate a bit more on the specifics and what kind of API you are thinking about?

[tronical]

And if you're up for an experiment, you can try out the simon/vulkan-cpp branch, which adds experimental API to import vulkan textures from C++.

[rjbaucells]

One thing we usually recommend is to use an Image with a texture

Both existing routes have C++ examples:

• examples/opengl_texture renders into your own FBO and displays it as an Image, which participates in layout, z-order, clipping and transforms like any other element. See Image::create_from_borrowed_gl_texture.
• examples/opengl_underlay draws with raw OpenGL beneath the scene via Window::set_rendering_notifier.

So does that fit your usecase, or were you thinking something different? Could you elaborate a bit more on the specifics and what kind of API you are thinking about?

Thanks, that's helpful. I was aware of both approaches, but my use case is somewhat different.

The Image + texture approach works well when the application renders into its own texture/FBO and then hands that texture to Slint. For my use case, I'd like to avoid that extra render target/boundary. The schematic canvas is continuously and interactively rendered, and I'd like the application renderer to draw directly into the rendering context that Slint is already using for that item.

The OpenGL rendering notifier is also not quite what I'm looking for. I don't want a separate underlay/overlay rendering stage, and I don't want to make the solution dependent on OpenGL. In particular, OpenGL is deprecated on macOS, so I would prefer not to build a new cross-platform solution around an API that is already deprecated on one of the target platforms.

One of the reasons I chose Slint is that its renderers already provide the appropriate native graphics backend on each platform. I'd like the custom rendering to take advantage of that rather than introducing another rendering path or requiring the application to manage platform-specific graphics surfaces.

What I'm envisioning is roughly:

Slint item tree
    |
    +-- CustomCanvas item
           |
           +-- Slint handles layout / clipping / transforms / dirty regions
           |
           +-- application render callback
                  |
                  +-- active renderer's drawing context

The important part is that the last component should be renderer-specific rather than defining a new universal drawing API. For example, with the Skia renderer it could be the existing SkCanvas, while another renderer could provide its own appropriate context.

I don't have a fixed API design yet. My main question with this issue is whether Slint would be interested in such a rendering extension point, and if so, what API shape you would consider appropriate.

The schematic editor is the motivating use case: thousands of dynamic wires, symbols, text, selections, etc., where creating a Slint item/path for every primitive or rendering a separate texture every frame isn't an ideal architecture.

[rjbaucells]

And if you're up for an experiment, you can try out the simon/vulkan-cpp branch, which adds experimental API to import vulkan textures from C++.

Thanks! I hadn't seen the simon/vulkan-cpp branch.

I think this is still similar to the Image and OpenGL approaches, though: the application performs the rendering separately and effectively sends Slint a "screenshot" (a rendered texture) of what should be displayed.

What I'm looking for is slightly different: having a custom Slint item that participates directly in Slint's existing rendering path, with Slint continuing to own the renderer/backend and the application providing the actual drawing for that item.

That distinction is important for my use case because the canvas is highly dynamic and continuously rendered. I want to avoid introducing a separate rendering target and then transferring/importing the result back into Slint.

Also, Vulkan on macOS isn't a native graphics API and requires a portability layer such as MoltenVK. Since Slint already has platform-appropriate renderer backends, I'd prefer the custom rendering to operate through the renderer Slint is already using rather than introducing another graphics path.