Repository navigation
Proposal for a Pi4j gRPC API #511
Replies: 10 comments 44 replies
|
I like this idea, and the approach to add it as a plugin makes total sense. How does this use the underlying plugin, like the new FFM, to do the actual interaction with the hardware? |
|
I do need to work on the README. Would it help if it were to include some example HTTP traces / code examples? There are code examples for the client, but they're not highlighted or documented by the README (yet) |
|
The more example code, the better ;-) What is the end goal of this proposal: to include it as a new plugin inside the Pi4J code itself? Or keep it as a separate project in your or the Pi4J repo? |
|
BTW a big ➕ to have this as pure Java (Kotlin) code, independent of the underlying provider. This looks to me like the cleanest and most future-proof solution! |
The $100 question. I know from past experience that I'm terrible at maintaining binary distributions. So, my ideal solution would for the Pi4j project to include the code in its repo and CICD workflow, as you guys are already into the routine of building and distributing packages. Otherwise this could either risk becoming more abandonware or simply a project that I use myself when working on my Pi/Java projects. I definitely imagine an HTTP plugin component could be a drop-in replacement for pigpiod. Whether or not this is it would be up to the team. Such a plugin does require a server component. I'd be happy to also contribute this to Pi4j for all the same reasons as above. However, OpenAPI spec proposals are included with the plugin, so the implementation could come from elsewhere if that was preferable. Long story short: I cannot commit to single-handedly maintaining anything, so would very much like to contribute to a more competent project and carry on as I have been more recently, contributing to the codebase. Any variation of the above would make me happy. I just don't want to code to go to waste |
|
I also like the idea. Can you send a proposal PR to include it? Then we can have a look at what this means. |
|
Great one! Couple of questions:
|
|
I think this is where we're going out of scope. It can certainly be interesting, but if it should be in the pi4j core i'm not sure... Perhaps that is something then for another repo. The OpenApi spec should be here and others can consume it anyway they like. |
Well, my only point is to generate valid specs for generators (plenty of them exists in open source), not writing them. |
|
ok, then i misunderstood you =) |
Not that I would discount it, but I think that would be a different plugin using a different protocol. A bit like making a choice between LinuxFS and Gpiod.
Spring is my day job, so it was the path of least resistance when I wanted to get something working quickly and needing also work on the plugin in parallel. It made a bunch of things much simpler, not least server side events to drive event listeners. I'm not particularly attached to it. I would refer to the server implementation as a reference (at least to myself). A more lightweight server using the JDK HttpServer might be fun to build given a slightly longer timeframe.
This wasn't really a consideration. The primary goal of the plugin was to allow Pi4j application development off the Pi. I wasn't really expecting the server to be used in a multi-user environment or exposed on the internet. (That said, it does currently support wrapping an existing device from the server in a client plugin. I suppose that might elude to multi-user) |
|
On these two points:
I kind of imagined something like this during the server development. To provide a contract for an API that the client plugin would be able to communicate with and for the contract to be used to build a compliant server in any framework. If Pi4j did have OpenAPI specs or a gRPC schema, it would make sense to have a plugin of some kind provided as part of the distribution, even if one or many pluggable servers where provided a different repo.
In general I would disagree with generating specs from code (apologies if I'm misunderstanding). It's a cleaner approach in the long run and avoids the possibility of quirks of one implementation bleeding into the spec and therefore any others too. |
People will use the plugin for their projects and will not think about multi-user usage - that's our job to think ahead. Imagine you have a relay for air conditioning and multiple control panels to adjust temperature in different rooms. If you are wiring this simple example directly multi-user access is managed by the provider itself (e.g. with FFM through linux kernel/driver). But once you are in a network, someone should hold the highlevel state to make things work as expected during simultaneous temperature adjusting. That's certainly worth designing beforehand :)
You can create an openapi schema and create clients with tools like https://openapi-generator.tech/ But schema should be written carefully to complain with spec standards. |
I would disagree. It's an impossible task to stop people designing terrible software. That's something they have to learn in their own time :) I get the point you're trying to make, but your hypothetical design would be a good example of such bad software. Pi4j and the like are low-level, driver-like APIs. Building an application which exposed these over a network for multiple clients to affect at random would have all kinds of issues. A well designed multi-user application would have to take into account a whole set of problems and design choices which are well beyond the responsibility of the Pi4j team. I would leave this up to the user to choose or find out the hard way. A network-capable driver shouldn't be considered for "production" use. My intention with it was that it would be a development aid to allow users to build hardware-capable applications with Pi4j and for them to be able to work "off-pi" in a more capable development environment and not to be constrained by the mock plugin. |
Well, that's the thesis we can have an argument in a bar with some beers one day :)
Oh, I got it now, sorry for misinterpreting your post. If this is for developers, then it makes total sense. |
|
Tangential: I use JetBrains Gateway to develop remotely on Pi with IntelliJ and I find the zero-effort setup absolutely amazing |
|
should we have a page on pi4j.com describing how to get started with JetBrains Gateway? |
|
Maybe as an option on the "install Java and Tools" page. It's just "If you use IntelliJ gateway, you can connect to your pi now" after installing Java |
|
Having been nerd-sniped about gRPC, my repository now contains a gRPC implementation of the plugin alongside the previous HTTP implementation. Having now spent time working on the gRPC implementation, I'm becoming of the opinion that this could be a better way forward. gRPC solves a major problem about API specs in that you're pretty much forced into defining the schema for the API before any code gets written. To this end, the repository now has a module dedicated to API specification, which should make reviewing and refinement simpler and I would like to propose this as either an addition or complement to the PI4J project. I would very much appreciate any feedback the team might have, particularly on the schema / API specs and would be happy for this to happen either on my own repo or in a PR to Pi4J. I understand the concerns around the original HTTP / Spring Boot server implementation. I think that maybe these are addressed somewhat by the more lightweight gRPC version. Happy for any suggestions as to next steps, even if just to polish the code in its current repository. The repo is still at: https://github.com/IAmNickNack/pi4j-plugin-http |
This is correct. The API with registers is for convenience. One could just as well use one array for everything where the first byte is the register/address to which is being written. |
|
This is off-topic, but there's a similar situation with the SPI API too where I had though of raising this as a separate discussion to ask whether or not this could be addressed and these functions removed / moved as a breaking change in v4 |
|
@IAmNickNack can you open this in a new discussion to look at? I am sure @taartspi will have a few comments. |
|
@IAmNickNack @eitch later in the day i will open a discussion on spi write for a large buffer (unless Nick beats me to it). I don't like what was done in pigpio and duplicated in linuxfs for the same user. I was going to ask for this topic on the call but we can solve the question in a discussion. Transfer with multi spi_io_messages or say a limitation |
+1 for having grpc, especially for development. I think it is a better option. |
|
I'm thinking about a PR for the gRPC plugin and server and there are some practical considerations that need to be taken into account.
This has now turned into a proposal for a gRPC solution and I'm currently questioning how likely it is that anyone is going to want to generate their own server stubs and implementation. I think this is an unlikely scenario, I would be tempted to suggest that the plugin and server both exist in the same repository. Is the main project the place for both of these components to live and if so, where in the repository would we expect to find the plugin and server components?
If so, a few pointers or discussion on the above would be useful. There's work for me to do to get things into an acceptable format and I'd rather not do a whole load of work based on assumptions, only to have to rollback and change stuff later. |
I would expect most users would use a maven dependency for the plugins they want and not clone the repo and build it from scratch
I've tried to keep this as simple and agnostic as possible, while also fulfilling the use-cases I have had. It may be simplest to take a look at the current repo. I think the part you're probably referring to is here => https://github.com/IAmNickNack/pi4j-plugin-http/blob/main/pi4j-grpc/pi4j-plugin-grpc-server/src/main/kotlin/io/github/iamnicknack/pi4j/grpc/server/config/ServerContext.kt There is also a v4 branch which explicitly includes |
true |
|
Right, the providers are currently relatively isolated... But I think @DigitalSmile pointed out somewhere that the project is considering deleting the "old" ones (= everything but ffm) Quoting Saint-Exupéry:
So my personal opinion is that it shouldn't be in core. What would be the advantage of being in core? |
This does make me wonder about how the server component, which is intended to run a standalone application (but could be embedded), could be distributed. It's entirely possible that the plugin distribution could also contain the server. The protobuf stubs are the same for both. This could make dependency management simpler. Bear in mind that none of this is intended for production use. The primary audience is developers who can't bear a laggy remote development experience or for some other reason, don't want to configure that stack. |
I would agree, it is more of build system plugin. I am not very familiar with maven/gradle plugin development, but may be we can create one. Another option is IDE plugin (intellij or vscode). It is also a good point of research. |
|
This discussion has been quiet for a while. In the interim I have archived the repository containing the HTTP version in favour of a repository containing only the gRPC implementation: https://github.com/IAmNickNack/pi4j-plugin-grpc. I've added minimal CI to this, published snapshot artefacts to Github Packages and released a shaded executable jar. Github Packages adds some complication to adding a dependency as GH requires users be authenticated by PAT. There's a brief summary of how I have achieved this for my own purposes here. This includes links to the relevant GH documentation for Maven and Gradle builds. The repo is now in a state where it serves my purpose. It may not be 100% feature complete and there may be issues to identify and resolve, but I'm personally happy to address these over time, as they are discovered. I would still be more than happy for Pi4J to adopt this and think the best way forward would probably be to fork my repository with a view to maybe detaching it later. |
|
@IAmNickNack this is really great work! Would you be interested in writing a blog and/or creating a video to add to the Pi4J website to demonstrate this? If you need help, we can also do a live stream or record an interview-style demo that I edit to a final result. |
|
Sure. Let me have a go at putting a blog post together first. If you'd like to get on a call to record a demo once that's done, we can talk about it then maybe. |
|
@FDelporte I've created a branch which contains a draft blog post. See what you think => https://github.com/IAmNickNack/pi4j-plugin-grpc/blob/NWS-pi4j-blog-post-proposal/PI4J_BLOG.md If you'd rather pair on a more interactive demo type thing, you may be able to convince me |
|
That's a great post @IAmNickNack ! Only thing I'm missing is some kind of diagram "what runs where". Correct the gRPC server is a RPi and the JBang example code can be executed from any other type of PC? |
|
I can have a go, but I'm not sure how interesting two boxes and an arrow would be ;) Does the pipeline which renders the website support Mermaid diagrams?
If you want hardware IO, yes. For testing purposes, it's possible to run the server on a non-Pi device too. It will just load the mock providers instead of FFM. I think I mentioned this in the post, but maybe not clearly enough. |
|
This went very quite :) Progress on the v4 APIs is obviously the current priority. I thought you might like to know that I've gone ahead and published the first snapshot build to Maven Central, so this is now publicly available to some extent. I've also updated the readme to reflect my earlier proposal for a blog post. I'd still be happy for Pi4j to own the repository and for me not to have sole responsibility for its maintenance, but at least for now it's out there. I'm pretty invested in v4, so hopefully it will be available soon so I can release a non-snapshot version of the plugin. |
|
Hi indeed, @IAmNickNack, my fault. I have been focusing on V4. As we are very close to being able to release, can we handle this shortly after? |
|
Hi @IAmNickNack Did you fully delete the HTTP version, or do you still have it somewhere? |
|
Yes, I was thinking the server could serve a simple form and accept HTTP post url-encoded commands as a kind of additional direct debug/control interface, while the HTTP post requests could also come from a machine directly. But when I thinking HTTP+Embedded I was assuming something along these lines: https://github.com/Pi4J/pi4j-example-devices/blob/master/src/main/java/com/pi4j/examples/server/SensorServer.java |
|
= something we could port to C and run on an ESP32 to control that from Pi4J, too |
|
There are some existing protocols, I am aware of these:
|
|
I think I know just what you mean. Maybe not particularly helpful in this context, but I did go down the HTTP route in another project (while also keeping gRPC) because I plan(ned) to build a GPIO UI in Godot, which has no inbuilt gRPC support. It probably wouldn't be too much work to port that HTTP server back to Pi4j. It would be interesting to adopt a pre-defined protocol though. But I think that might be a separate or complimentary piece of work. I will highlight that the primary intention of the existing gRPC implementation was/is to facilitate remote Pi4j development on a non-pi device using the Pi4j abstractions. As mentioned earlier, providing generally accessible low-level IO over the network was out of scope. |
The BPIO2 flattbuffer stuff doesn't look too bad... O:) |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Unless I've misunderstood something, remote development using pigpiod is no longer possible on the pi5.
I have been working on a possible replacement which uses HTTP. Obviously, performance might be thought to be an issue, but given the potential use-cases GPIO access over a network, these are arguably mute.
I would like to offer a proposal for a replacement for the pigpiod plugin which uses exposes exiting Pi4j plugins via HTTP.
The current implementation provides:
Nothing as yet has been reviewed or used by anyone other than myself. However, build and run should both be good.
Any feedback would be much appreciated. I'm hoping that this might soon be useful to somebody other than myself:
https://github.com/IAmNickNack/pi4j-plugin-http/
All reactions