In a distributed cluster with separate processor, querier, and ingestor nodes, the plugins assume the processor node can also query and write locally.
For example, with the Basic Transformation plugin, scheduled execution needs to both query source data and write transformed output. On a processor-only node, I hit these errors:
Node mode without query cannot retrieve table chunks executing query
and
cannot write to a read-only server: must enable ingest mode
This makes official plugins difficult to use in distributed deployments unless the processor node also has query and ingest enabled.
It would be helpful if official plugins supported remote query and remote write as an option, so a processor node could talk to a querier for reads and an ingestor for writes over HTTP.
For example, something like additional trigger arguments such as:
query_mode=remote_http
write_mode=remote_http
In my case, I already have the remote endpoints specified via environment variables so the plugin knows where to connect::
INFLUXDB3_QUERY_HOST_URL
INFLUXDB3_WRITE_HOST_URL
That would make the plugins work much better in a multi-node cluster without requiring process/query/ingest to be colocated.
In a distributed cluster with separate processor, querier, and ingestor nodes, the plugins assume the processor node can also query and write locally.
For example, with the Basic Transformation plugin, scheduled execution needs to both query source data and write transformed output. On a processor-only node, I hit these errors:
and
This makes official plugins difficult to use in distributed deployments unless the processor node also has query and ingest enabled.
It would be helpful if official plugins supported remote query and remote write as an option, so a processor node could talk to a querier for reads and an ingestor for writes over HTTP.
For example, something like additional trigger arguments such as:
In my case, I already have the remote endpoints specified via environment variables so the plugin knows where to connect::
INFLUXDB3_QUERY_HOST_URLINFLUXDB3_WRITE_HOST_URLThat would make the plugins work much better in a multi-node cluster without requiring process/query/ingest to be colocated.