Telemetry

[apljungquist]

When working on a project with multiple contributors it can be difficult to know which workflows are important (frequently used) and when changes by one contributor adversely affects a workflow that is important to another contributor. I'm curious about using just to gather telemetry about these things, e.g. by integrating with something like DataDog, Sentry or OpenTelemetry.

Reasonable functionality:

• Create a trace for each just invocation.
• Create a span for each recipe.
• Log the exit status.
• Propagate trace id using environment variables to support recursive just calls and integration with other tools that call or are called by just.

Crazy functionality:

• Inspect output to identify and relay warnings and errors.
• Inspect output to identify fine grained phases, such as compilation or even compilation of individual crates (Cargo specific).

Is this something that others have experience with, or opinions on?

[casey]

I'm not sure how I could add telemetry to just in a privacy respecting way. Any telemetry would have to be opt-in on a per user basis. I suppose opt-in support could be added for logging, and then users could voluntarily upload their logs.

[apljungquist]

I was thinking that just could look for a Sentry DSN, or the equivalent connection info for other integrations, in an environment variable. Teams or other communities would distribute their DSN to contributors and contributors would set it in their environment.

So the telemetry would not be for the just maintainers, but for teams and other projects that use just.

[laniakea64]

It could be gated behind a cargo feature that is disabled by default, so users who want telemetry could compile just themselves with the feature enabled, and most/other users would not even have telemetry code in the build.

[apljungquist]

Reasonable

[laniakea64]

Can you implement the telemetry in your specific recipes?

Even merely making telemetry theoretically available where it is not needed or wanted is seen as anti-user privacy invasion and will tend to be negatively received. Users of just who do not have a use case for telemetry only want "Just a command runner" without perceived privacy issues, and will push back on anything to the contrary.

Your OP describes a scenario where a specific group has opportunity to collaborate to opt into telemetry collection & come to consensus of what data to collect and where/how to store the data. That scenario is not subject to the same level of privacy concern as implementing telemetry for a broad user base where most don't need or want telemetry. If you can implement telemetry on your side it would give your group more flexibility and save a lot of headaches.

[apljungquist]

Thank you for your feedback.

Can you implement the telemetry in your specific recipes?

Yes, I have experimented with executing programs with a time-like program that gathers telemetry. That would however require updating all recipes, contributing noise and making it more laborious to add or remove the instrumentation for a project. The nice thing about instrumenting just is that it is a natural chokepoint in projects that use it.

Even merely making telemetry theoretically available where it is not needed or wanted is seen as anti-user privacy invasion and will tend to be negatively received. Users of just who do not have a use case for telemetry only want "Just a command runner" without perceived privacy issues, and will push back on anything to the contrary.

This is a valid concern. And If I'm the only user that stands to gain something from upstreaming telemetry support, then that concern definitely weighs heavier than any value I could derive from the feature.

[apljungquist]

To be clear, my intention with posting here was twofold:

1. to hear if someone else has tried something similar and, if so, see what I can learn from them (maybe my idea is fundamentally flawed and will never bear fruit)
2. to see if the feature is something that others would be interested in using if it was available.

[thePanz]

Would a "pre-invokation hook" solve your question?
I am thinking about a specific code (or recipe?) that would be invoked before executing any recipe... that could give you the place where to record that invocation, or something else

just my 2¢

[apljungquist]

Could work if the hooks are invoked at the right time and provided sufficient information. So far, I'm getting the most use out of these pieces of information:

• name of recipe
• duration of recipe
• location of justfile (there are multiple in my project and just build in one directory should be tracked separately from other directories)

I think value could be derived from other information too, but I haven't had much time to develop the concept.

[mlavrinenko]

I wish just streamed events to a file while it runs

export JUST_EVENT_TARGET=.ci/runs/$(uuidgen).jsonl
just -q check &

# my tool
justel status $JUST_EVENT_TARGET # job status codes
justel output $JUST_EVENT_TARGET 42 # a job output

It will be possible to create both telemetry and CI tooling around it.

[apljungquist]

Nix can write events to a socket which one of my colleagues is trying to tap into ¹. If just followed a similar pattern, detailed telemetry would be possible, vendor agnostic and avoid adding anti client. The main downside is that it's harder to setup, which makes getting collaborators to opt in more difficult.

Footnotes

1. https://tangled.org/bergsoe.net/nod ↩