Skip to content

OTLP adaptor sends trace_id/span_id as hex strings — strict OTLP receivers reject the whole batch ("invalid TraceID length") #3683

Open

Description

@mayanksingh09

Summary

Logflare.Backends.Adaptor.OtlpAdaptor.ProtobufFormatter.build_log_record_fields/1
copies trace_id / span_id values from the log event body verbatim into the
protobuf LogRecord.trace_id / span_id fields:

defp build_log_record_fields({"trace_id", id}) when is_binary(id), do: [trace_id: id]
defp build_log_record_fields({"span_id", id}) when is_binary(id), do: [span_id: id]

Log events carry these as hex strings (32 / 16 characters), but the OTLP spec
defines the protobuf fields as raw bytes of length 16 / 8
(opentelemetry/proto/logs/v1/logs.proto). Spec-conforming receivers reject the
payload; e.g. the OpenTelemetry Collector (otlp receiver, pdata unmarshal) returns:

HTTP 400
{ code: 3, message: "unmarshal: invalid TraceID length" }   (or "invalid SpanID length")

Because the error is at unmarshal time, the entire batch containing one such
event is rejected
— with per-second batching this silently drops all logs that
share a batch with any trace-bearing event. Observed in production with a
Supabase log drain (type OTLP, HTTP/Protobuf, mint/1.8.0) pointed at an OTel
Collector v0.155.0 on Cloud Run: batches of Postgres logs (no trace_id) are
accepted (200), batches containing API/edge events are consistently rejected
(400, invalid TraceID length).

Reproduction (against any OTel Collector OTLP HTTP receiver)

An ExportLogsServiceRequest whose LogRecord.trace_id is the 32-byte ASCII
hex string (e.g. "0af7651916cd43dd8448eb211c80319c") → 400 unmarshal: invalid TraceID length. The same payload with the 16 raw bytes
(Base.decode16!(id, case: :lower)) → 200.

Suggested fix

Hex-decode when the value looks like a hex ID, and drop the field (rather than
poison the batch) when it can't be made spec-conformant:

defp build_log_record_fields({"trace_id", id}) when is_binary(id),
  do: id_field(:trace_id, id, 16)

defp build_log_record_fields({"span_id", id}) when is_binary(id),
  do: id_field(:span_id, id, 8)

defp id_field(key, id, size) do
  case Base.decode16(id, case: :mixed) do
    {:ok, bin} when byte_size(bin) == size -> [{key, bin}]
    _ when byte_size(id) == size -> [{key, id}]   # already raw bytes
    _ -> []                                       # unusable — omit, don't poison the batch
  end
end

This is the same bug class as, e.g.,
open-telemetry/opentelemetry-php#1059
(IDs encoded as strings instead of the wire-format bytes).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions