'Unable to export traces to OpenTelemetry Collector on Kubernetes

I am using the opentelemetry-ruby otlp exporter for auto instrumentation: https://github.com/open-telemetry/opentelemetry-ruby/tree/main/exporter/otlp

The otel collector was installed as a daemonset: https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector

I am trying to get the OpenTelemetry collector to collect traces from the Rails application. Both are running in the same cluster, but in different namespaces.

We have enabled auto-instrumentation in the app, but the rails logs are currently showing these errors:

E, [2022-04-05T22:37:47.838197 #6] ERROR -- : OpenTelemetry error: Unable to export 499 spans

I set the following env variables within the app:

OTEL_LOG_LEVEL=debug
OTEL_EXPORTER_OTLP_ENDPOINT=http://0.0.0.0:4318

I can't confirm that the application can communicate with the collector pods on this port. Curling this address from the rails/ruby app returns "Connection Refused". However I am able to curl http://<OTEL_POD_IP>:4318 which returns 404 page not found.

From inside a pod:

# curl http://localhost:4318/
curl: (7) Failed to connect to localhost port 4318: Connection refused

# curl http://10.1.0.66:4318/
404 page not found

This helm chart created a daemonset but there is no service running. Is there some setting I need to enable to get this to work?

I confirmed that otel-collector is running on every node in the cluster and the daemonset has HostPort set to 4318.

ruby-on-railskubernetesopen-telemetryexporter


Solution 1:[1]

The problem is with this setting:

OTEL_EXPORTER_OTLP_ENDPOINT=http://0.0.0.0:4318

Imagine your pod as a stripped out host itself. Localhost or 0.0.0.0 of your pod, and you don't have a collector deployed in your pod.

You need to use the address from your collector. I've checked the examples available at the shared repo and for agent-and-standalone and standalone-only you also have a k8s resource of type Service.

With that you can use the full service name (with namespace) to configure your environment variable.
Also, the Environment variable now is called OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, so you will need something like this:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=<service-name>.<namespace>.svc.cluster.local:<service-port>

Solution 2:[2]

The correct solution is to use the Kubernetes Downward API to fetch the node IP address, which will allow you to export the traces directly to the daemonset pod within the same node:

  containers:
  - name: my-app
    image: my-image
    env:
    - name: HOST_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: http://$(HOST_IP):4318

Note that using the deployment's service as the endpoint (<service-name>.<namespace>.svc.cluster.local) is incorrect, as it effectively bypasses the daemonset and sends the traces directly to the deployment, which makes the daemonset useless.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1
Solution 2 dudicoco

Related Questions