Feb 4, 2026

Starting my journal yippee. :)
My biggest personal shortcoming that I want to overcome this year is putting my thoughts down in writing. Some people I work with might find this a bit silly given how much I write down when I can actually get myself to do it. My problem is how hard it is to get started writing a proper piece, be it doc or blog post; I get so caught up in all the details and all the things I wanna say, and say as accurately and correctly as possible, that I end up dragging my feet big time getting start. So in this journal I intend to track the different random stuff I’m doing and tech thoughts I’m having throughout the day. I’m hoping that keeping the pressure low helps me build a habit of writing.

The theme of today is definitely that migration is hard. It always surprises me how often the things we treat as black boxes are not so easy in-and-out as we’d like to think. I’d love to be able to think of telemetry collection as a Unix-esque utility, where data comes in, we do something to the data, and then we send it somewhere else. Thinking of it in this simplistic way does help simplify learning for those new to observability, but as an observability implementer and product creator, we simply can’t afford to model it like this. There are so many tiny details in observability instrumentation that can get lost. A big one is the exact shape of internal telemetry data; you’d like to think you’d have some standard golden set of metrics you can always get and expect everywhere, but even an agreed Semantic Convention isn’t enough to ensure that any possible observability solution you use is going to be instrumenting it exactly as expected.
Case in point, I was looking with a coworker at some of the Collector self metric instrumentation. Specifically trying to track down how GRPC metrics are instrumented in each component to try and determine why the googlecloud exporter has some slightly different looking GRPC metrics than the otlpexporter. They both call into otelgrpc and get dialoptions from it, though using a slightly different API to do so. Doesn’t explain why we get grpc.* metrics from the googlecloud exporter while the otlp exporter makes rpc.* metrics happen. Couldn’t come up with a conclusive answer to this since I don’t really deeply understand the configgrpc package or the otelgrpc package. But it feeds into my above thoughts because the reason we have to dig into the code to figure this stuff out is that we have to derive an API Request Count metric in a way that matches user expectations for the Ops Agent when we start to move to OTLP export. Sure, the two exporters under the knife use the same package to instrument, but even then we can’t come up with a 1:1 translation for the thing we need. We’re going to need to do some other tricks to derive the same answer. This isn’t even to mention the cases where we can’t get an equivalent metric, or a metric is recorded incorrectly and thus not produced properly by the exporter we’re moving to. Migrations in observability tooling is a painful experience, and in the current landscape requires migrators to have a pretty deep understanding of what it is their looking at and what the data all means.

I addressed a Collector Contrib issue about the system.processes.created metric in the hostmetrics receiver, which I didn’t realize that we only produce on Linux and BSD. That decision was before my time, but I assume it was done based on their state of implementation in gopsutil. gopsutil likely only gets that info for these two platforms because only these two have nice APIs to get this info (/proc/stat fork count in Linux, sysctl kern.forkstat in OpenBSD). You can technically get this info in Windows and MacOS if you wanted to do some super cringe stuff, but not things we’re willing to implement in hostmetrics. https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/45721

I started today generally distracted as I remembered that we need to make the OS packaging for Google-Built OpenTelemetry Collector link to BoringCrypto. Goreleaser is really powerful but not the easiest thing in the world to use; getting a template that worked right took a bit of effort. I’ve never been a huge fan of their docs which is just a giant yaml file with all available fields and a bunch of comments. I’m just kind of hunting and pecking around, having an easier time just letting Gemini link me out to the documentation it can find better than normal Google searches lol
Getting the config working for BoringCrypto linking wasn’t too bad once I figured out the config quirks, but doing cross-platform compilation is a huge pain; in normal the standard distrogen generated container for cross-platform builds, this was a bit simpler because we could leverage Docker cross-building capabilities to orchestrate it. But goreleaser needs to build for all platforms in one go. This means the container will need to always install all cross-compilation toolchain stuff, and the goreleaser config will need to assume it’s running on debian with all the cross-compilation toolchains set up. Kind of annoying but not the end of the world, since the expectation is typically going to be that users do the goreleaser releases from a containerized environment anyway.
In the end, I managed to get BoringCrypto working with proper cross-compilation to at least arm64. What a pain, but luckily I got it working and hopefully shouldn’t need to look at it again any time soon… https://github.com/GoogleCloudPlatform/opentelemetry-operations-collector/pull/490

I saw another AI slop PR in Collector Contrib today. My reaction to obvious LLM-proxy syncophant comments is getting more and more visceral each time. If you’re reading this and are thinking of submitting an LLM-powered PR… Think about something else. :)

Song of the day: A.M War by Karnivool
I have been listening to a lot of Karnivool since the new album comes out this Friday! First album in 13 years!!!