Experiment

To set this up I spun up a plain Debian VM and downloaded otelcol-contrib and otelcol from opentelemetry-collector-releases, both from the recent v0.145.0 release. If you’re not familiar with how these distributions are put together, they are actually built from the exact same upstream Collector library and component code. The only difference between them is the components included; otelcol-contrib contains every component available in the opentelemetry-collector and opentelemetry-collector-contrib repositories.

I ran the two collectors at the same time on my machine, using this config in each (technically two copies of it with modified port numbers in each so they could run simultaneously). If you’re following along, the command I used to run the collector in the background with redirected output was ./otelcol-contrib --config config.yaml > output.log 2>&1 & (and same with otelcol but using config2.yaml and output2.log).

Once these were both running, I checked htop, pressing F4 with the filter otelcol so we can look at both processes. I got the following result:

otelcol-contrib is using 4.4% of system memory, with otelcol only using 1.9%, basically half. The RES and SHR values for each of these processes reflects that difference directly. Why is it that two processes largely built from the same code, running the same codepaths since they are on the same config, have such a large amount of difference in memory usage?

Investigating System-level Process Memory Usage

Usually if I was investigating memory performance of a Go program, I would start with pprof. I’ve done this investigation before, so I know it’s actually less interesting to start with direct profiling. If you want to see that analysis see the Optional Colour section pprof heap analysis (though you may want to read the rest of the article first).

Looking at /proc/[pid]/status

So for starters, we’ll dive into more general Linux process memory statistics. We can start with /proc/[pid]/status to look at the memory values. The values in this file are often victim of approximation that makes the values slightly inaccurate, but we can still gather the trend of important numbers from it:

$ cat /proc/$(pidof otelcol-contrib)/status
Name:   otelcol-contrib
...
VmPeak:  1571824 kB
VmSize:  1571824 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:    177340 kB
VmRSS:    177340 kB
RssAnon:           30976 kB
RssFile:          146364 kB
RssShmem:              0 kB

Explaining the rows of interest:

• VmSize: This is the amount of virtual address space requested by the process. This is not a reflection of actual memory in use by the process, rather the virtual address space it has requested from the kernel. This is very high, but it’s not terribly controversial in my purposely contrived example. Go’s memory manager will always request hefty amounts of virtual address space.
• VmRSS: RSS is short for “Resident Set Size”. This means it’s the amount of actual space in RAM the process is currently using. This is the same as the RES column in htop. It is the primary number where we see a large gap between otelcol-contrib and otelcol. However, RSS doesn’t just mean “memory the program has allocated”. A process in Linux needs space in RAM for more things than just application memory, such as (crucially) executable code and data from the binary itself. More on this later.
• RssAnon: This usually is a better indication of memory used by the process that were requested by application code or operation in some way. There are some cases where anonymous mappings aren’t just standard heap allocations from the program, but in our case these anonymous mappings are taken up a fair bit by Go runtime heap allocations.
• RssFile: This an indication of how much of the space in RAM is taken up by data backed by files. If a process needs to read data from a file on disk, it will request memory from the kernel to hold while there’s available memory space to allow for reading that data directly from RAM instead of having to go out to disk every time. This is called “memory mapping” the file (read more at man mmap). As we can see, this is a large portion of the overall resident set size of otelcol-contrib currently.
• RssShmem: This is an account of how much of the resident set size the process is holding is shared by other processes in RAM. In our case, none of the memory held by otelcol-contrib is shareable. This is quite typical of non-CGO Go binaries. It’s all statically linked, meaning all the data that it loads to actually load the binary into memory is only usable by the process itself. This is unlike a typical C program for example, which will almost certainly share a glibc mapping that all other C programs on the system are sharing.

With this context in mind, let’s go check on otelcol’s status file for comparison:

$ cat /proc/$(pidof otelcol)/status
Name:   otelcol
...
VmPeak:  1384144 kB
VmSize:  1384144 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:     83520 kB
VmRSS:     83520 kB
RssAnon:           16656 kB
RssFile:           66864 kB
RssShmem:              0 kB

Let’s compare the relevant rows:

• VmSize: Lower, but is not a super important difference for reasons I stated above.
• VmRSS, RssAnon, RssFile: Lower, essentially taking up less than half space in RAM that otelcol-contrib is.
• RssShmem: Also 0 as expected, meaning none of the mappings here can be shared between any processes, and there’s no shenanigans like double-counting shared memory between running Collectors or anything; all pages of memory held by each Collector process are completely private.

Looking at Memory Map

The next step is to look into the direct details of the actual memory mappings of each process. We can get detailed information about all mappings through /proc/[pid]/smaps, but that information is actually a bit more detailed than what we need right now. Instead, we’ll use pmap(1), a utility that gives us a simplified view of a process’ memory map.

Starting with otelcol-contrib again:

$ pmap -x $(pidof otelcol-contrib)
6962:   ./otelcol-contrib --config config.yaml
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000  155076   72576       0 r-x-- otelcol-contrib
0000000009b71000  182424   79268       0 r---- otelcol-contrib
0000000014d97000    5560    4500     796 rw--- otelcol-contrib
0000000015305000     632     400     400 rw---   [ anon ]
000000c000000000   28672   28672   28672 rw---   [ anon ]
000000c001c00000   36864       0       0 -----   [ anon ]
00007f821bba0000     832     580     580 rw---   [ anon ]
00007f821bc80000      64       4       4 rw---   [ anon ]
00007f821bc94000    3968    3612    3612 rw---   [ anon ]
00007f821c074000    1024       4       4 rw---   [ anon ]
00007f821c174000      72      36      36 rw---   [ anon ]
00007f821c186000   32768       4       4 rw---   [ anon ]
00007f821e186000  263680       0       0 -----   [ anon ]
00007f822e306000       4       4       4 rw---   [ anon ]
00007f822e307000  524284       0       0 -----   [ anon ]
00007f824e306000       4       4       4 rw---   [ anon ]
00007f824e307000  293564       0       0 -----   [ anon ]
00007f82601b6000       4       4       4 rw---   [ anon ]
00007f82601b7000   36692       0       0 -----   [ anon ]
00007f826258c000       4       4       4 rw---   [ anon ]
00007f826258d000    4580       0       0 -----   [ anon ]
00007f8262a06000       4       4       4 rw---   [ anon ]
00007f8262a07000     508       0       0 -----   [ anon ]
00007f8262a86000     384      84      84 rw---   [ anon ]
00007fffabf3d000     132      16      16 rw---   [ stack ]
00007fffabfc5000      16       0       0 r----   [ anon ]
00007fffabfc9000       8       4       0 r-x--   [ anon ]
---------------- ------- ------- -------
total kB         1571824  189780   34228

The first 3 mappings are standard for basically any ELF binary. These mappings are all named with otelcol-contrib, meaning these are mappings backed by a file called otelcol-contrib which we know for a fact is the binary that this process is running. As we can see, a large portion of the overall RSS of our process is clearly coming from these mappings. I’ll discuss what these mappings actually mean in Optional Colour under ELF Binary Structure. For now, let’s simply take the point that these file-backed mappings take a large amount of the RSS of the overall process.

Let’s check this in otelcol:

$ pmap -x $(pidof otelcol)
8700:   ./otelcol --config config2.yaml
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000   79536   31340       0 r-x-- otelcol
00000000051ac000   73972   36104       0 r---- otelcol
00000000099e9000    2684    1824     376 rw--- otelcol
0000000009c88000     460     252     252 rw---   [ anon ]
000000c000000000   12288   12288   12288 rw---   [ anon ]
000000c000c00000   53248       0       0 -----   [ anon ]
00007f4007636000    4224    3720    3720 rw---   [ anon ]
00007f4007a56000    1024       4       4 rw---   [ anon ]
00007f4007b56000      72      20      20 rw---   [ anon ]
00007f4007b68000   32768       4       4 rw---   [ anon ]
00007f4009b68000  263680       0       0 -----   [ anon ]
00007f4019ce8000       4       4       4 rw---   [ anon ]
00007f4019ce9000  524284       0       0 -----   [ anon ]
00007f4039ce8000       4       4       4 rw---   [ anon ]
00007f4039ce9000  293564       0       0 -----   [ anon ]
00007f404bb98000       4       4       4 rw---   [ anon ]
00007f404bb99000   36692       0       0 -----   [ anon ]
00007f404df6e000       4       4       4 rw---   [ anon ]
00007f404df6f000    4580       0       0 -----   [ anon ]
00007f404e3e8000       4       4       4 rw---   [ anon ]
00007f404e3e9000     508       0       0 -----   [ anon ]
00007f404e468000     384      72      72 rw---   [ anon ]
00007fffbd111000     132      16      16 rw---   [ stack ]
00007fffbd1f3000      16       0       0 r----   [ anon ]
00007fffbd1f7000       8       4       0 r-x--   [ anon ]
---------------- ------- ------- -------
total kB         1384144   85668   16772

The actual structure of the mappings is almost identical as expected. We can see the file-backed mappings from the binary for the process (otelcol this time). A majority of the RSS is contributed once again by the binary mappings. However, they are much smaller in the otelcol process than the otelcol-contrib process.

This primarily demonstrates that the largest difference between the amount of space these two processes currently resides in RAM is the size of the root Collector binary. However, that exact choice of words is very deliberate.

What does RSS really tell us?

The only piece of information we know for sure when we look at an RSS value is the amount of bytes of RAM this process is currently taking up. Our model of this is slightly simplified due to the fact that our process is not sharing any of its memory with any other processes, whereass if we were we’d have to take into account the amount of RSS that our process takes up but shares with others (you can look at things like Proportional Set Size which will account for shared memory, but that isn’t necessary in this scenario). Overall though, knowing how much memory our process presently takes up in RAM is a really simplistic view of reality.

When a process wants memory, the Kernel will allocate space for it in a unit called a page. On Linux systems this is generally 4096 bytes, or 4KiB (you can check for sure on your system with getconf PAGESIZE). If you’re new to this concept and didn’t understand what I meant when I used the term “page” earlier in the post, now you know what I mean. :)
When the kernel allocates a page requested by the process, various things are taken into account such as whether the page is private or shared (in our case, all pages we are concerned with are private), whether the page is backed by some data readable from the disk, whether it is something special like page cache etc. All of this is to determine whether a page can be considered “reclaimable” by the kernel. User-space processes, drivers, or even kernel operations may be holding a lot of pages in RAM at a given time, and if a system is not under memory pressure, then the process might as well keep the pages in RAM because presumably these pages were useful for somebody at some point. However, once the system is under some manner of memory pressure, the kernel will do some work to reclaim the least important pages held in RAM at the moment to satisfy new requests.

I bring all of this up to say that some of the first pages to go in a high pressure reclaim scenario are often file-backed memory mappings. The first consideration is generally any inactive clean pages, but after taking an account of the LRU pages file-backed pages are preferred. An anonymous memory mapping is less preferred in part because if the page is dirty (aka has been written to) it needs to be written to swap-space on disk before the page can be reclaimed, otherwise the data would be lost. A read-only file-backed mapping is always clean and thus doesn’t have this restriction, as the data is readily available it can simply be read from disk again if it’s needed. While file-backed mappings are preferred, the reclaim still happens in a least-recently-used manner. Pages that are heavily actively referenced won’t be reclaimed as readily as dead pages. The otelcol(-contrib) file backed pages will be considerably active since they are constantly read for the process’s operation, but it doesn’t mean they aren’t reclaimable when push comes to shove. So while at time of checking we found that the first mapping of otelcol-contrib is taking 72576 bytes of RSS, that doesn’t mean many of its pages won’t be reclaimed in a memory pressure scenario.

That means that just looking at RSS doesn’t always paint a full picture of what matters in our process’s memory usage. The OOM (Out Of Memory) Killer is one of the biggest things you want to avoid, but a high RSS doesn’t necessarily mean you need to fear the OOM Killer yet; the OOM Killer will kill a process specifically when it is unable to reclaim enough space to fill a memory request. That is to say that it will first do everything it can to reclaim enough pages of memory to satisfy the new allocation request before OOM Killing a process.

Since I imagine the primary audience here is observability-minded folks, so let’s tie this back to observability for those who haven’t long since dropped off the article. Obviously the fresh VM I used for this experiment is experiencing essentially no memory pressure and can very easily satisfy memory requests for the foreseeable future, but even if we were getting tight, I might be getting worried about my collector by looking at the higher RSS value. If I’m looking at the memory usage of my system, and RSS as my primary per-process memory metric, I might consider the Collector to be contributing to some significant portion of that memory pressure at a glance. But RSS is a cumulative measurement that often measures a lot of reclaimable pages. The RSS itself could still be bad; since it’s also measuring dirty anonymous memory mappings, a consistently rising RSS can still indicate a memory leak in the actual program, and a process with a lot of RSS in a high memory pressure system may be not at risk of being killed but still at risk of causing heavy page thrashing for the system. However we can’t actually grasp the true nature of a given process’ memory usage and what effect it has on our whole system just by its RSS value, and in the case of my contrived example the high RSS is not such a big risk because we know how much of RSS is presently taken up by file-backed pages.

Note on cgroups

The Collector is often not running as a standalone process like this. Typically it will be running under a cgroup, either as a systemd service or as a container image. A cgroup itself can have a local memory limit. The page reclaim behaviour that I explained in the previous section when the entire system is under memory pressure also applies to when a cgroup is locally hitting its memory limit. However the page reclaim doesn’t occur on the whole system, only locally on the pages owned by the cgroup.

Conclusion

It still helps to keep Go binary sizes down. Using more RSS can still cause some problems. But the impact that the larger binary actually has on our practical system operation is not as bad as it looks.

From an OpenTelemetry perspective, as a user of the Collector looking at this you might be wondering whether this means you’re safe to use contrib or if you should still pursue building a custom collector. There remain great reasons not to use contrib:

• Larger binary/image takes more disk space
• More components = more dependencies = more threat space for vulnerability scanners to yell at you :)
• There can be some negative effects to the RSS overhead of the larger binary

But overall, even despite some of my previous public statements that the increased memory overhead of contrib is really bad, after looking much deeper into it and understanding Linux kernel memory management more, I understand that it is not as severe as it looks and as I may have previously made it out to be.

Optional Colour

Branching explanations of things that didn’t fit nicely into the overall investigation.

ELF Binary Structure

We can understand more information about those initial binary mappings in the pmap output by understanding a bit more about the ELF (Executable and Linkable Format) binary format. The readelf(1) tool can give us a nice readable output that can help us understand more clearly. From that output, let’s look at the two most relevant sections when running readelf -a otelcol-contrib

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x0000000000000150 0x0000000000000150  R      0x1000
  NOTE           0x0000000000000f78 0x0000000000400f78 0x0000000000400f78
                 0x0000000000000064 0x0000000000000064  R      0x4
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x0000000009770691 0x0000000009770691  R E    0x1000
  LOAD           0x0000000009771000 0x0000000009b71000 0x0000000009b71000
                 0x000000000b225688 0x000000000b225688  R      0x1000
  LOAD           0x0000000014997000 0x0000000014d97000 0x0000000014d97000
                 0x000000000056df80 0x000000000060b3c0  RW     0x1000
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x8

 Section to Segment mapping:
  Segment Sections...
   00
   01     .note.go.buildid
   02     .text .note.gnu.build-id .note.go.buildid
   03     .rodata .typelink .itablink .gosymtab .gopclntab
   04     .go.buildinfo .go.fipsinfo .noptrdata .data .bss .noptrbss
   05

There is no dynamic section in this file.

There are no relocations in this file.

The crucial information here are the 3 LOAD sections, and the respective Segment Sections 02, 03, and 04 that gives information about what sections from the binary end up in each segment. The LOAD sections are what the process will load into memory. You can see the PhysAddr values for each of these mappings are exactly the same as the mappings from the pmap output. The Section to Segment mappings then tell us what exactly is in each of those sections. In 02, the 0x400000 segment, we can see the .text section. This is where the actual CPU instructions are, hence it living in a section with the R E (Read and Execute) flags that get mapped to r-x mode when mapped by the process. In 03, the 0x9b71000 segment, it has the .rodata (read-only data) section which is where all constant data ends up. It also contains a couple interesting Go-specific sections, like the .gopclntab which I’ll discuss shortly. In 04, the 0x14d97000 segment, we have the .data and .bss sections. The .data section contains non-constant data that is known at compile-time, i.e. something like var x int = 0 at the top of a file would make it to this section because it’s initialized and thus known compile-time. The .bss section is where known uninitialized data will go, such as var x int without being set. These go in the writeable section because the program may end up changing the values of data from this segment.

Further elaborating on the .gopclntab, this is the Program Counter Line Table. This is a table of program counter values (program counter being an address of an instruction the program could be running) to source information from the original Go code used to compile the binary. Have you ever wondered how even a binary stripped of symbols via ldflags="-s -w" still produces a backtrace to actual source locations when panicing? It’s because no matter what, this section of the binary is always kept intact. The runtime uses it for various things, but the panic backtrace is the most obvious one. You can see some more discussion about this in golang/go#36555 where the option to not write symbols to the pclntab was requested and rejected.

The two most obvious ways that a Go binary grows when adding more dependencies is partially that more code = more instructions, and partially that more code = more symbols to write to the pclntab. This means that a collector with more components grows substantially in both of these sections.

pprof Heap Analysis

The direct heap analysis reveals some interesting things, but nothing all that substantially exciting.

Looking at the heap profile for otelcol first:

$ go tool pprof http://localhost:1778/debug/pprof/heap
Fetching profile over HTTP from http://localhost:1778/debug/pprof/heap
Saved profile in /home/braydonk_google_com/pprof/pprof.otelcol.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz
File: otelcol
Build ID: 785743821d9ac664638f9342e4270b2bdbcef397
Type: inuse_space
Time: 2026-02-06 03:14:43 UTC
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 4121.87kB, 100% of 4121.87kB total
Showing top 10 nodes out of 34
      flat  flat%   sum%        cum   cum%
 1536.51kB 37.28% 37.28%  1536.51kB 37.28%  go.uber.org/zap/zapcore.newCounters (inline)
 1024.28kB 24.85% 62.13%  1024.28kB 24.85%  k8s.io/api/core/v1.init
  532.26kB 12.91% 75.04%   532.26kB 12.91%  github.com/xdg-go/stringprep.map.init.2
  516.76kB 12.54% 87.58%   516.76kB 12.54%  runtime.procresize
  512.05kB 12.42%   100%   512.05kB 12.42%  runtime.(*scavengerState).init
         0     0%   100%  1536.51kB 37.28%  github.com/spf13/cobra.(*Command).Execute
         0     0%   100%  1536.51kB 37.28%  github.com/spf13/cobra.(*Command).ExecuteC
         0     0%   100%  1536.51kB 37.28%  github.com/spf13/cobra.(*Command).execute
         0     0%   100%   532.26kB 12.91%  github.com/xdg-go/stringprep.init
         0     0%   100%   768.26kB 18.64%  go.opentelemetry.io/collector/exporter.(*factory).CreateLogs

As we can see, the heap profile is only accounting for 4kB of inuse_space. This is because, as the rest of the article explains, the Go heap itself is only a small proportion of the RSS of the program. In here the top node is from zapcore, likely the result of different components setting up zap loggers, as well as things from package init, which is allocation that happens as a result of the package being imported at all (either allocations that happen in global variables, or in explicit func init()s).

We did see through our analysis that otelcol-contrib uses more RssAnon, so it might have more heap space allocated. Is that the case?

$ go tool pprof http://localhost:1777/debug/pprof/heap
Fetching profile over HTTP from http://localhost:1777/debug/pprof/heap
Saved profile in /home/braydonk_google_com/pprof/pprof.otelcol-contrib.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz
File: otelcol-contrib
Build ID: 3cd07bf5097963927ede2748cc4538b224f3906a
Type: inuse_space
Time: 2026-02-06 03:22:14 UTC
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 11801.29kB, 63.79% of 18501.38kB total
Showing top 10 nodes out of 143
      flat  flat%   sum%        cum   cum%
 2609.95kB 14.11% 14.11%  2609.95kB 14.11%  regexp/syntax.(*compiler).inst
 2561.41kB 13.84% 27.95%  2561.41kB 13.84%  github.com/aws/aws-sdk-go/aws/endpoints.init
 1064.52kB  5.75% 33.70%  1064.52kB  5.75%  google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile.func2
    1027kB  5.55% 39.26%     1027kB  5.55%  google.golang.org/protobuf/internal/filedesc.(*File).initDecls (inline)
 1025.88kB  5.54% 44.80%  1025.88kB  5.54%  regexp.onePassCopy
 1024.47kB  5.54% 50.34%  2051.47kB 11.09%  google.golang.org/protobuf/internal/filedesc.newRawFile
  768.26kB  4.15% 54.49%   768.26kB  4.15%  go.uber.org/zap/zapcore.newCounters
  655.29kB  3.54% 58.03%   655.29kB  3.54%  runtime.itabsinit
  532.26kB  2.88% 60.91%   532.26kB  2.88%  github.com/DataDog/viper.(*Viper).SetKnown
  532.26kB  2.88% 63.79%   532.26kB  2.88%  github.com/vmware/govmomi/vim25/types.Add

It is! Around 14kB more in total. I can’t exactly pretend that difference is anything to get that worked up about though. However it is interesting to note that the difference is a result of us having more dependencies in our binary. I’d have to dig further into the profile to see which package is precompiling regexes that isn’t present in otelcol, but the aws/endpoints.init is pretty dead obvious that the AWS SDK that backs the AWS exporters has some allocations either in a func init or in global variables. You can also see DataDog, vmware, and more represented in this snippet.

But hey, we didn’t configure any of those components in our test config, why are these package inits happening? We’re not gonna use the AWS stuff, so those allocations are a total waste for us. When the Collector starts up, it takes a registry of all components it was built with and needs to initialize all the component factories so that they can be instantiated in the case that they are configured. This means there’s no avoiding the fact that these packages are unfortunately imported on Collector startup whether we asked for the relevant components or not.

Sources

Following are the most important resources that I either pulled information from directly for this article, or that more deeply explain things I went over here:

• Official Kernel docs on memory management, most of the page reclaim info came from here: https://docs.kernel.org/admin-guide/mm/concepts.html
• The Linux Memory Manager by Lorenzo Stoakes, which I have been reading an early access copy of and loving: https://nostarch.com/linux-memory-manager
• I linked it in-place, but repeated here for posterity the Wikipedia page on Thrashing explains the concept pretty well: https://en.wikipedia.org/wiki/Thrashing_(computer_science)
• mmap(2) man page which explains the different parameters that can be used when mapping a page: https://man7.org/linux/man-pages/man2/mmap.2.html
• proc_pid_status(5) man page which explains what /proc/[pid]/status fields mean: https://man7.org/linux/man-pages/man5/proc_pid_status.5.html
• Other man pages referenced are linked in the spots I bring them up
• I sort of cut it out of the article, but this issue was great reading to see more about Go’s heap allocation strategy. And it made me learn a fun new feature of Go 1.26, which is that the 0xc000000000 special address where the heap always starts today can be randomized through an experiment. Neat! https://github.com/golang/go/issues/27583

Starting my journal yippee. :)
My biggest personal shortcoming that I want to overcome this year is putting my thoughts down in writing. Some people I work with might find this a bit silly given how much I write down when I can actually get myself to do it. My problem is how hard it is to get started writing a proper piece, be it doc or blog post; I get so caught up in all the details and all the things I wanna say, and say as accurately and correctly as possible, that I end up dragging my feet big time getting start. So in this journal I intend to track the different random stuff I’m doing and tech thoughts I’m having throughout the day. I’m hoping that keeping the pressure low helps me build a habit of writing.

The theme of today is definitely that migration is hard. It always surprises me how often the things we treat as black boxes are not so easy in-and-out as we’d like to think. I’d love to be able to think of telemetry collection as a Unix-esque utility, where data comes in, we do something to the data, and then we send it somewhere else. Thinking of it in this simplistic way does help simplify learning for those new to observability, but as an observability implementer and product creator, we simply can’t afford to model it like this. There are so many tiny details in observability instrumentation that can get lost. A big one is the exact shape of internal telemetry data; you’d like to think you’d have some standard golden set of metrics you can always get and expect everywhere, but even an agreed Semantic Convention isn’t enough to ensure that any possible observability solution you use is going to be instrumenting it exactly as expected.
Case in point, I was looking with a coworker at some of the Collector self metric instrumentation. Specifically trying to track down how GRPC metrics are instrumented in each component to try and determine why the googlecloud exporter has some slightly different looking GRPC metrics than the otlpexporter. They both call into otelgrpc and get dialoptions from it, though using a slightly different API to do so. Doesn’t explain why we get grpc.* metrics from the googlecloud exporter while the otlp exporter makes rpc.* metrics happen. Couldn’t come up with a conclusive answer to this since I don’t really deeply understand the configgrpc package or the otelgrpc package. But it feeds into my above thoughts because the reason we have to dig into the code to figure this stuff out is that we have to derive an API Request Count metric in a way that matches user expectations for the Ops Agent when we start to move to OTLP export. Sure, the two exporters under the knife use the same package to instrument, but even then we can’t come up with a 1:1 translation for the thing we need. We’re going to need to do some other tricks to derive the same answer. This isn’t even to mention the cases where we can’t get an equivalent metric, or a metric is recorded incorrectly and thus not produced properly by the exporter we’re moving to. Migrations in observability tooling is a painful experience, and in the current landscape requires migrators to have a pretty deep understanding of what it is their looking at and what the data all means.

I addressed a Collector Contrib issue about the system.processes.created metric in the hostmetrics receiver, which I didn’t realize that we only produce on Linux and BSD. That decision was before my time, but I assume it was done based on their state of implementation in gopsutil. gopsutil likely only gets that info for these two platforms because only these two have nice APIs to get this info (/proc/stat fork count in Linux, sysctl kern.forkstat in OpenBSD). You can technically get this info in Windows and MacOS if you wanted to do some super cringe stuff, but not things we’re willing to implement in hostmetrics. https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/45721

I started today generally distracted as I remembered that we need to make the OS packaging for Google-Built OpenTelemetry Collector link to BoringCrypto. Goreleaser is really powerful but not the easiest thing in the world to use; getting a template that worked right took a bit of effort. I’ve never been a huge fan of their docs which is just a giant yaml file with all available fields and a bunch of comments. I’m just kind of hunting and pecking around, having an easier time just letting Gemini link me out to the documentation it can find better than normal Google searches lol
Getting the config working for BoringCrypto linking wasn’t too bad once I figured out the config quirks, but doing cross-platform compilation is a huge pain; in normal the standard distrogen generated container for cross-platform builds, this was a bit simpler because we could leverage Docker cross-building capabilities to orchestrate it. But goreleaser needs to build for all platforms in one go. This means the container will need to always install all cross-compilation toolchain stuff, and the goreleaser config will need to assume it’s running on debian with all the cross-compilation toolchains set up. Kind of annoying but not the end of the world, since the expectation is typically going to be that users do the goreleaser releases from a containerized environment anyway.
In the end, I managed to get BoringCrypto working with proper cross-compilation to at least arm64. What a pain, but luckily I got it working and hopefully shouldn’t need to look at it again any time soon… https://github.com/GoogleCloudPlatform/opentelemetry-operations-collector/pull/490

I saw another AI slop PR in Collector Contrib today. My reaction to obvious LLM-proxy syncophant comments is getting more and more visceral each time. If you’re reading this and are thinking of submitting an LLM-powered PR… Think about something else. :)

Song of the day: A.M War by Karnivool
I have been listening to a lot of Karnivool since the new album comes out this Friday! First album in 13 years!!!

Feb 5, 2026

Had a horrendous sleep last night, let’s see what I can reasonably accomplish today. If I could, I’d pick some lower brainpower tasks, but I’ve got too much to do…

I worked a bit on distrogen documentation. I have a large wave of new features to add, and I’m excited to be able to share the tool with new OTel community members whilst actually having usage documentation to go with it.

I don’t have a lot else to say journal-wise today because of just how hard I worked on the Go Binary Size blog post. It was my hardest post yet to write, but I’m really proud of it.

Song of the day: Break Those Bones Whose Sinews Gave It Motion by Meshuggah
I focus well to music that is incredibly aggressive, catchy, and groovy. This song is all 3 to me. Maybe not melodically catchy, but man that rhythm hooks me…

Feb 6, 2026

I slept a little better last night at least.

Today I have a lot of writing to do. I have to write a handoff for the GBOC packaging project, I have to finish the distrogen docs, and I’m writing a small piece about Collector plugin loading at runtime. So it will be a sparse journal entry once again likely.

Everyone is all excited today about the fact that Claude agents were able to write a C compiler. Cool I guess? It feels like pure hype-bait to me, except worse than the web browser because this time it was something that “worked”; that web browser was an obvious piece of garbage, but this compiles C code wow! But I’m really not shocked at all that an AI could write a C compiler tbh. The language is very heavily specified in text, and there are lots of C compilers that the AI is likely already trained on. Building something that can successfully compile C code isn’t exactly rocket science. Building something that can compile C code with all of the immense optimization work that goes into popular compilers like gcc is where the rocket science really happens.

Yesterday we talked about issues with double-writing old and new schemas when receivers are transition to semconv. The reconciliation is quite difficult if the two schemas have a a metric with the same name but other details about the metric changing. We came up with a decent reconciliation pattern here, but it might be confusing for new users in multiple ways. https://github.com/open-telemetry/opentelemetry-collector/pull/14538#discussion_r2775609768

Song of the day: The entire IN VERSES album by Karnivool
The new album came out, and dare I say it was worth the 13 year wait. This album closer is transcendental.

Feb 7, 2026

I’m doing a bit of writing today, but will be busy with a family event for the rest of the day so not much to say!

Song of the day: My Pantheon (Forevermore) by Kamelot
They are kind of a guilty pleasure band for me. Common perception appears to be that they are corny and formulaic. But for me that corn is buttered and salted, and it’s a damn good formula.

The Issue: go-nvml crashes our OpenTelemetry Collector

One of the features of the Ops Agent is GPU Monitoring; if you install the Ops Agent on a GCE VM with a GPU, you will automatically get metrics for it through the NVIDIA Management Library (NVML), and optionally through DCGM. To achieve this, we built specific instrumentation using the Go bindings for NVML and for DCGM.

We learned when attempting to upgrade our build of the Collector to Go 1.21 that the Collector would crash on startup if a GPU was present on the machine. It produced the kind of panic you wouldn’t usually be used to seeing in a Go program:

SIGSEGV: segmentation violation
PC=0x0 m=0 sigcode=1
signal arrived during cgo execution

Seeing PC=0x0 was very surprising to me. I had no idea how this sort of thing could occur in a Go program, even with CGO. Even more strange was that this crash was only happening on certain systems. How could something like a segfault be system dependent?
I was absolutely hooked. I would not rest until I understood why this could possibly be happening.

You can read the original issue in go-nvml and the issue I opened in golang/go to see the real discussions, or read on for my direct retelling.

Intro to dynamic libraries

This is information that I feel is important to understand the underlying issue. If you are already familiar with how dynamic libraries are loaded, you can skip to How go-nvml works.

Dynamic vs Static Linking

In C and adjacent languages, there are two ways to link a library to your application: static, and dynamic. Static linking is pretty straightforward; the library code is included at compile-time, and when the library is compiled into an object, it is then linked directly into the resulting binary. When the compiled program is run and something from the library is referenced, the implementation is already present within the binary. With dynamic linking, rather than the libraries being built directly into the binary, the libraries are simply referenced by the application to then be loaded at runtime. These will be .so on Linux or .dll on Windows. When the application is run, the operating system receives instructions to look for the libraries on the system, and if they are found they are loaded for the program to use, or if not found the program fails to start.

Static linking sure does sound great, right? There’s not much to think about there, the code is just included in the binary rather than needing to worry about having specific dynamic libraries on the system. Why wouldn’t you always do that? Golang agrees with you; all binaries built with pure Go are completely statically linked. This is actually a selling point of the language, and as an avid user of it I can feel the benefits. It is so nice to build a giant Go program, and just have one nice clean binary at the end with everything the binary needs. As someone working on a tool written in Go, I love that building and distributing it is so dead simple because it’s one statically linked binary. No separate instructions that certain libraries have to be apt installed onto the system, or being forced to distribute a container image for the tool to be usable.

Dynamic linking does have a purpose though, especially when writing lower level applications. One of the most popular ones is C runtime libraries, an implementation of which is available on any Linux distribution, or can be installed on Windows through the Visual C++ Redistributable (something I’m sure many gamers have installed and not really known why). C runtimes can be statically linked in most compilers, however it often doesn’t make much sense to statically link something that is available on most any system the application will run on. One of the biggest reasons is binary sizes. I’ve seen people online be quite confused at the size of a simple Go Hello World program exceeding a megabyte (at least at the time), but the reason for this is that Go does indeed statically link its runtime with the binary which baloons the size of the binary.

Large binaries with lots of static linked libraries has other complications as well, such as the amount of memory the program can take to run. I’d like to write a separate blog post about this at some point, but in short, large statically linked binaries can take more memory to run because loading the binary instructions and data in the first place takes up more space in RAM. The difference with dynamically loading libraries is that the memory the libary takes up in memory can be shared by any other processes using the library. So if we just take dynamically linking libc as an example, there are probably tons of other applications on the system also dynamically loading libc and all sharing that memory in RAM. If all those same binaries had statically linked libc, then they would each have a private copy of libc with all the space in memory that would take up and would be unable to share with any other processes on the system.

Dynamic Loading

The other way to interact with dynamic libraries is by loading them explicitly. With dynamic linking, the required libraries are built into the binary for the system to discover when the program is loaded. However, sometimes the exact library to be used can’t be known at compile time. There may be multiple versions of the library that the program is built to work with, and there needs to be some logic done at runtime to determine exactly which library is loaded. This is common with versioned APIs, where there may be v2 versions of functions present in dynamic libraries (rather than just reimplementing the functions so that backwards compatibility can be maintained, which is really important for dynamic libraries).
So the alternative method is loading the libraries at runtime using dlopen in Linux, or LoadLibrary in Windows. This gives you a handle to the libary loaded into program memory, and to find symbols in it you can look them up in the loaded library using dlsym in Linux or GetProcAddress in Windows.

Exporting Dynamic Symbols (Linux ELF binaries)

We have now exceeded my knowledge of how this might work in Windows, so this section is specific to ELF binaries on Linux.

What typically happens in the linking step is the linker maintains all external references to dynamic symbols in two sections of the binary called the PLT (Procedure Linkage Table) and the GOT (Global Offset Table). The PLT maintains references to all dynamic symbols used, while the GOT maintains the actual address of known dynamic symbols. Upon usage of a dynamic symbol, the compiler references the PLT entry for that symbol. At the linking stage, the linker will add those known symbols to the GOT. At runtime, when a PLT entry is called, it will look for an entry in the GOT and jump to that address, otherwise it willtry to resolve the symbol manually.

Let’s see this in action with a very simple C program:

#include <stdio.h>

int main() {
    printf("hi\n");
    return 0;
}

I’ll compile the binary with gcc and immediately disassemble it:

$ make
gcc -o hello -g -Wall main.c
$ objdump -d hello > hello.s

Let’s navigate the dump to the main subroutine:

0000000000001149 <main>:
    1149:	f3 0f 1e fa          	endbr64
    114d:	55                   	push   %rbp
    114e:	48 89 e5             	mov    %rsp,%rbp
    1151:	48 8d 3d ac 0e 00 00 	lea    0xeac(%rip),%rdi        # 2004 <_IO_stdin_used+0x4>
    1158:	e8 f3 fe ff ff       	call   1050 <puts@plt>
    115d:	b8 00 00 00 00       	mov    $0x0,%eax
    1162:	5d                   	pop    %rbp
    1163:	c3                   	ret
    1164:	66 2e 0f 1f 84 00 00 	cs nopw 0x0(%rax,%rax,1)
    116b:	00 00 00 
    116e:	66 90                	xchg   %ax,%ax

What we care about here is instruction 1158, with the call to puts@plt. This is a reference to a symbol puts in the PLT, which is a result of us calling printf from stdio.h in our program.

In the dump we can also analyze the disassembly of the plt:

Disassembly of section .plt:

0000000000001020 <.plt>:
    1020:	ff 35 9a 2f 00 00    	push   0x2f9a(%rip)        # 3fc0 <_GLOBAL_OFFSET_TABLE_+0x8>
    1026:	ff 25 9c 2f 00 00    	jmp    *0x2f9c(%rip)        # 3fc8 <_GLOBAL_OFFSET_TABLE_+0x10>
    102c:	0f 1f 40 00          	nopl   0x0(%rax)
    1030:	f3 0f 1e fa          	endbr64
    1034:	68 00 00 00 00       	push   $0x0
    1039:	e9 e2 ff ff ff       	jmp    1020 <_init+0x20>
    103e:	66 90                	xchg   %ax,%ax

Disassembly of section .plt.got:

0000000000001040 <__cxa_finalize@plt>:
    1040:	f3 0f 1e fa          	endbr64
    1044:	ff 25 ae 2f 00 00    	jmp    *0x2fae(%rip)        # 3ff8 <__cxa_finalize@GLIBC_2.2.5>
    104a:	66 0f 1f 44 00 00    	nopw   0x0(%rax,%rax,1)

Disassembly of section .plt.sec:

0000000000001050 <puts@plt>:
    1050:	f3 0f 1e fa          	endbr64
    1054:	ff 25 76 2f 00 00    	jmp    *0x2f76(%rip)        # 3fd0 <puts@GLIBC_2.2.5>
    105a:	66 0f 1f 44 00 00    	nopw   0x0(%rax,%rax,1)

We can see that puts@plt ends up doing a jump to address 0x2f76, the location of that symbol from GLIBC_2.2.5.

All of this will be important when we get to the bug itself, so I hope you stayed awake!

How go-nvml works

The Go NVML bindings are an interesting challenge. NVML is a closed source library, and the intended usage is to link to the shared object on the system using a public header. So the way the Go NVML bindings work is as follows:

1. Provide a copy of the NVML header
2. Using a 3rd party tool called c-for-go generate a set of Go bindings
3. Wrap the Go bindings in a light API layer for user friendliness

The function that was segfaulting was actually the first function, nvmlInit. So let’s look at the process of loading this function:

1. The library libnvidia-ml.so.1 is loaded using dlopen with the flags RTLD_LAZY | RTLD_GLOBAL.
2. Much of the API is versioned in the library, so each of the versioned APIs are search in the loaded library using dlsym. If the v2 version of a symbol is present, then the bindings are told to use the v2 version of the symbol. In our case, we are using an NVML library that’s new enough to have nvmlInit_v2, so we will end up using that symbol.
3. Each of these symbols is wrapped with an exported Go function, that loads the library and checks for errors before calling into the generated bindings. So we would call nvml.Init() in our Go code.
4. This would lead to the generated bindings, which are what actually calls into CGO using import "C" and calls C.nvmlInit_v2().

The Bug

A considerable amount of time has passed since this investigation took place, so I am writing with a ton of hindsight here. This explanation will obscure a ton of straw-grapsing, which you can look through in the Go GitHub issue I opened. For the sake of this post though, I’m going to skip to the part where it all came together and the issue and solution became clear.

Ignoring the deep inner workings of how the NVML Go bindings work, I will focus on the most important core of it. This project generates C bindings based on an input header file. This header file represents the accessible API for libnvidia-ml.so.1, a proprietary binary that is expected to be installed on the user’s machine and loaded at runtime. It is not provided as part of the binding package, and will not be linked as a part of the build. To deal with this, the linker flag --unresolved-symbols=ignore-in-object-files is passed to the linker as part of the bindings. This flag makes it so the symbols from nvml.h, which are not going to be resolved in the build with the shared object missing, will be ignored by the linker and not considered an error.

Our initial knowledge was that the bug occurred under the following circumstances:

1. Using Go 1.21
2. Building on Ubuntu Jammy or newer, but not on earlier distros like Debian 10 Buster

While at this point in the investigation a lot of these concepts were somewhat new to me, I did have a feeling that given the issue was with a dynamic library loaded through CGO, the issue probably had something to do with linking, and I suspected the version of ld on the system was the culprit, and that something in the CGO layer of Go had changed in conflict with a new version of ld. It took me a non-trivial amount of time to realize why, but this ended up mostly correct.

Standalone Repro

In order to a) determine whether this was go-nvml specific or something inherent to Go, and b) to not require me to have NVIDIA libraries installed while developing, I created a standalone reproduction. This confirmed that setting up a small CGO program under the same circumstances (providing a header but no object and passing --unresolved-symbols=ignore-in-object-files to ld) panicked in the exact same way. We can work with this from here on out.

Comparing Go 1.20 to 1.21

Using the reproduction, I will build 2 binaries, one with Go 1.20 and one with Go 1.21.

The repro program includes a header that defines a function get42 and makes a call to it. This symbol should be unresolved in the build, and should show up as such in our binary. If we use nm on the Go 1.20 binary, we can find our get42 existing as expected as an unresolved symbol:

$ nm cgo_dl_repro_go120 | grep get42
0000000000483760 T _cgo_49665a31f432_Cfunc_get42
                 U get42
0000000000483580 t main._Cfunc_get42.abi0
000000000051b1c8 d main._cgo_49665a31f432_Cfunc_get42

However, checking out the Go 1.21 binary shows an important difference, which is that this symbol is missing!

nm cgo_dl_repro_go121 | grep get42
000000000047ce70 T _cgo_49665a31f432_Cfunc_get42
000000000047cca0 t main._Cfunc_get42.abi0
000000000051b1a8 d main._cgo_49665a31f432_Cfunc_get42

The only get42 symbols are the CGO calls we make in the Go code and the symbol from the C code that CGO generates.

I did not fully grasp what I was looking at when I found this, but this turned out to be the important difference. The get42 unresolved symbol being missing actually meant that the get42 symbol did not have an entry in the PLT. This results in Go generating assembly for this program that looks like this (disassembled by go tool objdump):

TEXT _cgo_49665a31f432_Cfunc_get42(SB) 
  :0			0x47ce70		4154			PUSHQ R12			
  :0			0x47ce72		55			PUSHQ BP			
  :0			0x47ce73		53			PUSHQ BX			
  :0			0x47ce74		4889fb			MOVQ DI, BX			
  :0			0x47ce77		e88416feff		CALL _cgo_topofstack(SB)	
  :0			0x47ce7c		4989c4			MOVQ AX, R12			
  :0			0x47ce7f		31c0			XORL AX, AX			
  :0			0x47ce81		e87a31b8ff		CALL 0x0 <-- EVIL!!!!	
  :0			0x47ce86		89c5			MOVL AX, BP			
  :0			0x47ce88		e87316feff		CALL _cgo_topofstack(SB)	
  :0			0x47ce8d		4c29e0			SUBQ R12, AX			
  :0			0x47ce90		892c03			MOVL BP, 0(BX)(AX*1)		
  :0			0x47ce93		5b			POPQ BX				
  :0			0x47ce94		5d			POPQ BP				
  :0			0x47ce95		415c			POPQ R12			
  :0			0x47ce97		c3			RET	

And a reminder of what that panic looks like:

SIGSEGV: segmentation violation
PC=0x0 m=0 sigcode=1
signal arrived during cgo execution

That explains how we’re getting program counter 0x0!

The Solution

While I spent a considerable amount of time experimenting and looking through go tool linker and cgo source code to try and understand what was going on, and I did learn a lot, I ended up finding the problem with a good old fashioned git bisect. I ended up at commit 1f29f39.
The message of that commit: cmd/link: don't export all symbols for ELF external linking
The problematic code change was from this:

// Force global symbols to be exported for dlopen, etc.
if ctxt.IsELF {
	argv = append(argv, "-rdynamic")
}

To this:

// Force global symbols to be exported for dlopen, etc.
if ctxt.IsELF {
	if ctxt.DynlinkingGo() || ctxt.BuildMode == BuildModeCShared || !linkerFlagSupported(ctxt.Arch, argv[0], altLinker, "-Wl,--export-dynamic-symbol=main") {
		argv = append(argv, "-rdynamic")
	} else {
		ctxt.loader.ForAllCgoExportDynamic(func(s loader.Sym) {
			argv = append(argv, "-Wl,--export-dynamic-symbol="+ctxt.loader.SymExtname(s))
		})
	}
}

What does this mean? The code used to always pass the -rdynamic flag to gcc, which passes --export-dynamic to ld under the hood. The change for the code changed to only pass -rdynamic to gcc if the particular linker flag is not supported. The justification for this is in this issue (TL;DR it’s because this is unnecessary in most cases and thus wastes space on a majority of binaries). While it’s hard to know exactly when the --export-dynamic-symbol flag was added to ld, it seems like the only plausible reason that this issue only occurs on an ld version that is high enough.

Since -rdynamic is now not always being passed in the CGO build process, the change I ended up on was to modify the binding generation in go-nvml to always pass the --export-dynamic linker flag. This doesn’t break if the -rdynamic flag is passed, but ensures that we still have the required ld flag being passed in newer versions of Go and ld.

Conclusion

This was a very hard issue to figure out, and was around a week’s worth of effort. The solution was 16 characters. This is why it’s hard to measure coding productivity by raw output! :)

I’m still glad I went through all of it, and glad I went through the process of re-documenting it by writing up this post. Hopefully you got some enjoyment out of my adventure!

My Bachelor’s Degree

During my undergrad, I hated pretty much everything about school. I knew I loved Computer Science, and I was utterly committed to completing my degree, but I barely made. The system really felt like it was a carefully designed torture chamber made just for me to pay thousands of dollars to suffer in.
I loved so many of the concepts and subjects I was learning in Computer Science and Math. However, particularly for Math, the shift in priorities coming to University were a shell shock. The goal of these classes didn’t feel like learning anymore; they felt like a game to achieve the best mark. It was a game I sucked at. I don’t think there’s any words to effectively describe how bad I was at exams. I never properly learned how to cope with my intense distractibility and struggle to focus throughout school, and my memory for concepts I didn’t deeply understand was incredibly fragile. My success in any courses, even Computer Science ones, hinged almost entirely on what percentage of the mark was derived from exams. Even worse were the courses that required you to pass the final exam to pass the course (this is the worst of the “torture chamber designed for me”). I failed 2 classes, both through final exams alone: Object-Oriented Programming (which I had done extensively even then, but choked writing Java on paper), and Probability (which terminated my then-burgeoning interest in Data Science).

I wouldn’t think so much about the good old torture chamber now, considering I’m years removed from receiving my degree, but I am constantly upset on reminder of what university could have been for me. My passion has shifted from just Software Development to deep Computer Science. My wife loves to poke fun at me for reading basically only CS textbooks, and my spare time is often spent learning about increasingly deep computer science concepts. While I was in school, all I could feel was the stress-rage-hybrid of looming exams, and the intense desire to be free and get what I really wanted; a job as a software developer. I’m so grateful to have achieved that goal, but I still can’t help but think about how different my life would have been if I hadn’t had to go through something I was so bad at to get there.

Believe it or not, this article is not just for me to complain how much I hated university and exams (although it was cathartic to write, and I’m leaving it in). Despite how much I hated it, I really can’t blame University for being… University. The validity of the post-secondary system isn’t really the dialogue I’m going for (at least today). The real point here is that University was simply not for me.

University wasn’t for me, but what choice did I have?

I was lead to believe that University was the only way to get into the software industry. When it came time to decide my future, there didn’t seem to be an alternative. Even going for college seemed like a death knell for your chances to break into the industry, and I couldn’t even fathom trying to get in as a self taught developer. These were obviously not true then, and are even less true now as the narrative around alternate paths to the industry has improved significantly. At the time however, I hadn’t connected to any tech communities even online, and was far too sheepish to reach out for mentorship. My pipeline to the industry was driven largely by how my high school directed me and my vapid attempts to make video games on my own time. I could only consume the information and assumptions that were easiest available to me. It sure didn’t seem like bad information at the time either; every job posting I checked required a Bachelor’s Degree. EVERY one. It sure seemed like my only course of action.

Disdain for Bachelor’s Degrees

I have been waxing lyrical about my own woe-is-me relationship with University, but I’m not alone. The glorified engagement farm widely known as tech twitter has been firing away the catchy tweets about how they got into the industry without a degree, or vaguely asking whether the twitter-verse thinks a degree is required to get a job as a developer. (I guess I shouldn’t be so cynical about it, they are generating infinitely more clicks than this blog with no SEO will).

I’ve spoken to many of my peers and colleagues in the community, and an (anecdotally) common sentiment is that they feel university did not prepare them adequately for the “real world” of software development. Common misgivings were the outdated technology used in courses, the heavy requirements for seemingly unrelated maths, and lacking guidance on realistic software industry skills (source control, software architecture, web development tooling).

It seems like so many people are on the same page about this in their own way: what is taught for a Bachelor’s Degree seems to be heavily at odds with the standard industry requirement for it.

Do universities need to “get with the times”?

You could get upset at post-secondary programs in general. Perhaps these programs need to teach more applicable, employable skills. Maybe they should be directing student learning toward more practical topics to increase their confidence to enter the industry. If these classes aren’t teaching students what they feel will be useful for their jobs, then what’s the point?

The counter to this is often to espouse the value of foundational knowledge. The things you learn in University may not be things most folks will do day to day in their careers, but these fundamental concepts are an effective way to become a well-rounded developer.
So which of my strawmen is right?
SIKE, they both are. Sort of.

The modern software industry

Software is a pretty young industry overall, however the prevailing goal of most software jobs has remained relatively constant: to create products that people use. While that goal hasn’t changed much, the tools available to accomplish that goal have changed drastically. The advancement of developer-focused tools and frameworks has lead to a major shift in the kind of skills necessary to get started developing software. The tools at a developer’s disposal have become so sophisticated that they abstract numerous fundamental building blocks that previously required deep knowledge to use. Web application frameworks blur the line between servers and clients; Kubernetes has made distributed systems a game of learning the available tools; UI design suites have broken the barrier between vision and implementation. (Disclaimer: I know all of these are severe oversimplifications). The general theme of modern tools is to abstract difficult foundational concepts to flatten the barrier to entry; your success in these tools would no longer hang on how well you understand the complex technical concepts it abstracts, and instead on how well you can learn the tool (which is usually a far faster process). I’d argue there’s very few tools that have fully achieved that goal, but I can feel the paradigm shift. Overtime, a new vacuum of the industry has formed entirely for talent with existing expertise in these specific modern tools.

Herein lies the crux of the problem; CS programs at post-secondary institutions produce well-rounded software developers who can leverage their foundational knowledge to numerous paths in software development, but their education may not have prepared them for the overwhelming number of jobs that require specific skills in industry tools. So many students simply wanted to start working in the industry, but the industry pushed them toward a seemingly false start.

Software Development as a Trade

I think there’s still a place for University. For these modern tools to exist as monolith abstractions of complicated foundations, there needs to be niche experts to build them. There are still a number of software jobs that would benefit greatly from folks with deeper academic knowledge. Still, a large number of jobs don’t seem to be after academics, rather after software developers as practitioners of a trade. I think framing software development as a trade vs. an academic pursuit serves the needs of the modern industry pretty well. Software development as a trade is more like using the development of software as a means to an end to accomplish business goals. Practitioners of software development as a trade would be trained specifically in the relevant tooling, and their expertise would be catered to the needs of the industry. Software developers as academics, computer scientists if you will, would be the folks doing intense research and studying. They would be the experts building the bedrock of computing, and the tradespeople would be the experts bringing it to wider society.

What would this sort of separation gain us? For starters, there are more paths for future developers to enter the industry. Rather than University seeming like the only path forward, perhaps there could be a shorter trade program, or something like a bootcamp that trains people explicitly to become practitioners; they would start with the basics as any software education should, but provide a more direct path to preparing specifically for jobs in the industry. Such a large number of developers endeavour only to build great things and use software development as their tool to do that; more focused trades programs would get them to that goal faster. It would also increase the rate of new talent joining the industry, and that new talent would arguably be more primed to onboard to the average company building products with popular tools. This would still leave a place for universities not only to continue teaching the important required topics of a Computer Science degree, but even relieves pressure on them to conform to the needs of the industry. Developers with aspirations to learn specifically Computer Science can go into post-secondary, and those interested mainly in Software Development can pursue it as a trade.

I think we may sort of be heading in this direction already; I have never been to one, but bootcamps do seem to be similar in spirit to what I’m trying to describe. I think one of the limiting factors for alternate paths to the industry is the persistent dogma around Bachelor’s Degrees, and the usage of the degree as an arbitrary barrier for new folks to enter the industry. I think the biggest realistic step forward for the industry would be to not only acknowledge the validity of alternate paths, but also understand where they may be advantageous instead of simply settling for them.

If I’m being honest with myself, much of this is pie-in-the-sky optimism; any immediate shifts like this would require disjointed demographics and organizations with different values to somehow shift their priorities in sync. We do appear to be taking baby steps though; there is a rise in vocal self-taught programmer pride, and an increasing number of developers are finding their way into the field through bootcamps and online courses.

To be Fair and Balanced though, this idea would be unlikely to become a utopia. It would help a lot more people enter our industry in ways that suit their goals and learning style, and would allow companies to hire in a way that more directly suits their requirements. However, in our present environment of late-stage capitalism -

I SNUCK AN ANTI-CAPITALIST PREMISE INTO MY BLOG POST

GOTCHA! YOU SHOULD SEE YOUR FACE RIGHT NOW!

In our present environment of late-stage capitalism, we apparently cannot get enough of social and class hierarchies. A field like software development attracts a lot of pearl clutching and vapid gatekeeping. A very loud minority of people are desperate to fight over the definition of a “real developer”, feeling personally offended and protective of the title because some developers never needed to untangle hundred line C++ template error messages. This sort of desperation to attain and maintain pseudo-intellectual superiority over each other would absolutely be exacerbated by a publicly accepted difference between “software tradespeople” and “computer scientists”. In the worst case (and probably most likely) scenario, companies will absolutely eat that up. Whatever their public messaging might be, they would likely use this difference to create new pay hierarchies, and find a metric-assload of creative ways to keep tradespeople underlevelled and underpaid. I think there’s a very real chance it could devolve into a sort of class system among software developers, where university degrees arbitrarily earn smarmy confidence, and higher wages for similar work; arguably, this is even today’s status quo because some people suck.

Conclusion

I don’t know that there is a perfect solution to the problematic relationship between our industry and Bachelor’s Degrees. I’m certainly not one to suggest sticking to something just because it’s the way we’ve always done it, so I can’t help but ideate some perfect balance we may never truly achieve.
If you’ve clicked on this post there is a high chance we know each other and you’ve already heard me say all this, but if not: first of all, hi! Thanks for reading! If you are a new software developer, I hope you don’t feel as trapped as I did when I started, and that you are aware of the paths open to you. If you’re already in University, I hope this doesn’t somehow sour your perspective; I may have hated school, but I still consider it an incredibly valuable part of my life, and I hope it is the same for you. If you’re someone who does hiring in any capacity, I hope this post inspires you, in however minor a way, to critically consider what you look for and how you can adjust to keep yourself open to the talent that’s waiting to find you.
Whoever you are, I hope you took something away whether you agree or disagree. As always, I’m happy to discuss either way, because I don’t claim to be in any way an expert and I would love to hear your thoughts.

UFW (Uncomplicated Firewall)

If I had gone with a more fully-featured cloud hosting provider, such as DigitalOcean or Linode (not affiliated with either), I would have been able to configure my VPS’s firewall through a UI console. However, I had already purchased a really large server for cheaper with another provider. This meant I needed to set up my firewall right on the server myself. As a software developer, I am horrible at SysAdmin by nature; the idea of setting up critical iptables shook me to my very core. This was why I chose to configure my firewall with the easier to use ufw.

The setup I needed was as follows: deny all incoming traffic by default, allow all outgoing by default, then allow traffic on the ports I needed (namely SSH, HTTP, and HTTPS).
What scared me the most was potentially locking myself out of my server. Working directly with iptables put me at risk of this, as iptable rules operate at the kernel level. If I changed one wrong thing, I could completely lock myself out of my server. ufw didn’t have this problem, because it runs as a service; I could configure all of my ufw and start the service when I felt everything was ready. I did a dry-run on a temporary tiny VPS to make sure I wouldn’t lock myself out (sudo removed for brevity):

ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow http
ufw allow https
ufw allow 25565 # I have a Minecraft server running on this machine already!
ufw enable

After doing this, I disconnected and tried to ssh into the server again. It worked as expected, and now that I’d verified it worked on the test VPS, I ran them on my main VPS with similar success.

Time to come clean though; this wasn’t the first thing I did. This was actually one of the last things I did (hence why I was extra scared of locking myself out). The main reason I did this was to make sure my server wouldn’t accept connections to http://<ip>:<port>. This worked for some of my services, but not for one of them. The reason it didn’t work is because ufw by default cannot stop Docker from accepting connections directly to published ports. On install, Docker makes entries in the iptable rules that are evaluated before ufw’s. I tried a veritable cornucopia of bad solutions before finding this repo that provided the perfect solution for me.

With my server locked down (let’s pretend that’s the first thing I did, like it should have been) it was time to move on to the server I decided to use for reverse proxying.

Caddy

In my previous attempts at hosting things myself, I had fumbled through nginx reverse-proxy tutorials. While nginx is a great skill to learn, and an incredibly mature tool, I decided to take a different route this time and use Caddy. I acknowledge that nginx is great technology, but after using it on this server I am officially sold on Caddy.

The two reasons I love Caddy are its super easy configuration and its automatic https. The biggest challenges I had with hosting things myself in the past is partially my poor nginx configuration abilities, but largely that messing with certbot (an admittedly great and easy to use project) was a lot more work than I wanted to constantly manage for every single project that I wanted to host. HTTPS is a process that can be automated, and Caddy proves that. Now, I simply add a new site configuration block to my Caddyfile and I already automatically have HTTPS for that site (provided the domain name I specified has DNS configured correctly, more on that later).

I installed Caddy on my system through the stable apt repo. I used it as a systemd service, and added configuration to the /etc/caddy/Caddyfile. When I mention “adding something to the Caddyfile” further down this article, I am referring to editing this file and running sudo systemctl restart caddy to reload the configuration.

Gitea

The first thing I wanted to set up was my own git server! I used a fantastic open-source project called Gitea that replicates a lot of GitHub’s features. It’s missing some of the more advanced GitHub features, but as a place to toss my personal project code it seemed perfect.

I installed Gitea through a user-maintained deb package. Honestly, if I were starting from the top, I probably would just install it through Docker, but this is working fine for now anyway. Installing this package created a gitea user for me, so I created the necessary directories from this Gitea tutorial and gave the gitea user access instead of the git user that these docs suggest manually creating. The rest of the steps from then on in the docs ended up working for me. So now I was able to run the gitea systemd service on port 3000. The next step was to set up the reverse proxy so I could get into my Gitea instance.

I have a domain (ragecage64.com) with Google Domains, however I don’t use anything specific to that system. All I had to do was a DNS Address record for the git subdomain I wanted. It looked something like this:

Once this was set up, I added the following block to my Caddyfile:

git.ragecage64.com {
    reverse_proxy localhost:<gitea port>
}

After restarting caddy, I had https://git.ragecage64.com ready to go! There was a first-time set up screen that I forgot to take a screenshot of, but is pretty self-explanatory. I spent a little bit of time selectively migrating the repos I wanted to keep to my new Gitea instance, and setting up my SSH key and new username. Really loving Gitea so far!

Get files from Gitea repos within the server

This was an important step to the next couple things I’m going to talk about. Once things are pushed to my Gitea instance, I’m able to access those files on the server to perform any kinds of builds I may need to run them.
To do this, figure out where you Gitea is storing its repos (for me it was in the default directory /var/lib/gitea/data/gitea-repositories/<gitea username>). In this folder you can find the bare repositories. To get the data from these bare repositories from any where on your server, you can clone the bare repository, i.e. git clone $GITEA_REPO_DIR/<your repo>.git. Now you have a copy of the code on your server to do with whatever you please.

Bub the Discord Bot

This was probably the easiest thing to set up. My bot is written in Go, meaning all I need is a Discord Bot Token in my local env and to run the compiled program. First, I pushed the bot code to my Git. Then I pulled the bare repo on my server, ran the command in the Makefile, and ran the compiled binary. Pretty simple setup!

The challenge came when I needed to run multiple apps at once on my server. The more correct thing would probably be to create systemd services out of everything, but that’s hard. :)
The two things I needed to run and quickly get at logs for are my Minecraft server and Bub. I used multiple GNU screen sessions to accomplish this. I started named screen sessions like so:

screen -S minecraft

Once I created the screen session, I ran the server and detached from the session with Ctrl+A, D.
Then when I needed to reattach to the screen, I could use the command:

screen -xS minecraft

I did the same thing with my bot. Pretty good setup overall!

This Blog

I am now also hosting this blog on my server! This blog is a static site made with Hugo, which I highly recommend if you’re looking to make a blog. The great thing about this was that a static site is similarly easy to host through Caddy!
I started by installing hugo, cloning the bare repo (I had to --recurse-submodules because I installed a theme, important step), and running hugo. This built my site to the public folder (optionally, could output this public folder to a smarter place in the server). Next, I added the following block to my Caddyfile:

blog.ragecage64.com {
	root * <full path to site folder>/public
	file_server
}

And added the similar DNS record as above.
Now I have the site you are currently on! To update my blog now I push to my repo, pull the server copy of the repo, and run hugo. A bit more work than GitHub Pages where I previously hosted this site, but every part of this is more work than it used to be and I’m still having fun!

BitWarden

The last thing I got working was my own BitWarden instance to share with my partner and family. To do this, I decided to run a docker container of the Rust implementation of Bitwarden. I created a docker-compose file for the container (which maybe wasn’t necessary because I’m just using SQLite anyway, but that makes it easier to add a real DB later) and ran it in the background with docker-compose up -d. I then created a DNS record and Caddy reverse proxy similar to Gitea above, and followed the instructions to connect to BitWarden clients to my instance. When I first started the instance, I used the container environment variable SIGNUPS_ALLOWED=true. This allowed me and my partner to quickly sign up, before I restarted the container with this environment variable set to false. This means only the people I want to sign up for my instance can; it’s only on a SQLite database, it’s not exactly web scale!

Who knows what else!

Now I have an easy to way to host any future projects on one server! It’s pretty exciting, and I don’t know what’s going up next, but next time I think of something exciting it’s fun to know I always have somewhere to put it!

For years I’ve listened to software engineers more experienced than myself poke fun at the left-pad incident. Usually used as a joking throwaway comment about keeping package-lock files in sync, or in accordance with the related xkcd comic (which seems to get more relevant the older it gets). It was technically just before my time as a professional developer (my less-than-stellar jQuery experimentation was safe from this at the time), so I would take it as a cautionary tale that taught us an important lesson about the software supply chain.
It also informed a lot of the learning I have done over the years about what it means for software to be open source, the nuances of open source software licensing, and the difference between freedom and beer. I’ve always been passionate about software that is at the very least source-available; the collaboration between so many talented and passionate people has always felt like something of a panacea to me (depending how rosy my glasses are that day).

This is all to say that reading about what happened with the npm packages colors and faker left me with a lot to say. I would have gone the lazy route and tweeted my thoughts to the void as usual, however I haven’t posted to this blog in checks notes 10 months! Some content creator I am. The shareholders will have my head!

So without further ado, I’d like to take as nuanced a look as I can at all the moving pieces of this fascinating case study.

What happened with colors and faker?

The headline is not clickbait enough to attract anyone who does not already know about this situation (other than my proofreading partner, hi dear!). However, for the purposes of this post, I’m going to pretend you have no idea what’s going on and summarize quickly so we can build some context.

colors is an npm package that enables the user to colour their console text in their command line applications. Command line applications may not be the first thing that come to mind when you think of Node.js, but a vast majority of JavaScript dev tools have a command line interface and leverage this package to improve the appearance of their output.

faker (no link for this one; will explain shortly) is an npm package that will randomly generate data, however this data is believable; it falls into common data patterns like names, street addresses, movie quotes, etc. I am not exactly sure which was first, but this library was heavily inspired by counterparts in other languages such as Perl, Ruby, PHP, and Python.

These packages are authored and maintained by the same developer: Marak Squires (see his GitHub). These packages were used by thousands of Node.js applications, all published to and subsequently downloaded from the node package manager’s central repository. This large repository of packages is owned by npm, Inc. and GitHub, and is the source from which virtually every node application pulls at least some open source dependencies. colors and faker were both open source and published with the MIT License.

Last year, the author of these two packages decided that he was no longer interested in developing and maintaining the packages. They opened this issue, declaring that they would no longer be working on the package. Last week, they took this a step further: they intentionally introduced an infinite loop with spooky text in colors and, as we zoomers might say, yeeted faker from existence (not really, but I will expand on that later on). This affected thousands of Node.js applications, which means it affected a ton of developers and companies of all sizes. And I mean “all sizes”; one of the affected packages I am personally familiar with is Amazon’s aws-cdk, and this is just one of many widely used packages that were essentially bricked until the issue was resolved.

Now that we have a general idea of what happened, I’d like to add my interpretation of what it means to work with npm.

What does it mean to download an npm package?

One of the earliest lessons I learned when I first started using Linux is to not download and execute random scripts without reading them first and understanding the risk. They require that you make a conscious decision to trust the source of the script. This makes sense when you think about it; a bash script (especially with sudo permissions) has the power to do an incredible amount of damage things to your system (or maybe just fork bomb you as an epic prank, anything goes). Usually, where possible, you were encouraged to install your software through your distribution’s central package repository. This large repository of packages, all built specifically for the distribution, is owned and maintained by a dedicated group of volunteers or employees who vet and approve each one. There are ways for independent users or organizations to host their own repositories of packages, and integrate with the distribution’s respective package managers. These require that you trust their source, similarly to downloading and executing bash scripts.

The reason for this tangent is to relate it back to npm installing a package. Installing a package through npm is a combination of these two flavours of installing software; it is a package manager similar to the ones commonly included in Linux distributions, however each package in its central repository is not vetted and managed by a group of volunteers or employees. When you download and execute a package from npm’s central repository, you are trusting the author of that package.

Now it’s obviously a bit extreme to directly equate installing an npm package to sudo executing a bash script. It’s a lot more nuanced than this since the most popular packages in the repository also have the most security experts’ eyes on them at all times. They may not be constantly approved by a central group of people, but in a perfect world issues are swiftly reported and dealt with by package maintainers and consumers of the package. npm also has a number of mechanisms to keep dependencies at a certain version until you trust that an upgrade is up to your standards.

What does it mean to publish an npm package?

Every JavaScript package on npm is open source by nature. JavaScript is an interpreted language, and no amount of obfuscation will truly hide the JavaScript being shipping when a package is published to the central repository. This code can be licensed under any open source license that suits the project’s needs. This license legally defines the way that the copyright holder approves the code to be used. Once the license (or lack of one) is defined, and the minimum setup requirements are present, you are free to publish whatever you would like. You could publish the next big JavaScript framework, a useful new CLI tool, nothing, whatever you’d like provided it is legal.

How we’ve forgotten this

Node.js and npm are tools that have come about as close to ubiquity as a very short list of technologies ever have. The number of new developers who’s first step of their journey was/will be to run npm install is staggering. The largest companies in the world continue to rely on npm in varying capacities, and have contributed a large number of popular packages to its ecosystem. When such an apparent consensus of people are doing something, it’s easy to interpret some guarantee of safety. You wouldn’t jump off a bridge just because someone else did, but if 10 000 people jump off a particular bridge every day it’s gotta be safe, right?

One-way Trust

Let’s have a quick look at that MIT license again. The first line of this license states: “Permission is hereby granted, free of charge, to any person obtaining a copy of this software[…] to deal in the Software without restriction[.]”. Further down, the license states: “[sic, all caps] THE LICENSE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED[.]”.
I am not a lawyer, but my interpretation of this in plainer terms is that anyone is allowed to use this software as they see fit, but that the copyright holder provides no guarantee of anything that may come with that freedom. If it doesn’t work for some reason, doesn’t do what you want, or has some kind of major flaw, the copyright holder does not bear the responsibility. When some of the biggest packages on npm are maintained by such large, efficient, and dedicated teams, it is easy to forget that warranty of any kind is almost never included by default; that’s the one of the exchanges that is made when the software does not cost anything.

How this relates to colors and faker

All of this is to contextualize my general takeaway from this situation; the author of colors and faker were within their right to self-sabotage their packages. A quick disclaimer that I personally dislike what they decided to do, and I’ll shortly expand on that, but I want to explain why I think what they did is within their right.
I think this goes right to the root of what open source is, distilled to its core: some person writes code, they put it somewhere for the public to see, the public can do with it whatever the license permits. To be reductionist for the sake of brevity (let’s laugh and pretend I have that), I think every single piece of open source code reduces down to this. Even if code is published by a Fortune 500 company who have great motivation to maintain that software until the end of time, even if software is so popular that volunteers will diligently shepherd it along the moral high ground to the heat death of the universe, open source software is still this at its core. When you use this published code, you have made the implicit decision to trust the person who published it. The only binding promise that the copyright holder have made in return is that the code exists and you can use it.

Marak wrote these packages and published them under the MIT license. They can update this code however they want. If they want to intentionally corrupt the package making it print LIBERTY a bunch of times to scare the bejeezus out of me, it appears to me that they are within their right to do so. I saw a number of people on Twitter decrying a “breach of trust”, and how it ruins the image of the npm ecosystem for a package author to do this. In my opinion, something is only a “breach” of trust when that trust is established two way and both parties are responsible for it. Package authors are not responsible for the one-way trust you placed in them, nor are they responsible for the effect their actions have on the reputation of the npm ecosystem.

The way Marak went about this was pretty “chaotic evil” for my tastes, and I don’t personally appreciate that they broke the trust so many people placed in them. However, I think a large number of us have forgotten that the people we download all these gigantic dependencies from are not in any way obligated to maintain a relationship of trust, and on paper could go nuclear at any time. The modern software supply chain has led to us unknowingly permitting this at an unprecedented scale for decades.

You are being ridiculous

Yeah, sort of. I am being pretty “doom and gloom” on purpose to frame this crucial portion of the software supply chain in a particular way. Let’s come back down to reality for a bit to talk about what all of this means for well meaning software developers just trying to get their job done.

How to improve npm safety (while still using npm)

I should preface this section by clarifying that I’m relatively new to this problem at scale, and there are experts far smarter than myself working to solve and educate on these problems. I’d still like to close this article off with tips that I have for developers who are worried about how to protect themselves further in the future.

I imagine there are a number of you reading this who are upset that I am appearing to suggest every line of open source code you pull down be audited. We all know that’s really not feasible at any scale larger than “demo”. This is why there are so many tools, such as sonarqube, snyk, and every project’s most diligent contributor dependabot (I am not affiliated with any, just a few I’m familiar with) built with features to track and audit dependencies you’ve brought in that may contain vulnerabilities. However, these tools don’t necessarily help if you’ve accidentally pulled in a bad dependency during development.

When a package is published on npm, save for select circumstances, the version published is there in perpetuity unless npm decides to take it down. Even though faker@6.6.6 which essentially deletes all of its code is published on npm, it does not remove the history of faker releases. Code can only be unpublished from the registry if the package has no dependents, which faker had a number of. In this case, and the case of colors@1.4.44-liberty-2, npm provides the tools to protect against these releases if you are a direct dependent.
If you are a newer developer, I recommend understanding semantic versioning as fully as you can; it is one of the greatest defenses to much of what I’ve mentioned in this article. The most common practice when using semantic versioning is to use the ^ caret prefix on most of your dependencies because this is what npm does by default when installing a new dependency. It means that any updates to the major version will not be installed, but the latest release of that major version will be used. Similarly, there is the ~ tilde prefix is similar, which will not allow any updates to the minor version. Providing no prefix will pin a dependency at a particular version. If you aren’t already, it is highly recommended to use more discretion when deciding which prefix to use on new and existing dependencies you choose to bring in.
An important caveat here is that even people who were more reserved by only allowing patch releases of colors, which should suggest only bringing in bug/vulnerability fixes, still got screwed here by unexpectedly allowing a very breaking change. However, this defense is still good against typical benign cases.

The issue with increased discretion is it usually means you have to do more manual work when it’s time to update. Working in the Node.js ecosystem is implicitly accepting that everything moves incredibly fast, and it’s a danger to your application’s continued health to let things fall too far out of date. While far from a perfect solution, one of my favourite ways to combat this is npm-check-updates. It provides an optional interactive environment to select updates to packages that you feel confident are safe. It is a nice convenience in a process that haunts Node.js developers everywhere.

The sad truth is that there probably is no true way to stop this from affecting you. Semantic versioning is the biggest help when you are a direct dependent of the code you are trying to control. Unfortunately, npm package dependency graphs can go many layers deeper than we bargain for. If you pulled in even one odd dependency that doesn’t pin some sub-dependency nicely, and the sub-dependency becomes problematic, you could have an issue that you often can’t directly do anything about. This can lead to a frustrating amount of work, and what feels like a lack of control over your codebase if it happens often. For this one, I don’t have a great solution. I wish I did, because it’s a problem I have had for most of my time in the industry. I wouldn’t want to say that a great solution doesn’t exist somewhere, but it’s probably going to be a burden we have to bear in the Node.js ecosystem to remain safe and secure. The biggest piece of advice is to make sure you are controlling your direct dependencies as tightly as possible the more strict your security requirements are; even when a dependency pulls in a bad sub-dependency, you can protect against the ripple effect if you keep tight control over when you bring the direct dependency in.

Conclusion

I think what happened with colors and faker is a fascinating case study into how many of us have become complacent with npm’s hidden safety concerns. I love open source software, and I believe we can all do our part to ensure we use it safely. I hope this article provided a new perspective to the situation, and whether you agree or disagree feel free to reach out and discuss! I am interested to hear about your experiences.

Generally introduced to new developers around chapter 4 or 5 of their proverbial Intro to Computer Science books, loops are one of the most fundamental coding constructs a developer learns. The different simple ways we iterate over collections of data are often the core of the most complex applications ever built. This is to dramatically justify the probably-overkill rant I am about to write regarding iterating over a collection of data.

Truthfully, this title is a misnomer. I don’t think for loops need to die. My goal with this post is to present a case for the available alternatives to traditional for loops. Though they aren’t technically wrong, I hope to demonstrate the benefits of the alternatives and how I believe they contribute to the enhancement of code quality. (Ha, my first clickbait title. Unfortunately, this site earns me nothing. Your click paid me $0.)

I will use JavaScript for the code examples since types aren’t going to be a factor here, and I feel it’s the simplest language to convey the concepts in the post. I will stay as language-agnostic as possible.

The traditional for Loop

I’ll start by laying out the traditional for loop everyone knows and loves. While learning the fundamentals of coding you will write your first for loops like this.

for (let i = 0; i < 10; i++) {
	// Code to execute on each iteration
}

I still think this should be the first loop a new developer learns. It’s easy to understand; execute the code inside the braces 10 times. Once the developer gets to arrays, and they learn that arrays are addressed 0, 1, 2 etc. to retrieve data, the use case of for loops suddenly clicks:

const arr = [1, 2, 3, 4, 5];
for (let i = 0; i < arr.length; i++) {
	console.log(arr[i]);
}

“The loop runs from 0 to 4, so I can use i to choose an item from the array on each iteration of the loop! Now I understand what loops are for!” - A dramatic re-enactment of me getting to the array lesson in my first coding book, feeling like a genius.
It’s understandable why a developer would reach for this by default; any professional developer is certain to have written hundreds of for loops exactly like this throughout their career, so there is rarely anything new for a developer to learn or understand.

Why fix what isn’t broken?

foreach loops

How many times have you seen a loop like this?

const arr = [1, 2, 3, 4, 5];
for (let i = 0; i < arr.length; i++) {
	const value = arr[i];
	// do stuff with currentValue
}

This code isn’t inherently wrong, but doesn’t it seem like a waste to write a traditional for loop just to assign the value to a local constant variable every time? We don’t actually need to modify the source collection, we just need to iterate through it and read each value individually.
Enter the foreach loop. This style of loop cuts out that boilerplate step that assigns a local constant in each iteration of your loop. Instead of each loop iteration having an index, each loop iteration will have an item. In JavaScript, this is implemented using the for...of syntax.

const arr = [1, 2, 3, 4, 5];
for (const value of arr) {
	// do stuff with value
}

In my opinion, this second option is a lot cleaner. Using this new syntax, we keep our code more concise and focused by demonstrating our intentions with the data through the syntax.
This is a microcosm of what you’ll see in the rest of this post; while it’s not wrong to use a traditional for loop, we should find ways to be more explicit about our intention when we iterate through a collection.

Higher-order Functions

Rather than mince words to try and explain what a Higher-order Function (hereby referred to as HOF) is, I will simply link its Wikipedia Article, as well as this chapter of Eloquent JavaScript since this article is largely in JavaScript.
Why are HOFs important to the goals of this post? This fundamental construct unlocks a number of elegant ways to use and transform collections of data with more specificity than is generally possible with native looping constructs.
Assuming that you have read the suggested articles or are already familiar with the required anonymous function syntax, let’s look at some HOFs that we can use to work with collections of data. While the examples will still be in JavaScript, nearly every modern language has some version of the methods we’ll discuss here.

forEach

We have learned about foreach loops, implemented in JavaScript as for...of. However, there’s a HOF to do essentially the same thing. Let’s restate the previous example here:

const arr = [1, 2, 3, 4, 5];
for (const value of arr) {
	// do stuff with value
}

The foreach function takes in an anonymous function with one argument that represents an individual element of the collection. So the above example could be refactored to this:

const arr = [1, 2, 3, 4, 5];
arr.forEach(
	value => {
		// do stuff with value
	}
);

When I see this function, I think “We are going to perform an operation that reads each element of the array individually” and I can focus on how we will use each element.
I started with the forEach function because it’s a great introduction to HOFs in general. However, to someone not sold on HOFs as a concept, this might look no better than a for...of loop. In truth, the justification for this goes deeper into the concept of immutability and side effects that are core to the Functional Programming Manifesto. (I capitalized that like it was a real book, but unfortunately it’s not. Lots of great reading if you search that exact phrase, though.)

In the interest of staying in scope for this post, let’s instead move ahead to some similar HOFs that I believe provide a clear new advantage.

map

map is designed to handle the scenario where we want to apply a transformation to every element of an array. For example you may have a loop that wants to build the 2’s timestable.

const arr = [1, 2, 3, 4, 5];
const twoTimestable = [];
for (const value of arr) {
	twoTimestable.push(value * 2);
}
// twoTimestable = [2, 4, 6, 8, 10]

When another coder reads this, they will be able to tell that this is a loop to build a new collection of data based on each element of a source. Using the map function, we can instead specify that a new collection is a result of transforming the source’s elements individually.

const arr = [1, 2, 3, 4, 5];
const twoTimestable = arr.map(value => value * 2);
// twoTimestable = [2, 4, 6, 8, 10]

When I read the second example, I see the map function and instantly think “This is a new collection that is a transformation of the original”, and I can focus on what exactly the transformation is. That’s the important part anyway; the extra code that manages assigning the results to a new collection is simply boilerplate around what I would consider the unique behaviour of the program. It’s what makes this program special, if you will.

filter

filter is for when we need specific data out of a collection. Let’s say we want an array containing only the elements of our source that are divisible by 3.

const arr = [1, 3, 6, 8, 12];
const divisibleBy3 = [];
for (const value of arr) {
	if (value % 3 === 0) {
		divisibleBy3.push(value * 2);
	}
}
// divisibleBy3 = [3, 6, 12]

filter will give us similar benefits to map here. filter is a function that will produce a new collection that only contains the elements of source for which the specified function returns true. So the above example can be refactored to this:

const arr = [1, 3, 6, 8, 12];
const divisibleBy3 = arr.filter(value => value % 3 === 0);
// divisibleBy3 = [3, 6, 12]

When I see filter, I think “This will be a new collection of data that passes some criteria”, and then I can focus on the criteria. As with map, that’s what makes this program special.
In my eyes, this seems like the easiest HOF to sell. It is in my opinion the most intuitive because it’s the word we would probably use in plain English to describe what we are actually trying to do.

reduce (traditionally known as fold)

This one may be the hardest to sell of the 4 HOFs we’re exploring.
reduce is used for when we want to take the elements of a collection and deduce some final result from it. A good basic example would be calculating the sum of all elements in an integer array.

const arr = [1, 2, 3, 4, 5];
let sum = 0;
for (const value of arr) {
	sum += value;
}
// sum = 15 

While the idea of reduce is simple in explanation, its usage is a bit harder to wrap your head around at first. While the previous HOFs have accepted an anonymous function with a single argument (which represents the “current element” so to speak), the anonymous function we pass into reduce requires 2: the running value known as the “accumulator”, and the current element (like the previous examples). This function will then return the new value for the accumulator after whatever action for the current element. In this example, the “accumulator” will be the sum we’re calculating. We’ll seed the accumulator with some value (in this case 0) as the second argument to the outer reduce function.

const arr = [1, 2, 3, 4, 5];
const sum = arr.reduce(
	(sum, value) => sum + value,
	0
);
// sum = 15 

Building a result that combines all the elements in a collection is a great use of reduce, but it can also be good for finding an element in a collection based on some criteria relative to the other elements in a collection. For example, if we wanted to find the max element in an array with a traditional for loop it would look something like this:

const arr = [1, 2, 5, 4, 3];
let max = Number.MIN_VALUE;
for (const value of arr) {
	if (value > max) {
		max = value;
	}
}
// max = 5

The name “accumulator” becomes a slight misnomer in this scenario, because rather than being an accumulation of all the values, it is simply the end result we are interested in. Ignoring that, the reduce we write is pretty similar to the earlier example:

const arr = [1, 2, 5, 4, 3];
const max = arr.reduce(
	(max, value) => {
		if (value > max) {
			return value;
		}
		return max;
	}
	Number.MIN_VALUE
);
// max = 5 

You might be thinking “you silly goose, this is more lines than the original for!”
Correct, I have bamboozled you to demonstrate a common gotcha for writing these HOF argument functions; they do need to return a value. The way we’ve been writing them (without braces) implies that the calculation is the return value of the function. However if you go to write your first reduce and wonder why the heck it’s not working, the first check is to ensure that all code paths are returning a value.
This example can be written as a one-liner using a ternary expression:

const arr = [1, 2, 5, 4, 3];
const max = arr.reduce(
	(max, value) => value > max ? value : max,
	Number.MIN_VALUE
);
// max = 5 

When I see a reduce, I think “This will take the source collection and build some kind of result out of it”, and I can focus on what it needs to do to find that result.

This is so sad, I love for loops. There must be some use for them!

Fear not! HOFs are awesome, and in a pure functional language like Haskell, you would only be using them at all times. However, if you are not living in the Pure Functional Utopia, there are some still some great uses for traditional for loops.

Modifying the source collection

This post has so far assumed that you are only reading from the source collection and producing a new result. I always strive not to modify values in code; it’s so nice to always know with certainty what everything in your program is going to contain/equal. However, if for whatever reason you are required to modify a collection in place, a traditional index for loop is still the best way to cleanly do so.

const arr = [1, 2, 3];
for (let i = 0; i < arr.length; i++) {
	arr[i] = 69;
}
// arr = [69, 69, 69];

Comparing elements directly to previous or future elements

If you are in a scenario where you need to take different action based on elements before/after the current element, the traditional index for loop is going to be your best bet.

const arr = [1, 1, 1, 2, 2, 3];
for (let i = 0; i < arr.length; i++) {
	if (arr[i] == arr[i - 1]) {
		// do something 
	} else {
		// do something different 
	}
}

You are writing Go

Let’s take a brief intermission from JavaScript. To present a new perspective.
If you’re writing Go, you can probably throw everything I’ve said in this post out the window. The way Go handles loops is essentially the antithesis to what I’ve presented so far.
Not only is a for loop the only way to iterate through a collection of data, the for keyword even replaced the while keyword. This is in service of Go’s language design philosophy, which is (in brief) to standardize around one way to do things as much as possible (a philosophy I’d love to rant on in a future post).
Go does have a func type, meaning one could implement their own map function like so:

func Map(arr []int, transform func(int) int) []int {
	result := make([]int, len(arr))
	for i, value := range arr {
		result[i] = transform(value)
	}
	return result
}

func main() {
	arr := []int{1, 2, 3, 4, 5}
	twoTimesTable := Map(
		arr,
		func(value int) int { return value * 2 },
	)
	// twoTimesTable = [2, 4, 6, 8, 10]
}

(I wrote this myself, but would not have been able to without this post from Algorithms to Go.)

The beauty and curse of Go is that it’s up the developer to implement this if they want it. I don’t know any Go developers personally, but I imagine a some would turn their nose at this while others would welcome it. If you happen to be a Go developer, I would love to hear your thoughts on this!

You are writing C

Just use for loops.

Conclusion

You may read this post and think to yourself “this guy is stupid, it’s more readable to just use a for loop”.
You might be right given the context of the code you’re writing. “Readibility” is pretty subjective, and some people may prefer to see the boilerplate that comes with the for loop examples I presented here. My goal with this post was to present my perspective; I love the way HOFs describe explicit intentions for an iteration through a collection, allowing me to focus on the part of the code that matters.

I hope that you enjoyed reading this post! This is my first public blog post and I’m pretty nervous to show it to the world, but I really hope you gained something out of it and did not leave in an unbridled rage. Please follow my socials if you enjoyed this post and want to read more of my ramblings!

Tools and Libraries

yamlfmt https://github.com/google/yamlfmt

Language: Go

A command line yaml formatting tool, also structured as a library for extensibility or custom wrappers.

This is my largest open source success. The tool has over 1k GitHub Stars, and each release gets tens-to-hundreds of thousands of downloads.

go-utf8-codepoint-converter https://github.com/RageCage64/go-utf8-codepoint-converter

Language: Go

Tool to convert UTF-8 codepoint text to the unicode character the text represents.

Fluent Bit Lua Tester https://github.com/RageCage64/flb_lua_tester

Language: Rust

Allows you to run Lua scripts meant for Fluent Bit scripting in a sanitized environment with specific input and expected output.

collections-go https://git.ragecage64.com/RageCage64/collections-go

Language: Go

A library that implements common data structures for Go with best possible time complexity and minimal allocations.

multilinediff https://git.ragecage64.com/RageCage64/multilinediff

Language: Go

A library to write multiline diff output to the command line.

Open Source Work

Most of my open source work is done under my Google Github profile: https://github.com/braydonk

OpenTelemetry https://github.com/open-telemetry

Language: Go

Contributing to OpenTelemetry in a couple of ways:

• Codeowner of the hostmetricsreceiver in the OpenTelemetry Collector
• Member of the System Semantic Conventions Working Group

Fluent Bit https://github.com/fluent/fluent-bit

Language: C

An open source observability agent, which we use on my team at Google as part of the Ops Agent. I help fix a number of bugs in Fluent Bit, as well as doing code reviews and maintenance on the out_stackdriver plugin.

Monkey https://github.com/monkey/monkey

Language: C

An HTTP server written in C. It is a crucial component of Fluent Bit, and I have done some work on this repo to support fixes in Fluent Bit, as well as adding unit tests to the repo.

YouTube

I do have a YouTube channel at https://www.youtube.com/@RageCageCodes-ik2ue. I only have one video as of writing, I was thinking I might make more but I found making tutorial content not as exciting as I’d hope. I’m keeping it on the backburner just in case!

Gaming

TrustFall https://git.ragecage64.com/RageCage64/TrustFall

Language: C++

A Root Beer Tapper ripoff that I wrote as a school project. Uses Allegro 5 because I had to (well technically I had to use 4 but I refused to do that and accepted the consequences).

SpaceForce https://git.ragecage64.com/RageCage64/SpaceForce

Language: C++

A SHMUP that I wrote also for a school project.

SeeNoEvil https://git.ragecage64.com/RageCage64/SeeNoEvil

Language: C#

My entry to the 8-bits-to-infinity game jam. I sadly did not save the assets, which is too bad but my partner who drew them insists they weren’t worth keeping. I thought they looked pretty good. :D
The main thing I want to extract out of this is the code that worked with Tiled, I thought it was reasonably sophisticated for something I coded in under a week. Would be cool to extract it into a standalone library.

Prepared Talks

Deep Dive: How Fluent Bit Collects File Logs

https://www.youtube.com/watch?v=KrlvWBCGagI

This is my talk for Observability Day North America, a co-located event with KubeCon NA 2024. It was a lightning talk, but it ended up being a really dense talk and probably could have been full-sized. To compensate, I talked really fast!

T-shirt: Iron Maiden

Tuning OTel Collector Performance Through Profiling

https://www.youtube.com/watch?v=qMxxjB4meXo

This was a talk for OpenTelemetry Community Day 2024. It goes through my experience profiling parts of the OpenTelemetry Collector to find performance improvements.

Retractions: One of the solutions I talked about in this talk for Windows getting Parent Process ID had a flawed premise, ignoring the fact that the increase in WMI memory usage did offset the gains made in the Collector. So it ended up not being that big of a win, and we’re still working to find another alternate method for getting Parent Process ID.

T-shirt: Brook from One Piece Wanted Poster

How Much Overhead: How to Evaluate Observability Agent Performance

https://www.youtube.com/watch?v=BIaftvtFPHg

This is my talk for Observability Day 2023, a co-located event with KubeCon NA 2023. It was inspired by situations at work where people would ask things like “which agent has less overhead?” without fully qualifying their goals. I wanted to break down the problem down into more actionable pieces.

T-shirt: Meshuggah Catch-33

Learning To Fly: How to Find Bottlenecks in your Agents

https://www.youtube.com/watch?v=jf7t1CpoKlg&t=176s

This was a remote talk I did for the Is It Observable YouTube channel (awesome channel, highly recommend subscribing). This was perhaps the hardest I ever prepared for a talk, because it came with an in-depth reproducible demo, that ran in a Dockerfile and included code to graph OpenTelemetry Metrics directly in the CLI. It was a lot of fun to prepare and I think it’s one of my best talks. If you can get around the fact that my mic sounded TERRIBLE).

T-shirt: Zoro from One Piece

Background friend: A plush of the character Acrid from my favourite video game, Risk of Rain 2

Tutorials

5 Levels of Go Error Handling

https://www.youtube.com/watch?v=y5utZCeHys0&t=1s

This was my one attempt at “content creation”. It’s a relatively beginner-focused tutorial about Go error handling and how to do some more advanced things. The video was picked up by the algorithm this past summer and started getting a lot more attention. I’m not completely cutting myself off from making more videos in the future, but I did not have as much fun as I thought I would making this video so I’m not sure if I’ll make more. I think this tutorial is pretty good for what it is though and I’ll keep it around anyway!

T-shirt: It’s obscured!

Background friend: Acrid from Risk of Rain 2 again

Interviews

KubeCon NA 2024 with Is It Observable

https://www.youtube.com/watch?v=qf0OjAEzprs&t=365s

This was an interview with the Is It Observable YouTube channel. I talked a bit about the talk I was giving the next day at Observability Day, as well as some general best practices for managing performance of agents collecting logs.

T-shirt: Coheed and Cambria, Vaxis II tour shirt

Humans of OTel - KubeCon NA 2024

https://www.youtube.com/watch?v=TIMgKXCeiyQ

I was featured in the Humans of OTel series of interviews at KubeCon NA 2024, along with a lot of amazing peers from the OpenTelemetry Community!

T-shirt: Video filmed too high to tell!

KubeCon NA 2023 with Is It Observable

https://www.youtube.com/watch?v=5arixRhAIbs&t=161s

An interview with Is It Observable from KubeCon NA 2023. This was my first time doing something like this so I was definitely more nervous, but it was great practice!

T-shirt: Meshuggah Catch 33