Michael N. Kingsnorth<p><strong>Coffee Fix and Code: Goroutines, Ghosts, and Grounds</strong></p><p><em>Saturday. 7:03 AM. Woke up late, which is rare. I was out clubbing until midnight, not something I usually do, but it felt good to unplug from the terminal and dance with strangers instead of debugging them. Now the Nespresso machine hisses like it’s judging me. I take it black, no sugar. First sip: grounding. I pull up the logs. They’re eerily clean. And that’s when I know, something’s not right.</em></p><p>We are building our messaging system in <a href="https://go.dev/doc/" rel="nofollow noopener noreferrer" target="_blank">Go</a>, because we wanted low memory usage, statically linked binaries, and concurrency that didn’t require a PhD. But Go’s concurrency model, while powerful, has sharp edges. Especially when you treat goroutines like cheap threads and assume they’ll just behave. They don’t.</p><p>In early <a href="https://pkg.go.dev/testing" rel="nofollow noopener noreferrer" target="_blank">testing</a>, we have approximately 20 goroutines spinning at once, most of them legitimate. But some just wouldn’t die. They lingered. Quietly. Until they started eating memory, then performance, then our patience.</p><p><strong>The Symptom: Ghost Messages and Threads That Never Return</strong></p><p>It wasn’t a panic or a segfault, we would’ve noticed that. What we saw was subtler. A message that got processed on one end… and then vanished. No reply. No follow-up. Just silence.</p><p>In our test rig, we used a simulated network with predictable latency and packet loss. We knew what <em>should</em> happen. We even logged the packet hitting the handler. But inside that handler was a goroutine waiting on a channel that no one was reading from.</p><pre><code>func handleMessage(msg Message) { done := make(chan bool) go func() { process(msg) done &lt;- true // this hangs forever }() // forgot to read from done}</code></pre><p>In our mock tests, we could reproduce the bug under heavy load, but not always. That made it worse, because it wasn’t deterministic. And bugs that hide behind timing are the worst kind.</p><p><strong>Fix #1: Buffer Your Channels (or Burn in Debugging Hell)</strong></p><p>After half an hour of staring at the code and rerunning the same failing test, it clicked. That <code>done <- true</code> line was blocking because the main thread never read from the channel. In unit tests, it passed. But in the live system, it failed silently.</p><pre><code>done := make(chan bool, 1)</code></pre><p>That single buffer slot was enough. Suddenly, no more hanging goroutines. Our concurrency test suite, which had been flaky, stabilized overnight.</p><p>It felt stupidly simple. But that’s the thing about Go: it’ll let you shoot yourself in the foot quietly.</p><p><strong>Fix #2: Pass Context Like It’s Oxygen</strong></p><p>We were inconsistent with our use of <code><a href="https://pkg.go.dev/context" rel="nofollow noopener noreferrer" target="_blank">context.Context</a></code>. Sometimes we passed it; sometimes we didn’t. And when a request timed out or a user disconnected, the goroutines didn’t know. They just kept going.</p><p>In our test harness, we simulated rapid connect/disconnect cycles. Without context cancellation, the memory footprint ballooned after just 500 cycles.</p><pre><code>ctx, cancel := context.WithTimeout(parentCtx, 2*time.Second)defer cancel()</code></pre><p>Once we rewired every handler, worker, and sub-process to honour context cancellation, the system became smarter, and leaner. Our stress tests showed a 40% reduction in goroutine count after timeout events.</p><p><strong>Fix #3: Use Worker Pools. Not Chaos.</strong></p><p>Before the fix, our service would spin up a new goroutine for every incoming message. That felt fine, until a peer sent us 10,000 messages in a burst. CPU spiked. Memory thrashed. Our CI pipeline timed out trying to run the benchmark tests.</p><p>So we implemented a worker pool:</p><pre><code>msgQueue := make(chan Message, 100)for i := 0; i < 5; i++ { go worker(msgQueue)}func worker(queue <-chan Message) { for msg := range queue { process(msg) }}</code></pre><p>Our performance tests, which measure latency across five types of message paths, showed consistent response times even under load. No more uncontrolled spikes. Just steady flow, like well-poured espresso.</p><p><strong>Fix #4: Goroutine Metrics Are Not Optional</strong></p><p>We wrote a tiny wrapper to track goroutine count and channel lengths. It prints every 10 seconds in dev mode:</p><pre><code>import "runtime"fmt.Printf("goroutines: %d\n", runtime.NumGoroutine())</code></pre><p>In one test, we noticed that our message send path created <em>two</em> goroutines per request instead of one. That doubled over time, causing system-wide latency.</p><p>A postmortem revealed a leak in a timeout watcher. We would’ve missed it entirely if we hadn’t been watching goroutine metrics in real time.</p><p><strong>Final Thought: Concurrency Isn’t a Toy. It’s a Test of Discipline.</strong></p><p>You can write Go code that passes your tests and still fails in production, because concurrency bugs often hide behind timing, not syntax.</p><p>Now, when I reach for a goroutine, I think twice. Is it cancellable? Will it exit? Who’s listening? What happens if they’re not?</p><p>It’s like that feeling you get after a long night out. The music’s faded. The lights are on. And you ask yourself: <em>Did I leave the stove on? Did I leave a goroutine running?</em></p> <p><em>Second Nespresso. 7:41 AM. I’m awake now. The logs are clean. The metrics are normal.<br>And the ghost threads?<br>Gone. For now.</em></p><p><span></span></p><p><a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/1/" target="_blank">#1</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/2/" target="_blank">#2</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/3/" target="_blank">#3</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/4/" target="_blank">#4</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/asynchronous-programming/" target="_blank">#AsynchronousProgramming</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/channels/" target="_blank">#Channels</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/coffee-and-code/" target="_blank">#CoffeeAndCode</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/concurrency/" target="_blank">#Concurrency</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/context/" target="_blank">#Context</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/deadlocks/" target="_blank">#Deadlocks</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/debugging/" target="_blank">#Debugging</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/go/" target="_blank">#Go</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/go-programming/" target="_blank">#GoProgramming</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/golang/" target="_blank">#Golang</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/goroutines/" target="_blank">#Goroutines</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/message-queues/" target="_blank">#MessageQueues</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/peer-to-peer/" target="_blank">#PeerToPeer</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/performance-optimization/" target="_blank">#PerformanceOptimization</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/real-world-go/" target="_blank">#RealWorldGo</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/software-engineering/" target="_blank">#SoftwareEngineering</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/tech-blog/" target="_blank">#TechBlog</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://vortexofadigitalkind.com/tag/worker-pools/" target="_blank">#WorkerPools</a></p>