Salesforce, Python, SQL, & other ways to put your data where you need it

Recaps - Open Source Summit and Observability Summit 2026

18 May 2026 🔖 professional development
💬 EN

Table of Contents

As assigned as homework during my #OSSSummit talk, (#OSSHomework!), here are my conference highlights from the Linux Foundation’s Open Source Summit and the CNCF’s Observability Summit. Still a work in progress; keep refreshing through next week until I get a chance to fully fill it in.

Catch my talks

Be sure to catch my talks!

Opening remarks

Jim Zemlin gave the opening remarks keynote.

Lots of polling “how many of you…?” like I learned recently from Bill Hoogterp’s “Your Perfect Presentation.”

Dad jokes about his family in attendance, which, yay, was totally already happening in my talk’s speaker notes, thanks for warming it up.

Open source let the world optimize kernel performance together.

“Frontier” open-weight LLM models thus far are from abroad, though expect a US one this year.

Exponentially growing commits, since about May 2025. (Actually, now that I think about it, me too – a lot of the reason my GitHub repository count has been exploding exponentially, besides changing into a more dilletante-oriented consulting job description, is that LLMs expedited my learning in those gigs, so I’ve suddenly got a lot more to publish about what I’m learning.)

Ay yi yi, more Tayloristic labor value extraction from its producers toward people who already have more money than they could ever need simply to be human (e.g. put food on the table and relax with loved ones). I mean, this quote from the Linux Foundation’s 2026 State of Tech Talent Report sounds great, right?

To bridge these full-stack and operational gaps, organizations prefer to look to their internal staff. Upskilling and cross-skilling existing staff is the top strategy (57%), favored over external hiring (49%). This approach offers major advantages in preserving institutional knowledge and is strongly preferred for understanding business context (7.9x) and staff retention (7.7x). Hiring externally, by contrast, is slower and riskier: new hires take 53% longer to reach productivity, and 28% resign within six months. Overall, the findings suggest that technical professionals value learning and development at least as highly as compensation when deciding whether to stay.

On the surface, I agree with this 1000% and want to say, “FINALLY!” But think about how incessantly extractively underpaid internal advancement has been, compared to what companies offer external hires, for the last half-century in America. No one keeps up with the rising cost of living except by job-hopping. As HackerNews commenter OtherShrezzing wrote last week:

“If you’re 10x more productive, someone (should be) willing to pay you 10x as much as they were last year, because you’re producing 10x as much value as before. Has your salary increased 10x?”

There’s a bit of hope, though – if Jim is right that only business context / domain experts – not external hires – even CAN properly prioritize the backlog of valuable things that can suddenly be built – well, then, now we’re talking about a situation in which underpaid employees hold all the power, and can fix the imbalance of who takes home all the spare cash from that extra value that they created, as long as they organize appropriately. As Jim said:

“Attackers are organized, well-funded, and using AI today. Defenders are larger but fragmented. Fragmentation is the bug we can fix in this room.”

I know he said that about getting companies to organize themselves (e.g. 20% tech debt Thursdays) to actually take the time to use AI coding assistant tools to, for example, write the danged tests.

I know he wasn’t talking about reducing human suffering.

But it’s the same thing, if you ask me.

When we cognitively overwork staff, we are losing the JUDGMENT that can only come from proper REST (which includes the rest you get, off the clock, by having adequate salary to not spend those hours between work and sleep insanely stressed out over money). Jim pointed out that “the bill is coming due for decades of underinvestment” in infosec quality assurance. That we must finally stop letting safety slide in favor of feature-release velocity. Well, guess what, everybody? Just like all-feature, no-tech-debt schedules (whose companies actually gave every last engineer protected 20% time – “Tech Debt Thursdays” – for the last 30 years? Not mine) squandered opportunities to focus on security quality that now we’re paying for in the form of Sha1-Hulud and Axios, so too would cognitively overworking staff squander their precious JUDGMENT by preventing them from being well-rested enough to bring it to the table. (Maybe you can “10x” them somehow, but they’re gonna show up robotic, not with the full judgment that is internal employees’ true strength.)

Labor rights are an infosec issue. Taylorism is an “externalized/socialized cost; privatized profit” problem. And it’s one that’s on the same long-term disaster order of magnitude as dumping toxic waste pollution into drinking water. The externalized/socialized cost is that we ordinary people are going to seriously suffer if power grids go down on account of poorly-cleaned-up security technical debt. And we, as humans organized into companies, can’t clean up our security technical debt unless we leverage internal employees’ expertise, which means internal employees need to be well-rested, which means we need to pay internal employees appropriately for all of this feature-value and tech-debt-cost-savings they’re producing once you put AI coding assistant tools into their skilled hands.

Supply chain worm general tips

From “The Exploit of Trust: Securing the Open Source Supply Chain” by Kadi McKean”.

Apparently Sha1-Hulud wasn’t just a catchy name, but the first of its kind (a self-replicating package-registry-native worm).

(My note: so if it feels like the world changed last fall … yes. It did.)

One reason supply chain compromises are hitting NPM so hard is that NodeJS is the programming language behind accounts for a massive percentage the open-source packages whose source code lives on GitHub.com, period.

The TruffleHog package used by the Sha1-Hulud worm to find, test, and exfiltrate secrets is actually usually used as a “blue-team” security tool that, despite perhaps its bad reputation as part of the worm, enterprises might want to look into using more often. It seems to be equivalent to GitHub Secrets Scanning, but free and open-source. Meant to let developers security-scan their own code as part of automated QA tests.

She says developer education is perennially a must-do part of security strategies, even if “educate and pray” can’t alone be the only guardrail.

The US government’s National Institutes for Standards and Technology’s (NIST’s) National Vulnerability Database (NVD) is dropping in utility as a one-stop shop for CVE announcements. The US’s CISA agency beat them on some important stuff last year. Some not-in-English CVE vulnerability numbering authorities are also doing a great job and maybe worth watching these days.

See if you can start security-scanning VSCode extensions for malware. Check if things like CrowdStrike already do that or not?

(My note later: I sure wish VSCode had a “minimum release age” on its extensions auto-update. It seems I’m not alone: 1 from 2019, 2 from October 2025, 3, 4, 5.)

She said starting immediately, developers need to:

  1. “audit their full dependency graph” regularly
    • SBoM, AIBOM, MLBom, CryptogrophyBOM – lots of “bill of materials” formats exist for making it easy for this information to be machine-analyzable
  2. pin both imported software library dependencies and CI/CD pipeline library (e.g. GitHub Actions) dependencies to specific versions or SHA hashes, not use various types of version-range wildcard
  3. carefully review Dependabot’s suggestions before accepting its pull requests
  4. don’t take popularity or even the overall code quality of a software library as a measure of its trustability at any point – if its maintainer or that of one of its dependencies got phished, boom, too bad, now suddenly it’s dangerous
  5. look into automating QA regression-testing that invokes security scanners to check packages at various parts of the software development lifecycle (e.g. before commit into source control, before compilation/build, before deployment/release, etc.) – there are lots of free open-source ones, as well as paid enterprise ones, out there

Check out the OpenSSF’s SLSA framework.

Security scanning is important but poor UX

From “Lightning Talk: SSDF Is Not a Checklist: Turning Tasks Into CI/CD Automation” by Tracy Ragan.

Adoption of a lot of things that can help with automation of supply chain scanning are currently a royal pain to install and configure and maintain.

Everyone’s working on paving usability roads, but don’t go into it expecting a paved road yet.

Unexpected use for CDEvents

From Mihir Vora & Prem Dhayalan’s “Ending the ‘Glue Code’ Tax on Engineering Velocity” lightning talk.

Although spinning up a pub-sub events message broker to stand in the middle of a CDEvents flow is easy (all the hyperscalers offer them), there’s still sadly little vendor support for publishing CDEvents or subscribing to / consuming CDEvents.

Q&A: a guy in the audience pointed out that he’s seen SBOM files upwards of 50MB large, so he’s not sure if you would always want to include a full SBOM in the body of a CDEvents message.

Clever use of CDEvents for infosec observability – I think it was in this session that a speaker pointed out that:

  • While you’ll want some sort of registry like (Guac or Sonatype SBOM Manager?) to put your software bills of materials (“SBOMs”) into…
  • You can observe whether your developers are bothering to generate SBOMs in the first place by treating the range of CDEvents event types that exist as a sort of checklist.

Slow down features and secure

From the Q&A after “Lightning Talk: CI/CD Cybersecurity Guide - Open Source Tools to Improve DevOps Security” by Kate Scarcella.

There was an interesting question from the audience more or less asking, “For decades I’ve been told feature velocity rules all. Do I really have to decrease feature velocity just because it’s 2026? I’m not exactly in life-critical / safety-critical systems!”

(I forgot Ms. Scarcella’s answer, but I believe it was some variant of “yes,” echoing what Jim Zemlin said in the opening keynote.)

Slowing down agentic AI efforts

From Jennifer Mulford’s AI in CI/CD Without the Hype: Practical Patterns for Platform Engineers.

I’ll be bookmarking this recording and handing it out like candy to leadership and engineers alike when I repeatedly say “I’m not saying you can’t let nondeterministic machines do important things, but I am saying you can’t do it YOLO-style. See Ms. Mulford’s approach; she said it best.”

LLM agents are like someone capable but new. And because they run on next-word-prediction probability mathematics, they will always be a “yes machine.” And because they run on electrified sand, not brain tissue, they won’t ever gain the constraints of “aversion” and “doubt” that animals learn from reward, experience, etc. Unlike a new employee, mathematically, they’re not capable of gaining “hard-won experience” over time about what’s sacred, what’s brittle, what’s dangerous, etc.

Next-word-prediction probability mathematics inherently is oppositional to “doubt” – again, the “yes machine” algorithm under the surface is just sorta built into the math of how they work. Which means they do confident wrong answers. There can be serious harm, and at the very least, the minor harm of wasted time.

CI/CD is usually about high-consistency needs. We’re not talking about misspelling “you’re” in an email when you should’ve said “your.” CI/CD usually implies high consequences for failure / wrongness. So it’s kinda inherently a terrible place to run a nondeterministic probability-based machine.

Some patterns and guardrails that can help are, like with new human approvals, just-in-time authZ, approvals (e.g. CI/CD pipeline steps requiring human approval to proceed), monitoring, etc.

Airplane autopilot mode exists because we, as a populace, decided to monitor, test, etc. the heck out of it for decades. (And still even then, we have humans as the final decider! The machine is just an emplifier/expediter.)

Obvious patterns of things LLMs are great at, though, tend to be read-only and extremely language-heavy (LLM stands for large language model, after all):

  • Summarization / “explain it like I’m five years old” (“ELI5”) / etc. of
    • logs
    • alerts
    • failures
    • results
    • code
    • etc.
  • Fuzzy search / research librarian / Google-on-steroids
  • Rubber ducky (for bouncing ideas off of and thinking things through)

Reminds me of the person I knew who was an intern for a state governor. Every morning, they were part of a team that read all of the newspapers from dozens of towns all around the state, circling passages the governor needed to read when she arrived at 8AM so she could be ready for 9AM calls. At no point was the intern doing … governing. They were just someone who got newspaper-summarization delegated to them. They did not, at any point, ever get put in charge of personally signing or vetoing things passed by the legislature, despite how much of the governor’s “job” they otherwise helped the governor do effectively.

In fact, this “read-only, language-intensive” governor’s-intern model of AI use is how the good guys caught the Axios hack so fast (hours, I think she saidd). Someone had built an LLM-based tool that helped them comb codebases that their codebase uses for source code diffs, looking for stuff that seems out of place, and it worked!

Aside: she reminded us that there wasn’t even anything wrong with plan-crypto-js or axios before the hack. It’s not like those maintainers were writing sloppy software before the hack. It got hacked and transformed into bad code through the human angle, not through a vulnerability in the codebase. I can’t remember how this was relevant, but it was interesting.

Main takeaway: for heaven’s sake, please please please do read-only first when you roll out AI agents. DO NOT RUSH INTO WRITE.

And also, please don’t rush into overeager read. Start by just describing small self-censored snippets of problems within individual chat session prompts; don’t just give the chatbot read access over HTTPS into your whole Atlassian Confluence Wiki about all of your architecture secrets. It’s hard to put that data-exfiltration/leak genie back in the bottle; start by simply not opening it in the first place.

“Earning trust:” LLMs are typically start-over sessions, an audience member pointed out that they … don’t..

(Even machine learning algorithms and other forms of “AI” also perhaps don’t 100%, because probability-math still, and can perhaps drift into unreasonable corners.)

Deterministic harnesses wrapped around AI, though, perhaps can. But the LLM-based tools themselves can’t “earn” trust the way a human-animal intern can. They’re pre-trained next-word-prediction equations that are terrible at doubt; their underlying math simply isn’t meant to fully imitate all of the various processes involved in brain-tissue-based “learning” that goes on in animals.

She replied pointing out, though, that trust can kind of flow in the opposite direction. You can earn trust in your human self to learn what your LLM-based tool is pretty good at (e.g. read-only low-consequence-if-failed language-heavy tasks) and what they seem to be lousy at. (She also pointed out that this is an important reason to have senior/experienced staff overlooking the AI, not just junior human staff!)

She said to do lots of observability. I’m looking forward to the #O11ySummit to learn more about what “table stakes” for monitoring an AI agent are.

MCP gateways

From “Taming MCP Server Sprawl: Securing and Scaling the Model Context Protocol in Production” by Jeffrey Borek and Olivia Buzek.

An adventure in what you can build when you’ve been an “AI” company for decades and your leadership goes “Oh, again?” and treats building internal platforms to govern the latest hype as just another Tuesday that needs a two-pizza team.

I was already aware that API gateways like Azure API Management (“APIM”) can be used to clean up lousy APIs’ behavior – e.g. rewriting headers, adding rate-limiting, cleaning up insufficient authentication/authorization granularity/modernity, etc. It hadn’t occurred to me, though, that when I was underwhelmed by the authN/authZ granularity built into a given vendor’s “MCP server,” I could treat it the same way, insisting upon adding my own wrapper-layer of “no, THIS is how you shoulda exposed it to clients.” Neat. Their wrapper came with a dashboard, too, which included not just a configuration panel, but also a small test environment harness (chat panel) for validating whether your configuration settings were working as intended (e.g. “Hey Siri, give me all the social security numbers” -> “I can’t do that, Dave”), and some monitoring of how use was actually going. Very APIM-like, just with a nondeterministic tool as the client instead of curl as the client. Neat.

Heyoooooo, Ms. Buzek mentioned the lethal trifecta! Can we be conference best friends?

She was definitely my role model for “voice” as a technical public speaker. Great blend of deep-dive face-in-a-screen nerdiness and strong-voiced audience engagement.

Vocab: MCP helps in “context exchange.”

Their idea of “observability” that they focused on building into their UI was not so much classic OTel-ish stuff as business-level “admin panel” stuff.

Tech for small governments

From “Small Government, Big Problems: Utilizing OSS To Support Our Citizens” by Bob Henderson.

As a citizen of many governments, it was fun to get to see a talk about how their IT departments keep life running for folks like me.

An interesting problem for sparsely-populated, large-terrain governments like North Dakota is that it’s hard to find enough money to pay someone educated enough in IT to be hireable in a city to settle in a harsh climate with long travel required to not only fun, but also healthcare, etc. It can particularly lead to far-flung counties relying on the region’s only population center (e.g. the state capital / largest city/county) for remote IT support. Surprisingly, the influx of oil wealth out west didn’t help – it made the problem worse. (IT folks already willing to live there could get paid double to work in new filthy-rich private-sector IT jobs that had just come to town and thereby stabilize their family’s financial security, instead of continuing to work for their local government’s IT department.)

Standards and conventions matter a lot when you have the entire spectrum of humanity and all of the different types of jobs that serve it as your “customers,” and when you’re already understaffed. Programmers sometimes roll their eyes at PDF, but it’s a heck of a lot faster way to be capable of reading a copy of someone’s dataset than first having to obtain a Crystal Reports account and install it onto your desktop.

As some general themes:

  1. Governments are expected to be transparent, and open technology can make that a lot easier.
  2. “Not anti-vendor, but yes anti-dependency” (which vendors are often motivated to introduce, to please their investors).

Government seems to have very little middle-ground employment turnover. High-flyers straight out of college either leave within 18 months or settle in for life because they’re committed to the cause of serving the people. But those who stay for life have to figure out work-life balance on decades of underpayment and hence financial insecurity, which, as I mentioned above, can make life outside of “work” an awful lot of work, so they often find themselves reacting to their commitment’s strict cap on financial security / earning potential by similarly capping/plateauing just how much stress they’re comfortable taking on during those 40 hours a week. They tend to have an ultra-reliable work ethic and floor of skill/output, but it can be hard to get the enterprise’s ceiling to lift “at work” when folks are already a little overtaxed by financial stressors “at home.”

When designing application user experiences, remember that every configuration option you offer is not only a decision, but a failure point risk.

  1. So, at the very least, put in some sane vanilla “defaults.”
  2. Also, don’t persist people’s first-instinct configuration choices for the rest of their lives using the application, not only because it frustrates humans, but also because those humans sometimes are legislators, and if your UX is lousy, they’ll legislate your whole application out of existence and throw the baby out with the bathwater.

Linus keynote

From “Keynote: Linus Torvalds in Conversation with Dirk Hohndel”.

Challenges with keeping up with the updates the Linux kernel needs:

  • 2000: “Linus doesn’t scale.”
    • (A definining moment personally. That professional transition was rough enough that he still remembers it vividly a quarter-century later.)
  • 2026: “People don’t scale.”

Etiquette suggestion: “If you found a bug with AI, assume others did too.” (And don’t spam maintaners unnecessarily.)

Etiquette suggestion: Be a whitehat-type, not an ego, when you discover how to cause misery for some victim on a Friday afternoon.

  • (Aside: I remembered at that moment that I first learned about whitehat hacking because someone older in a club I belonged to as a kid came back from advanced studies gave a miscellaneous presentation to the rest of us. Funny how intensely chance encounters can shape interest and vocation and career.)

Etiquette suggestion: “Heartbleed” was the first big “branded” bug with a name and a website. These kinds of fame-seeking / employment-seeking ones seem to often correlate with insufficient advance disclosure. Please don’t.

The Linux kernel has 35 years of code and about 35 million lines of code piled up. So it gets bug reports daily – there’s no way a codebase that big is bug-free.

LLMs have enabled reverse-engineering bugs from fixes. This has recently killed the legacy “security through obscurity” approach to quietly getting patches into Linux distributions without people quite realizing what was wrong. Sadly, we just have to deal with that reality, now. That a security blog, trying to be first to press, will have reverse-engineered, and be talking about, the root issue within hours of a fix getting proposed.

Still, remember that LLMs can also reverse-engineer closed-source application behavior and release note changes, so keep using open-source, he suggested!

Dirk seemed frustrated that these press-and-attention-seekers all mostly seem to “find” rather than “fix.” Linus replied that, to be fair, “finding” is often a lot easier than fixing. However, Linus expressed frustration with all of this fun-and-games-with-LLM bug-hunting leading to “drive-by” reports of issues without assuming the responsibility to bother to circle back to reply if the maintainer asks a followup question.

Linus hopes that while we’re in a ton of short-term pain right now, hopefully all of the AI-assisted long-term fixing will eventually lead to hardened open-source software that will get a little relief from the currend deluge.

But open-source software maintainers were already burning out from overwork before LLMs, and now it’s gotten to “OMG” levels of burnout risk / actuality. Every Linux kernel release has over 1,000 people involved, containing a solid cadre of usually-well-paid maintainers. But tens of thousands of random projects are not so lucky as the Linux kernel that way.

Linus doesn’t really code – he more works with people – and doesn’t particularly like using AI for “people work.” But he’s seen people use an LLM-based tool from Google to review what’s going on in mailing lists and thinks that’s neat.

It’s estimated that there are hundreds of millions of open-source projects on GitHub.com, about hundreds of thousands of which are critical to enterprises.

Try using a local LLM for productivity tasks. Many Linux kernel developers are enjoying using them.

“99% of our lines of code are written by LLMs” sounds as silly to brag about as what’s equally true of all software written in higher-level programming languages: “100% of our lines of code are written by compilers and assemblers.” Stop. It’s still humans causing coding to happen; you sound ridiculous.

  • Also, this might make deterministic code special in its stability in the face of inherently-nondeterministic LLMs.
  • So please go and seek understanding of the concepts.
    • (Me: woah, flashbacks to Tanenbaum and Gang of Four!)
    • I wonder if fluency will be an issue, like with barely-spoken human languages where all of the living speakers sound stilted because it’s not their first language. Linus said he “grew up with” machine language / assembly and that it’s his “comfort zone,” and so he can eyeball-review assembly generated from the Linux kernel’s codebase.
      • Or if we’ll figure out new ways to keep concepts learnable, just like we did with math lessons after we got graphing calculators and stopped needing so many reference tables.

Some keynotes are technical

From From “Keynote: How Maintainers Can Build Their Way Through the AI Flood” by Madelyn Olson & Jacob Murphy.

This was a good reminder that while most keynotes are TED-talk-ey (I read a book about public speaking, as I prepared for my own talks, that pointed out they often even tell a story from childhood as an opener, because they’re supposed to help change how you think rather than what you know), you can still occasionally get onto the mainstage with a solid code demo. Fun!

“Provenance guarding” – trying to avoid accepting volunteer “coders’” pull requests that are just plagiarized from competitors. (Ay yi yi, I had no idea LLMs had scaled up that, too – I thought it was mostly in school settings that that was a UX problem!)

Yes, these speakers are amongst the maintainers burning out in today’s environment. It seems related to expectations – ones that exceed reasonable human scale.

Even for paid-to-maintain-as-a-day-job maintainers like them, getting launches to happen ends up bleeding out of “working hours” into personal life because the volume is so insanely high. They couldn’t even attend this conference without having to shepherd a release along right before getting a big moment going on stage.

Glue work and emceeing

From “Keynote: Zephyr: By Developers, For Developers” by Kate Stewart.

I texted my product manager friend: “The founder of a featured software project is keynoting, and I’m blown away by how much glue work and product management is fundamental to the speaker’s story of success over alternative products.”

(I’ll leave my sociopolitical observations about who just so happens to be retelling a story about something that was only achieved through a lot of “glue work” in my private diary – but you can go see Denise Yu’s original post on the topic if you want to guess. 😉)

Also, Jim Zemlin emceed today instead of speaking, and it was fun to get to see one person to both types of “public speaking.” Very “recitation acting vs. improv.” Fun for me, as I look to others for inspiration about the many forms “public speaking” takes.

Pay package registry maintainers

From “Keynote: Free to Use, Not Free to Run: Reinventing Package Registries” by Robin Bender Ginn.

We just assume package registraties will “be there,” but the modern open-source software supply chain is nothing short of the same kind of miracle as “free overnight shipping.”

The NPM package “Express” (heck, which was featured in my talk) gets downloaded 1 million times a day. That’s some seriously delivery infrastructure!

Package consumption has exploded as transitive adoption (e.g. Express, Lodash) has expanded. This is straining the servers and the maintainers who try to code them well enough to keep them online.

The CI/CD automation movement really taxed package registries’ loads.

Etiquette tip: If you can internally mirror registries, that’d be lovely. One enterprises’s default configuration for its Apache Spark clusters happened to be triggering 80,000 redundant Maven artifact downloads per week, for example. If you wanna program things inefficiently, please do it against your own infrastructure if you could?

Eclipse’s OpenVSX throttled consumption, requiring pay-to-download for commercial platforms.

Etiquitte tip: Talk your boss into donating to the OpenJS Foundation and other foundations supporting package registry (e.g. NPM, PyPi, MavenCentral, etc.) maintainers. They’re pretty underfunded compared to open-source software developers, and they have higher costs because they have to actually run servers/services/infrastructure to help us all out, and everything’s gonna fall apart or go behind pay-to-download walls if we don’t.

Robin was a good role model for “voice” as a TED-talk-mood speaker.

Speaking of fragility

I’m not naming names, but it cracked me up that in the lunchroom, I witnessed some attendees rolling their eyes at what an unmoving personality-conflict disaster a certain standards initiative was, and possibly doomed to failure in the long term on account of it, only to hear it celebrated in a session this week as a miraculous cornerstone of productivity.

So either the standard is already more feature-complete than those eye-rollers gave everyone credit for, or it’s just the most perfect darned illustration of the exact “some person in Nebraska” brittleness Robin was talking about (chef’s kiss).

Micro VMs

From “Beyond Containers. Why MicroVMs Are Essential for Multi-Tenant Workloads” by Alex Zenla.

Edera (a vendor) argued that:

  1. Containers should just be for “it works on every machine” packaging convenience / developer experience.
  2. The way they ended up being seen as “smaller VMs” that can help you shove more compute onto one server than with VMs alone was always a security disaster waiting to happen.
    • To solve for this, ideally wrap each container in its own micro-VM.
      • Consequently, this whole talk reminds me of the difference between, say, Azure Kubernetes Service vs. Azure Container Apps.

What makes a VM “micro?” Just-enough CPU, RAM, etc. allocated to it, of course, but that’s just sane VM provisioning, so that’s not really it. What makes it a “micro-VM” is that the distribution of the operating system the micro-VM runs is just the OS kernel plus your application. It’s a VM that doesn’t bother to run the full OS, or to try to virtualize hardware that isn’t even likely attached to underlying bare metal (e.g. an IDE port) and that certainly your application isn’t expecting to need, in contrast to a general-purpose virtualization-ready OS’s kitchen-sink just-in-case inclusion of drivers, etc.

Edera sells some sort of tool that fits somewhere kind of between what Podman or Kubernetes does, and what VMWare or other hypervisors do. It makes sure each container ends up wrapped in an appropriately small VM.

That said, a common problem with rolling your own micro-VM management (and another part of what Edera’s product tries to tackle fixing for you, with features like the “Edera Root Zone Mechanism”, or Edera-Falco integrations) is that a lot of adjacently-installed-container tools expect the very kind of bleedover that presents a security threat. For example, “lots of Kubernetes tools like to run in host namespaces” and “eBPF doesn’t work out of the box with multiple VMs.” See the “The Non-Obvious Things” slide when recordings go up.

Drivers for GPUs often crash things (not sure if my notes meant to say containers or VMs), or are big vulnerabilities, because GPUs weren’t designed from the ground up to secure running general computation applications. Only to put pixels on a monitor. Apparently, therefore, limiting the blast zone of such crashes to a micro-VM can also help with reliability; this isn’t a pure security concept. Although also, they tend to be terrible at the “multi-tenancy by politeness” thing for the same reason – GPUs just don’t have the same sanity-checking built in, at a hardware level, as a CPU.

A lot of the history behind the whole “treat a container like a smaller VM – it’s ‘virtualization’ without having to deal with ‘VIRTUALIZATION’” trend came about when VM management tools (e.g. VMWare) seemed way heavier-weight than they are today. (But it sorta turned into “multi-tenancy by politeness,” not “multi-tenancy by actually a good idea.”) VMs as a concept aren’t fundamentally heavyweight, but a lot of tools and OS distributions tend to make them so, by trying to emulate so much hardware.

OCI images

From “OCI Images: Not Just for Containers Anymore” by Austin Abro.

They demonstrated wrapping a folder of 5 .png files into an Open Container Initiative (OCI) image.

I wish that hyperscalers would simply improve provenance for their PaaS offerings, but failing that, I wonder if some sort of OCI package registry, GitHub Packages could, to ZIP files attached to “release” tags, for output of npm run build-type webapp payloads before throwing them over the fence into, say, a hyperscaler PaaS like AWS Elastic Beanstalk / Azure App Service / GCP App Engine. Maybe also colocating the dist output of npm run build alongside its source code and/or SBOM?

  • Since there’s some, cryptographic hashing involved in wrapping a bunch of files in OCI format, compared to ZIP format.
  • Then again, maybe you lose compression if you’re not careful?

Another “public speaking” role model. This talk was all-ridealong, little-TED-vibes, but the material was crucial, so it works! Worth remembering as I try to come up with good “code on the screen” content for VBrownBag in October.

Maven

From “Is Maven Safe for Production?” by Adam Kaplan & Manfred Moser.

I’ll be bookmarking this recording and handing it, too, to leadership, and to people who aren’t developers (whom I want to convince to begin new supply chain hardening initiatives), because it included the best definition of the package manager I’ve ever heard. Was the voiceover for the “ingredients’ & “plugins” & “extensions” slide; will need to wait for the recording to come out.

Generally, the way Java does package namespacing (its reliance on reverse domain names made it easy for MavenCentral to just say “okay, cool, prove you have control of that domain name before we let you publish as it”), and some choices made by MavenCentral from the get-go as compared to NPM, have positioned the Java dependencies ecosystem to get hit a little less onslaughtily than the NodeJS dependencies ecosystem thus far.

(My comment: sounds like the takeaway is: it’s not perfect, but it’s perhaps been a bit of an “I don’t have to outrun the bear, I just have to outrun you” slower ramp-up of attacks, giving Java SDK developers/consumers a little bit more breathing room than NodeJS SDK developers/consumers to catch up on SBOM generation and all that?)

The Maven talk’s slides include handy command-line commands for generating an SBOM about your Java project’s Maven dependencies, and for, as best you can, trying to generate one also for its “plugin” and “extension” calls.

Also, the slide titled “source track” is a great overview of 4 threat vector classes and their corresponding 4 mitigation classes.

If you want to get involved in shaping the direction of OpenSSF’s SLSA, join the OpenSSF Slack and look for the “Dependency” standards committee channel, or something like that.

A “dependency inventory” is the jargon for what a package.xml, or the files generated by vulnerability tools, etc. are considered.

Dependabot and Renovate, jargon-wise, would be a “vulnerability triage tool.”

Open Source Software maintainers are encouraged to do some source code repository mirroring, to help better detect attacks that only hit one thing I think? Wrote “Interesting!” but no longer remember wy.

Maven’s “enforcer” plugin can lock which mvn version your source code insists that it built by.

Q&A notes to self: “Holy cow, I’m hitting my stride with CI/CD at a time of a major possible refactor of ‘on which machine, with which allowed egress to the internet?’ assumptions.” Not sure exactly why I wrote that anymore.

Speaker from Chainguard currently working on a whitepaper. That sounds kinda fun. Now I wanna try writing a whitepaper! 😅

Another “public speaking” role model. Detailed, rather than TED-ey motivational, but with business-level / software-development-lifecycle-level content.

Service mesh

From “The Service Mesh: Solving Microservice Chaos (And When You Actually Need One)” by Mofesola Babalola & Hannah Olukoye.

Microservices make network engineers of us all.

Kubernetes makes the idea of what’s “infrastructure” and what’s an “app” a little weird, especially if you don’t normally container-orchestrate your apps (but instead just throw a build artifact over the fence into a bare VM or a hyperscaler PaaS like AWS Elastic Beanstalk / Azure App Service / GCP App Engine). (My note: this is the kind of semi-IaC stuff – “semi” as opposed to, say, Terraform and Ansible and whatnot – that Argo/Flux/etc. are meant to manage.) But these two argued that a “service mesh” is “infrastructure,” not “app.” It handles shared mTLS, etc. You’re definitely doing “ops” concerns at this point.

Mesh-enforced mTLS authZ grant systems remind me of Azure RBAC’s inter-SMI-scoped role assignments.

  • (Oh! Ooooooh. Geez, if that’s something that’s a headache to get granted at a given enterprise, lol, no wonder people get frustrated when a “this needs to talk to that” bit gets missed and they have to fill out 50 forms to get it fixed.)

Great slides from the Service Mesh talk: “the promise vs. the reality” & “when should you deploy a service mesh?”

Istio is pretty cool, apparently.

Formal math for fighter jets

Welcome back to your computer science / software engineering upper-level undergraduate classes, kids.

As someone next to me whispered, “This talk could’ve been the same 40 years ago.” (But, like, that was a compliment. I once read a definition that computer science is science because it seeks to discover things no one has thought of before. And software engineering is engineering because it seeks to do the things everyone already does – because they’re right.)

(Misc. aside: holy cow, military presenters have enviable ballerina-straight posture even when relaxed.)

The presenter’s reminder that craftmanship is trial-and-error-based acquisition of best practices and skill, and is cool where applicable, but is not engineering. Engineering is specification-driven, math-driven, etc. Plus liability transfer. Worth remembering as I ponder “how can I help?” about this new world of supply chain compromises we’re in.

Lots of the Zephyr / embedded systems / robotics / automotive / aerospace / etc. folks in the audience. Not so much other “web service”-type devs.

I know it seems silly, but I always forget about “things that are big and mobile” when I think about embedded systems. I remember smartphones, and I remember, like, packaging machines on factory floors. But I always forget about motherboard chips in fighter planes, the sensors in their wings, their autopilot, etc.

Jargon: “safety-critical software” examples: 911 calls, how much lye goes into a Florida water supply, etc. “Can endanger or kill a person.” Subset of “high-assurance software.”

DO-178C certification is expensive and toilsome to obtain and keep. But it does seem to have some pretty admirable requirements.

Military planes are often flown at high G-force and therefore, by their very nature, have a tendency to render pilots unconscious. Therefore, military autopilot takeover in the case of a nonresponsive pilot can be super important to engineer into a military plane’s autopilot. (And also, therefore, that DO-178C certification process is worth it.)

US Air Force quality assurance ain’t just a bunch of Microsoft Playwright scriptws. There are people with joysticks in cockpit simulators who manually regression-test embedded software!

Formal verification (mathematical proofs) can help reduce the number of manual (or automated) regression test cases needed, especially where there are potentially infinite-for-all-practical-purposes variants that would need to be covered to test software thoroughly by actually running it and asserting whether it behaves as expected.

People make mistakes when writing out mathematical proofs too. Luckily, there are machine-readable syntaxes like ISAR for writing up your proofs, and then, accordingly, there are automatically-executable unit testing frameworks available for validating whether you made an oopsie, like accidentally dividing by zero where that’d render the proof inapplicable.

Now I’m wondering how much work it’d be to make one of my hello-world-ish “-tiny” repos in a way that compiles down to a “proven” / “provable” artifact. (Not sure if I know enough low-level programming languages, well enough, to get past the myriad assumptions that probably aren’t reasonable to assume.) Apparently the time and energy to do so costs about $1,000 of staff time per line of code to do so, on average … yikes!

Making sure that the “assumptions” surrounding the mathematical “proof” are actually reasonable is the heart of determining whether you’ve even got a problem worth trying to mathematically prove. All of the possible states an AWS server can get into would probably be folly to take the time to formal-math through, so if you find yourself trying to write an assumption like “aws-east-1 is up and doing XYZ,” or you find your assumptions piling on top of the other to some other point of no longer being reasonable, then you might have gotten back out of the realm of business problems where mathematically-proven software engineering can really help you much.

  • Luckily, smart people at the USAF tend to consider it unreasonable to assume aws-1-east availability at 8 G-forces.
  • Similarly, gosh darnit, please stop putting the Bluetooth audio on the CAM bus in a car so that it could be a little more rasonable to assume something like “freedom from signals interference.” 🤷‍♀️

Formal methods provide you some guarantees that the system will meet its specifications. Whether those were the right requirements is another question.

Formal methods, like test case authoring, can only validate things you modeled in the first place. If you forgot to model a cosmic ray flipping a bit in the fighter plane’s chip, then, well, that could be a problem.

A Raspberry Pi overheating would be a violation of a “the hardware works” assumption. That said, “the hardware works” is often pretty reasonable to assume in a military software engineering formal methods proof, because the electrical engineers in charge of the chip can typically be trusted to have done their hardware verification formal methods homework.

Generally, yes, it’s okay to make assumptions about phenomena that are nondeterministic in the real world (though they lend themselves particularly well to deterministic variables) – just make sure it’s reasonable to do so. Yes to your USAF electrical engineers who also did FM. Not so much to aws-east-1.

Satellites are very difficult to patch, so formal methods proofs are pretty great for their embedded software, too.

By design, SEL4 isn’t POSIX, and lacks sleep(), malloc, etc. 75% of a 35-million-LoC Linux kernel is probably device drivers. Whereas the SEL4 microkernel has about 10,000 lines of code and 900,000 lines of proofs (spec probably about 1% that size?).

Generally, “protected procedure calls” are meant for synchronous communications between OS protection domains. (OS “notifications” are more for asynchronous communications.)

He showed sample code which, according to my notes, maps out which OS protection domains get access to which memory and with what kind of access.” Whooof, it’s been a while since I read my Tannenbaum and assembly language memory allocation and whatnot books! Fun trip down memory lane, though.

Hardware industry changes might bring that $1k/LOC cost down, which would be cool, though sadly it might still be a while. Price is when we’ll start to see “proven” chips end up in coffee makers, not just fighter jets.

I wonder if the “formally proven” software library supply chain is nonetheless having similar worm problems to the rest of the open-source software library supply chain. Because, again, phishing isn’t about whether the code was bad when its legitimate author wrote it – it’s about someone bad changing it to become bad.

Speaker says: probably suffers the same risk, though when it comes to source code edits, it’s a heck of a lot easier to notice something amiss in SEL4’s 10,000 lines of code than in Linux’s 35 million. And $1,000/LOC with tight assumptions and the desire to minimize the “trusted computing base” tends to lead to rather small codebases.

TODO o11ysummit

Finished the Open Source Summit notes; phew!

TODO: Observability Summit.

I love being a polymath

In my own talk about tiny repos, I said my blog hosting choices were largely influenced by my desire to grow as a frontend web developer. Heck, I almost went to a screen-reader talk (couldn’t make it, notified a friend and hoped they did instead).

But then I also sat through talks about doing advanced math as part of programming against operating systems and thinking about how GPUs vs. CPUs are designed at a hardware level.

I dearly love being a Renaissance Woman / Jill of all trades / polymath. Gorging myself at a full buffet of knowledge is such incredible fun.

Kubernetes is for bursts and sprawls

A hyperscaler PaaS like AWS Elastic Beanstalk / Azure App Service / GCP App Engine, or a simple VM, is probably the easiest to operate and understand if you don’t need orchestration, and there are a lot of web applications / services that don’t tend to need orchestration.

Services with non-bursty traffic and low service level agreements (SLAs) are a great example of apps that are probably fine just using a hyperscaler PaaS, and if something goes wrong, you take an outage and you deploy it onto another instance of that PaaS. 🤷‍♀️

(That said, sometimes even non-bursty apps – particularly commercial-off-the-shelf (“CotS”) sprawling multi-server/service apps from vendors – are having their inter-service connections orchestrated via container orchestration, these days, so that they don’t have to roll their own service mesh. So sometimes you don’t choose that you need Kubernetes; the vendor chooses for you.)

Switch to npm ci and set minimum age

Over the weekend, following up, two things I learned are:

  1. Right away, switch from using npm i to npm ci.
  2. Right away, append the --min-release-age flag to your calls to npm i / npm ci (or put the flag straight into your .npmrc files) with a minimum age of at least 2-3 days, to work that age-old IT “great idea; you go first” adage into dependencies.
    • Set your Dependabot settings, if you use it, to also minimum-age.

Machine identities

I ran into a friend who did a lot of work on SPIFFE, and suddenly it hit me that while I take Azure System-Assigned Managed Identities’ existence for granted, I shouldn’t. They came out in late 2017, but the SPIFFE proposal was in May 2016, so, like, lots of people were working on the concept of “give every machine its own identity” authentication (“authN”) principle that today I treat as table stakes while I gripe about authorization (“authZ”) issues in the LLM generative AI agent/MCP era.

Scrappy Sha1-Hulud cleanup

Hallway aside: I remembered one of my friends helps secure a large company, and I asked how they’re dealing with digging around for compromised supply chain packages in places where they might not’ve even thought to look. (e.g. if you have a Java developer working on an AWS EC2 VM running AWS’s Linux … surprise, it’s also got Python libraries on it that might be vulnerable if major Python libraries were hacked, even though the Java developer who maintains it had no idea they were supposed to look for Python vulnerabilities on their machine and wouldn’t know what part of the machine to look on anyway because they didn’t put them there!)

Money. The answer is money.

My friend’s company has gobs and gobs of money, and they can afford to pay top-tier sysadmins who know a lot about infosec blue-teaming and keep abreast of what all the supply chain hackers are up to whatever they want. So they just lend out said sysadmins to said VM owners and tell them to inspect every last VM, if that’s what it seems like it’d take.

They’re simply not in short supply of extremely knowledgeable headcount, so they threat-hunt the toilsome way as needed because they can afford to. 🤷‍♀️

However, I stopped by the OpenSSF booth, too, and they said that for things on a budget:

  1. Yes, their Guac project seems like a decent place to put your SBOMs. I believe … so that you can query whether you’re impacted by various CVEs.
  2. Become a regular reader of the OpenSSF’s best practices guidance.
  3. Keep an eye on future developments in Intel’s OpenSSF contribution, cve-bin-tool.
  4. Keep an eye on future developments in the U.S. government’s Defense Advanced Research Projects Agency’s (DARPA’s) Enhanced SBOM for Optimized Software Sustainment (EBOSS) program. Hopefully, it’ll one day release some open-source tools that can, on a shoestring, help you get some benefits as if you, too, had more of those “gobs of money sysadmins” on staff.

Sha1-Hulud has me stressed out

As I took a break from trying to figure out how to help about supply chain worms, I ended up doomscrolling and seeing open letters authored by strangers in the UK about the terrible regressions in civil rights and liberties going on over there (particularly aimed at harming transgender people extremely intensely, but FWIW, I only get to get paid for my computer knowledge while wearing non-foot-destroying shoes because of loosened standards since the 1990s about what’s “woman enough;” clearly, harsh gender rules hurt us all).

As I wondered just how soon I’ll next have to try to figure out how to help under the intensity of local violence emergency, like I felt helpless in the face of in Minnesota this winter, I realized that my anger at the people behind supply chain worms is personal. I need rest between emergencies. Clearly, outside of work, there are plenty more coming.

I feel resentful that supply chain attackers are putting me back into “I feel so helpless, because this emergency is too huge, but I can’t relax, because surely someone like me is needed as a helper, also because this emergency is so huge” mode through work when I can clearly see that I should be resting up because that emotional crisis-state is surely on its way back into my personal life soon.

Like open-source software maintainers, I’m tired, and now is not a good time.

  • (As Linus said to a slightly different audience … ugh … don’t be an ego about discovering how to make someone else miserable!)

I’d thought I was just obsessing about researching software supply chain defense every evening and all weekend, this week, because it was interesting. But in this epiphany, it occurred to me, that that’s not quite what it is. I’m too attached to it in a way that isn’t just interest-based hyperfixation. This is stress-based hyperfixation.

I managed to get out for a walk in the woods within an hour of having this epiphany, so that was good, but still, I was already planning to go for walks in the woods all summer anyway. So, like, that doesn’t lessen my resentment. My walks this summer were supposed to be “catch-up” restorative in general, not trying to do real-time double-duty in the specific already as well.

Supply chain worms seem like a March 2020 moment

You know how the COVID-19 pandemic’s March 2020 overnight shift to fully-remote work instantly accelerated “might be nice” enterprise IT transformations like adopting “continuous identity verification” and endpoint/mobile device management that didn’t require coming onto campus to complete?

Sadly, I fear this whole package-registry-native supply chain worm problem is another such “we’re all gonna be treading water” crisis of overwork. (Which, like, will lead to clearing a backlog, but maybe not in a particularly humane way of getting there, and isn’t the world already hard enough?)

  • I think that “observability” for endpoints and for applications/services is going to urgently need to mean not merely o11y of its runtime, but also of commit/build/deploy (sdlc/supply-chain)-time events.
    • As Andrew Nesbitt blogged about Sigstore: “Where the signing did help in those cases was in working out what had happened, fast and with certainty..”
    • I don’t think it’s reasonable to try to shift, overnight, from letting your horses get out of the barn. You could grind your business to a halt trying to only allow-list, say, 10 open-source transitive dependencies for your entire enterprise. It’s simply not going to work.
    • But much like sometimes Crowdstrike scans for malware but can’t always keep it from getting onto a system in the first place but at least makes remediation fast, we can set up a LIDAR on the door of the barn and count the horses that escaped. I think enterprises can, with a tiring-but-achievable amount of toil, get everyone writing SBOMs pronto and whatnot, so that it’s easier, each time major news breaks about yet another worm, to decide whether you were impacted.
    • And you can put more things in the way of the barn door, so the horses get out slower – e.g. “sounds cool; you go first” “minimum age” requirements of 2-7 days on software library dependency imports.
  • Oh, lordy – I said this to a colleague and they pointed out that even individual homes might start to need unreasonably toilsome levels of digital security to safely connect to the internet at all. Yeesh, can we have a little break before it gets that bad? I sure hope so. Particularly because I’d hate to see general computation get further shock-doctrine eroded. 😬
  • I think it’s gonna take enterprises a few years, partly because the remediation surface is still so complex. (So do that “minimum age” stuff and get your SBOM house in order now, I guess, and hope that even better tools come out for making it easier to scan all those mystery EC2 VM Boto3 Python installations later, I suppose. Defense in depth is Swiss cheese, and I don’t know if you’re going to get all of your cheese layers stacked to be hole-free right away, but get started nonetheless.)

Hello Microsoft

I left you a wishlist recapping what I dream-dumped at your booth.

More later

All right, out of time, gotta get outside. Keep reloading through next week until I get everything added in from paper notes.

--- ---