Salesforce, Python, SQL, & other ways to put your data where you need it

Need event music? 🎸

Live and recorded jazz, pop, and meditative music for your virtual conference / Zoom wedding / yoga class / private party with quality sound and a smooth technical experience

Buy then build

05 Feb 2025 🔖 devops prompt engineering
💬 EN

Table of Contents

I finally gave up on saving the planet by abstaining from LLMs and took a peek at Google NotebookLM. I have to admit, it’s done an amazing job helping me organize my thoughts about DevOps and SaaS (software as a service).

Hot take: there’s almost no such thing as “buy vs. build.” It’s “buy and also build” or “build and build more.”

Not that buying the core functionality is a bad idea! Please don’t reinvent the wheel when you don’t have to! Software developers use 3rd-party programming libraries to make repetitive tasks easier; there’s nothing wrong with using SaaS tools to accomplish the jobs they’re good at.

It’s just that … you’re still going to have to build the rest.

This might seem obvious, but I couldn’t put the details of what that means for you, the SaaS client, until I got a little help.

Here’re the takeaways.


Podcast

Firstly, check out the 17-minute podcast that NotebookLM made me!

It TL;DRs this better than I ever could, diving into a case study about a pizzeria named “Pizza Time” that just bought a new Shopify-style SaaS.

I’ve included an audio player above, but here’s the direct link to the MP3 file, if you prefer.

(I’ve found, on my devices, that the direct link is more likely than the embedded player present playback speed controls.)

It’s an absolute work of art, as a “tech talk.”

100% worth a listen.


Helpdesk articles

If you hadn’t noticed … I’m a bit wordy. (Even NotebookLM backhandedly complimented me as “detail-oriented” when I fed it this blog.)

I constantly bubble “what if?” and “aha!” syntheses.

I love to communicate them in the form of a helpdesk-style article / FAQ, but struggle a lot to make them short enough to hold a beginner’s attention.

Luckily, I didn’t have to this time.

Check out what NotebookLM made me (further cleaned up as I wrote this blog post, but it gets all the credit for the structure and prose.)


For the business owner

Congratulations, you’ve got a new online ordering system!

  • (click expander icon at left – it’s a rightward-facing triangle – if the article below is not showing up)

Great news! You’ve invested in a new online ordering system to make running Pizza Time even easier! This system will help you take more orders, keep customers happy, and streamline your operations. It’s like getting a brand-new, high-powered pizza oven that can handle any rush, but to get the most out of it, you’ll need to make sure everything is set up correctly. Think of this new system like a race car: it’s powerful, but it needs the right pit crew to keep it running at top speed!

This article will cover the things you need to know, without getting too technical, and will tell you what to discuss with your tech team to make sure you’re making the best use of your new system.

What you need to know

  • Your online ordering system is like a second restaurant:
    • It’s not just about taking orders; it’s about making sure the orders are right, they get to your customers on time, and that everyone is happy with the process.
  • It needs to be set up and managed:
    • To work correctly, your new online system needs a little bit of management, just like your physical restaurant.

What to discuss with your tech team

Here are some key points to discuss with your tech team.

When you do, make sure they know to check out the corresponding sectionsin the technician’s guide (linked in parentheses below).

Representing your pizzeria in the system
  • Menu and pricing
    • You need to ensure that your menu and pricing are accurately reflected in the online ordering system, including new specials, changes to ingredients, and seasonal prices.
    • Talk to your tech team about having an easy way to make changes and ensure all prices are up to date.
    • Technician’s guide cross-link: “Managing Configurations: Menu and Pricing
  • Delivery zones
    • Make sure your delivery zones are correctly defined in the new system. This is very important so that customers in the right areas can place orders, and the delivery drivers know where to go.
    • Your tech team should also ensure they are prepared to make changes to the delivery zones when needed.
    • Technician’s guide cross-link: “Managing Configurations: Delivery Zones
  • Customer loyalty program
    • If you have a customer loyalty program, discuss with your tech team how it will be integrated into the online ordering system, and how to make sure points are awarded correctly. Make sure the process is seamless for your customers.
    • Technician’s guide cross-link: “Managing Configurations: Loyalty Program Rules
  • Payment processing
    • Discuss with your tech team how the system will process payments. This will help ensure your customer’s card information is secure. You need to be sure your customers have a fast and reliable payment method.
    • Technician’s guide cross-link: “Automated Security and Compliance: Payment Processing
Keeping the system working
Computers
  • Monitoring
    • You need to ensure the system is always available and working correctly.
    • Talk to your tech team about setting up monitoring probes so they can get an alert if the system isn’t available.
    • This will help make sure your customers can always place orders and that you don’t miss sales because of technical issues.
    • Technician’s guide cross-link: “Monitoring and Observability
  • Testing
    • It’s vital that the system is tested to make sure it works properly with all the other components.
    • Have your tech team confirm that all components are tested regularly, so that if a customer places an order, the whole process will work as expected.
    • Technician’s guide cross-link: “Automated Testing and Quality Assurance
  • Security
    • Ensure that your customer data is kept safe. Customer privacy is important and they should know how their personal data will be protected.
    • Discuss with your tech team how they will be integrating security to ensure compliance.
    • Technician’s guide cross-link: “Automated Security and Compliance: Security Integrations
People
  • Training
    • Make sure that all employees that need to use the system have proper training, and that all employees know how to use the new system and troubleshoot issues. For example:
      • Do cashiers know how to enter orders?
      • Do delivery drivers know how to open a support ticket if they encounter an issue with the mobile app?
      • Do you have several employees thoroughly trained on how to add a special seasonal pizza to the menu?
    • Technician’s guide cross-link: “Knowledge Sharing and Training
  • Incident response
    • Make sure you have a plan for when things go wrong.
    • Discuss with your tech team the process for handling any issues, including contacting the vendor, troubleshooting and fixing the problem quickly.
    • Technician’s guide cross-link: “Incident Response
  • How well is the new system working?
    • It’s important to understand whether your new system is impacting your business goals, and how to track its performance.
    • When you are discussing the system with your tech team, be sure to ask them about these key metrics:
      • How is this new system impacting sales and customer satisfaction?
        • Technician’s guide cross-link: “Business Metrics
      • How quickly are you getting feedback about new features added to the menu?
        • Technician’s guide cross-link: “Business Metrics: Lead Time of Value Delivery
      • How long does it take to make new features available to customers?
        • Technician’s guide cross-link: “Business Metrics: Lead Time of Value Delivery
      • Are your customers satisfied with the online ordering process?
        • Technician’s guide cross-link: “Business Metrics: Stakeholder Satisfaction

By discussing these points with your tech team, you can ensure that your new online ordering system works well for Pizza Time and meets your business goals.

Remember, this new system is a tool to help you grow, and with the right setup and management, it will help you to serve more customers than ever before!


For the technical product owner

Not shown is my subsequent argument back and forth, in which NotebookLM later finally conceded that menu, pricing, delivery zones, and loyalty program rules should be data-entered directly into the production SaaS using the SaaS’s built-in administrative UI. That’s what a SaaS is for. Please, NotebookLM, don’t make a pizzeria’s chef use Git to update the menu.

However, we came to an agreement that this kind of medium-pace-of-change business data, which is used to configure a SaaS to better represent the way a particular business operates, definitely should be automatically backed up to some sort of versioned data store any time that this type of data gets changed within the production SaaS instance.

Ideally, the tech team helping the business make the most of their SaaS has also created an automation (e.g. CI/CD pipeline) that can ephemerally spin up a short-lived nonproduction emulation of the SaaS – prepopulated with such data.

While a nonproduction system doesn’t need every historical client’s home address in it, there’s no good reason that people testing it out should be distracted by inaccurate pizza flavors and prices.

Technical guide to supporting the new online ordering system

  • (click expander icon at left – it’s a rightward-facing triangle – if the article below is not showing up)

This article provides a technical guide for the IT team supporting Pizza Time’s new online ordering system.

It outlines the key areas you need to manage, and provides details on the metrics to track, and how they map to business goals.

Managing configurations

It’s crucial to manage and track all configurations for the online ordering system to ensure consistency and reliability.

  • Menu and Pricing
    • Use a version control system to track changes to menu items, prices, and descriptions.
    • Implement CI/CD pipelines to deploy changes to different environments (development, staging, production).
  • Delivery Zones
    • Store delivery zone definitions in version control.
    • Use CI/CD pipelines to deploy any changes to delivery areas.
  • Loyalty Program Rules
    • Use a version control system to manage changes to loyalty program rules.
    • Use CI/CD pipelines to ensure consistency and automation for loyalty program changes.

Automated Security and Compliance

  • Payment Processing
    • Ensure the online ordering system is compliant with PCI DSS standards.
    • Implement encryption for sensitive payment data at rest and in transit.
  • Security Integrations
    • Use single sign-on (SSO) for employee access to the online ordering system.
    • Implement regular security audits of the SaaS platform.
    • Utilize data loss prevention (DLP) tools to safeguard sensitive customer information.
  • Policy as Code
    • Implement security and compliance policies as code to ensure all environments of the SaaS are secure and compliant with standards.

Monitoring and Observability

  • Synthetic Monitoring
    • Set up probes to monitor the availability of the ordering page, mobile app API, and other critical endpoints, from various geographic locations.
    • Use tools such as Prometheus and Grafana to monitor system performance.
    • Set up alerts for any failures or performance degradation.
  • Logging
    • Collect and analyze logs from web servers, application servers, and databases.
    • Set up monitoring for application performance as well as for identifying any security threats or patterns.

Automated Testing and Quality Assurance

  • Integration Testing
    • Create automated tests that verify the system correctly integrates with the payment gateway and the delivery management system.
    • Test the transfer of customer data between systems.
  • End-to-End Testing
    • Use tools like Selenium to simulate customers placing orders, testing all the steps from browsing the menu, to entering payment information.
  • Performance Testing
    • Use load testing tools to simulate high traffic during peak hours.
    • Monitor response times for critical customer actions.
  • Security Testing
    • Integrate security testing tools (SAST, DAST, SCA) into the CI/CD pipeline.
    • Scan for vulnerabilities in the menu items and customer data handling processes.

Infrastructure as Code (IaC)

  • Manage the infrastructure for monitoring systems and databases using Terraform or similar IaC tools.
  • Track infrastructure changes using version control and CI/CD pipelines.

Knowledge Sharing and Training

  • Create documentation on how to configure and use the system.
  • Provide training materials for all employees.
  • Create troubleshooting guides for common issues.
  • Build a knowledge base for sharing tips and tricks.

Incident Response

  • Develop clear incident response plans for issues related to the system.
  • Include plans for contacting the vendor and rolling back changes.
  • Plan for how to isolate systems in the event of a security breach.
  • Conduct regular drills to simulate potential incidents.

Business Metrics

  • Number of DevOps initiatives mapped to business outcomes: Track the number of projects that are impacting business goals such as increases in online sales and improved customer satisfaction.
  • Quantifiable impact of DevOps on business metrics: Measure the impact of DevOps efforts on business metrics such as increases in revenue and reductions in costs associated with delivery.
  • Lead time of value delivery: Measure how quickly you can add a new topping to the menu on the SaaS platform, and make it available to your customers, and track this over time.
  • Stakeholder satisfaction: Gather feedback from the business team on ease of use, and whether it is meeting their objectives.
  • Cost of the SaaS: Track the cost of the SaaS platform compared to the business value it is delivering.
  • Percentage of compliance checks that are automated: Track the number of manual compliance checks that have been automated.
  • Percentage of teams completing security training: Track the percentage of teams that have completed security training.
  • Average time to remediate critical vulnerabilities: Track the time it takes to remediate vulnerabilities and reduce security risks.
  • Usage of shared documentation: Track the frequency with which the team is accessing and using documentation.
  • Tool integration success rate: Track the rate of successful integrations between the SaaS platform and your other tooling, and measure how tool malfunctions are impacting the team.
  • Retroactive Metrics: Utilize historical data to track the impact of the SaaS on metrics such as deployment frequency, build frequency, build success rate, build time, average response time, error rates, and test automation.

Additional Technical Metrics (if desired)

  • CI/CD Metrics: Track the duration of pipeline runs, number of active pipelines, and number of errors in the CI/CD system.
  • Version Control Metrics: Measure the number of changes (pull requests/merge requests), the duration of changes in a given state, the time to merge, and the number of contributors to a repository.

By addressing these technical areas, your team can ensure the successful operation of the online ordering system, aligning it with Pizza Time’s business objectives.

This comprehensive approach will enable you to manage the new system effectively, and contribute to the success of the business.


Technique

It was fun getting to this point. I only gave NotebookLM one source, and all it said was:

“Architecture & DevOps for enterprises”

Then I asked it to give me a sample strategic priorities that had to do with the topic. It had to go off and search the internet, because obviously I hadn’t given it a lot to go on.

Every time it answered a question in a way I liked, I added its answer back as another source.

I argued with it probably about 40 times, continuing to feed itself back into itself.

I’ve been very impressed with what else it was able to generate:

  1. Summaries that made me feel like I just re-read the DevOps Handbook in record time..
  2. Helpdesk article about what technical product owners should specify when opening a ticket asking for technicians to set up a new synthetic monitoring probe.
  3. Sample synthetic monitoring probe specifications, tailored to the imaginary pizzeria SaaS, complete with:
    • HTTP request details
    • HTTP response expectations
    • run schedules
    • intended recipients
    • considerations if the URL in question happens to be authentication-protected
  4. Sample queries against an observability platform, like Azure Monitor or Splunk, that might help a technician keep their eye on the “business value” ball, rather than staring at a wall of contextless ping response times.
    • With the samples clumped by topic!
    • So basically, for example, “sample queries of CI/CD statistics, what they teach you about the big picture of how the online ordering system is doing, and what those results might imply for the business.”

I’m still not thrilled about the planet-burning or the profitized theft of humanity’s creativity (I saw a great point raised: it’s our words it trained on, so shouldn’t it be free to get back out, like NIH-funded studies should be published for free in academic journals?).

But I’ll admit that I definitely got a new perspective on SaaS for DevOps in a hurry, 12 hours after a chance conversation about synthetic monitoring.

Career reflections

It’s interesting coming to LLM usage mid-career, rather than pre-career (e.g. as a teenager). I’ve already got strong career-specific “Google-fu.” It’s not hard for me to figure out exactly what I want to argue with the LLM about. When I use it, I’m basically doing the “hallway track” at a conference – meeting with other experts and arguing through my mental blocks about a problem I already largely understand in depth. I think it’s a bit like they say about the “Oregon Trail Generation” – it feels like there’s a weird transitional relationship to a new technology that is going to get everyone all excited about what it can do, but that it’s actually the not growing up with the technology that makes it possible to leverage it to that extent.

I can definitely say that my relationship to tech has been influenced for the better by StackOverflow. I tried programming out of paper books. It didn’t work. I almost didn’t become a programmer. And then by the time I had my second chance, Google and StackOverflow and the notion of Developer Relations teams had all matured, so I was able to learn in a way that did work for me.

Are LLM agents going to make upskilling even more accessible to junior technicians than StackOverflow made the world to me (compared to paper books)?

Maybe. But I learned so much about the way I think – and argue with the LLM – from trusted individual humans asking me for clarification, suggesting that I was looking at the problem the wrong way and pointing me at a totally different way to solve the problem and then answering my followup questions and being so kind as to give me a tiny example that wasn’t hallucinated garbage. I get the impressions that LLMs just want to help, and won’t fight you too hard on the idea that your design/architecture is all wrong, unless you know how to really push them. (I’ll have to do a followup blog post about trying to get it to set up Sanity CMS for me, as an example.) I learned how to do that by the way I was spoken to by other humans who were paying personal human-to-human attention to me.

If LLM agents are the new “graphing calculator,” then how do we senior devs teach “calculus” in light of their existence?

What are the details of the way we might need to be mentoring in junior devs that respect their work-life balance by helping them use them well? (Ahem – presuming we don’t burn up the planet so fast using them that it’s a moot question.)

--- ---