Salesforce, Python, SQL, & other ways to put your data where you need it

Need event music? 🎸

Live and recorded jazz, pop, and meditative music for your virtual conference / Zoom wedding / yoga class / private party with quality sound and a smooth technical experience

Salesforce packagelets

06 Sep 2022 🔖 salesforce vscode
💬 EN

Table of Contents

Time for another “miscellaneous notes with the SFDX and CCI command lines” post.

I always strive to create a brand new folder on my hard drive, and with it a brand new Git repository in the cloud, each time I’m asked to do something in collaboration with a team of admins who, in the end, use org-driven development.

Lately, I’ve been working on developing an ETL job (which lives in a tool outside of Salesforce) to data-load information downloaded by FTP from the CommonApp into EASY data structures inside of a Salesforce org.

I’m having to build out a few custom fields and a permission sets in a sandbox as I work. Without stepping on the toes of other admins who have their own Git repositories of work they’re doing, I want to track my changes so that they’re easy to put into production.

This is my equivalent to more or less keeping a big ol’ Outgoing Change Set open as I work, only with Git.


Set up base data in my new folder

I wrote at length about this in How I set up VSCode for a new Salesforce project, so here’s a TL;DR:

  1. Create a new folder on the hard drive of a computer that already has Git installed. Let’s say I create the folder C:\example\myproject.
  2. Into the folder, put a .gitignore file, a sfdx-project.json file, a .sfdx\sfdx-config.json file, a cumulusci.yml file, and an orgs folder with beta.json, dev.json, feature.json, and release.json files within it.
  3. Git-track this folder (git init), git add --all and git commit my work from step 2 with a simple comment like “First commit”, create a corresponding folder in the cloud for it, tell my computer where in the cloud to find the corresponding cloud copy, and send a copy of my latest “commit” to the cloud (git push). All of this can be done to GitHub without so much technobabble using software called GitHub Desktop.
  4. Run cci flow run dev_org --org beta and cci org browser --org beta to make sure that I didn’t make any mistakes in my files in step 2 and that CumulusCI knows how to spin up a scratch org off of everything in this folder on my computer.
  5. I think maybe it gets mad at me if there’s nothing of note to actually put into the scratch org? I don’t remember. If so, I know that sometimes I’ve put these 2 files into a force-app\main\default\classes folder after committing (I’m going to get rid of them later; no point in committing them to Git tracking) before spinning up a scratch org:
    • Hello_TEST.cls: ```java @isTest public class Hello_TEST {

      static testMethod void runTest() { System.assertEquals(0,0); }

    }

             * `Hello_TEST.cls-meta.xml`:
     ```xml
     <?xml version="1.0" encoding="UTF-8"?>
     <ApexClass xmlns="http://soap.sforce.com/2006/04/metadata">
         <apiVersion>55.0</apiVersion>
         <status>Active</status>
     </ApexClass>
    
  6. And if those 2 files were also in my project folder before I ran cci flow run dev_org --org beta, then once it’s spun up, I might as well run a quick cci task run run_tests --org beta --test_name_match Hello_TEST to validate that Hello_TEST made it safely into the org.
  7. If I had to include them to get a scratch org to spin up at all, then I can delete the force-app\main\default\classes\Hello_TEST.cls and force-app\main\default\classes\Hello_TEST.cls-meta.xml files from my project folder as soon as I start working for real and come up with anything actually useful to put into the force-app\main\default\ folder.

Tip: Sometimes I create more than 1 such project folder if a lot of the actual work I’m trying to do to “solve my problem” involves Salesforce work itself (like new Flows or new Apex classes). For example:

  1. I’ll have one folder for “housekeeping that would lay a stronger foundation for solving any business problem.”
  2. I’ll have another folder for “stuff that actually solves the business problem at hand once the housekeeping is done.”

Then I’ll use the VSCode extension Peacock to color-code the two folders so I don’t mix them up with each other.

In other words, I break up my business problem into two separate problems that, “coincidentally,” I am working on in parallel, and give each mini-problem its own Git-tracked codebase.


Make sure SFDX is logged into relevant orgs

Let’s say the sandbox I need to add a few custom fields & permission set configurations to is at https://customdomain--mysandbox.sandbox.my.salesforce.com/:

I’ll follow the steps under “SFDX global authentication and aliasing” from “How I set up VSCode for a new Salesforce project,” giving it an SFDX CLI tool alias of, oh, maybe … customdomain-mysandbox.


Download work from persistent orgs into my project folder

Let’s say I add a custom field in that sandbox, and I know that I’m going to have to do that again in production or another sandbox – making sure that my computer’s command prompt is in a context of operating from C:\example\myproject, I can run this command:

sf retrieve metadata --metadata CustomField:Custom_Object__c.Custom_Field__c --target-org customdomain-mysandbox

And … boom! I’ll end up with a new file C:\example\myproject\force-app\main\default\objects\Custom_Object__c\fields\Custom_Field__c.field-meta.xml.

Oh, but also, I had to make sure that this field was readable as part of a new permission set I’d created … so let’s grab that too:

sf retrieve metadata --metadata PermissionSet:Custom_Permission_Set_Name --target-org customdomain-mysandbox

And … boom! I’ll end up with a new file C:\example\myproject\force-app\main\default\permissionsets\Custom_Permission_Set_Name.permissionset.

If I keep making changes to this custom field or to the permission set, I can just run these commands over and over again, and the files on my hard drive will get their contents updated with the latest reality of the work I’ve been doing point-and-click inside my Salesforce sandbox.

Now I can “stage” these 2 files into being tracked by Git (git add --all), git commit them into the formal Git history with a message like “We need Custom Field to make data load better,” and git push a copy of these 2 new files into the copy of my Git repository that lives in the cloud. (Again, this is much easier to do with GitHub Desktop if you’re using GitHub for your cloud copy.)


Upload work from my project folder into persistent orgs

Let’s say someone refreshes the sandbox I’ve been working in: the sf deploy metadata command has a lot of well-documented variations.

For example, I could deploy all of my work at once back into my sandbox:

sf deploy metadata --target-org customdomain-mysandbox

Or maybe, for some reason, I only need to deploy a single custom field, because it’s not so much that someone broke all my work with a sandbox refresh. Maybe I broke my own work, only on that one custom field, by clicking the wrong button inside the Sandbox point-and-click configuration, and now I want to get things back the way they should be:

sf deploy metadata --metadata CustomField:Custom_Object__c.Custom_Field__c --target-org customdomain-mysandbox

(I’ve also done an sf retrieve metadata with a --target-org option pointing to production and an sf deploy metadata against just that one file with a --target-org pointing to my sandbox when I discovered that a custom field was out-of-date in my sandbox but various admins hadn’t yet had a chance to migrate the change into all sandboxes. Or when I found out that a colleague had created a useful custom object in a different sandbox than the one I was currently working on, but that it might help my project too. In both cases, this is where having separate “housekeeping” & “my actual project” folders can come in handy, as anything someone else is also using is probably a “housekeeping” kind of thing I should eventually roll into a bigger company-wide codebase.)

Read the sf deploy metadata documentation for a lot of useful variations on this theme.


Upload work from my project folder into a scratch org

Honestly, I don’t tend to push a lot of code into a running scratch org from my computer. Such commands exist, but I don’t tend to do them.

Instead, I typically let my scratch orgs die (or run cci org scratch_delete) and re-create them daily with cci flow run dev_org. It keeps me honest and helps me avoid accidentally building a project folder that can’t stand on its own sufficiently.

If I did, though (maybe I just grabbed a custom field out of a colleague’s sandbox into my project folder with sf retrieve metadata and don’t want to wait half an hour to rebuild a slow-building scratch org with this extra custom field), I suppose I’d use something like cci task run dx_push --org beta.


Download work from a scratch org into my project folder

On the other hand, I’m a little more likely to need to download copies of my point-and-click configuration work out of a scratch org into my project.

For that, I tend to use cci task run list_changes --org beta and cci task run retrieve_changes --org beta.


Hack 1 scratch org into serving 2 projects

So, there’s this Git repository shared around my office that typically has all of the “baseline” stuff used by my company in it.

Problem is, no one’s quite figured out how to package it up so that it works nicely as one of the project.dependencies in cumulusci.yml.

The only way to spin up a scratch org with this repository’s elements installed into it is to open up a copy of that repository on my computer (let’s call it C:\example\baseline) and, from within a command-line prompt out of C:\example\baseline, run a command like cci flow run dev_org --org beta.

When I develop mini-projects in a scratch org, I don’t want to develop my custom mini-project against this larger “baseline” codebase – I want to develop them in their own Git-tracked project folder.

So here’s what I do: once I run cci flow run dev_org --org beta in the baseline folder, I note down the project.name property’s value in that “baseline” repository’s cumulusci.yml. Let’s say it’s called Corporate-Baseline.

Then I go over to a command-line context within my actual mini-project, C:\example\myproject, and I run cci org import Corporate-Baseline__beta scratchwithbaseline.

If I run cci org list within C:\example\myproject, I can see that there’s a scratchwithbaseline org available to me.

I can now run commands against this scratch org just by choosing “scratchwithbaseline” instead of, say, “beta,” as my “--org” parameter:

  1. Open the scratch org in a web browser with cci org browser --org scratchwithbaseline
  2. Upload data from C:\example\myproject into the scratch org with cci task run dx_push --org scratchwithbaseline
  3. Download data from the scratch org into C:\example\myproject with cci task run list_changes --org scratchwithbaseline and cci task run retrieve_changes --org scratchwithbaseline.

That said, before I did any of that, if I had some project.dependencies in C:\example\myproject\cumulusci.yml, I’d want to run cci flow run dev_org --org scratchwithbaseline from within a C:\example\myproject command line.

CumulusCI is smart enough to know not to delete-and-recreate my scratch org, but just to finish off installing dependencies into it based on C:\example\myproject\cumulusci.yml (and to take all of the actual things that are part of my project’s source code and upload them into the scratch org for the first time).

When dependencies don’t work

If you get Resolving dependencies... & Exception in task dependencies.update_dependencies & Error: 404 Not Found when you run cci flow run dev_org --org scratchwithbaseline, try this:

Get rid of the reference to your external dependency in the project.dependencies object of cumulusci.yml.

Add a section to cumulusci.yml kind of like this:

sources:
    my_extra_dependency:
        github: https://github.com/UsefulPublisher/UsefulLibrary
        branch: main

Then run cci task run my_extra_dependency:deploy --org scratchwithbaseline.

Do the cci task run for as many “extra dependencies” as you have.

Finally, you should be able to get your “actual project work” into the scratch org with cci flow run dev_org --org scratchwithbaseline or cci task run dx_push --org scratchwithbaseline.

--- ---