Salesforce, Python, SQL, & other ways to put your data where you need it

Need event music? 🎸

Live and recorded jazz, pop, and meditative music for your virtual conference / Zoom wedding / yoga class / private party with quality sound and a smooth technical experience

Intro to using APIs over HTTP(s)

21 May 2019 🔖 tutorials api salesforce python integration architecture
💬 EN

Table of Contents

Previously, I explained than an “application programming interface,” or API, is a set of protocols for communicating with a computer as if you were knocking on the door of a fortified castle.

This series will do a beginner-level deep dive into how you go about knocking on these castle doors.

This skill often enables you to communicate with your favorite cloud services in new ways or to automate the ways in which you already interact with them.

Scope: other people’s HTTPs APIs

This is not a guide to “carving a door into your own fortress” and providing an “API” to other people that would allow them to ask your computer for favors.

The internet contains many fabulous tutorials about setting up a “web server” and turning it into an “API endpoint,” but this isn’t one of them.

This is merely a guide to using the “APIs” that other people have provided.

Furthermore, this only covers APIs that communicate over the internet using the “HyperText Transfer Protocol” (HTTP(s)), which is the most common everyday use of the word “API” today.

Preparation

Read about HTTP(s)

I can’t strongly enough recommend reading some theoretical explanations of the HyperText Transfer Protocol (HTTP).

A protocol is simple: it’s a set of rules for interacting with someone else.

In my introduction to APIs, I said that if we were talking about a real-world protocol for communicating with people inside a fortified castle, the communications protocol might look like this:

“If you have something for the fortress, bring it in a series of pink 12x12x12 boxes. Paint the first one with green stripes and the last one with purple polka dots. Put your name and return address on all of them.

“If we like them, we’ll figure out what to do with the stuff inside the boxes from there.

“After we’re done, we’ll post a receipt on the door telling you whether we handled your boxes neatly or burned them.”

The protocol for HTTP(s) is, obviously, a bit more technical. Something more like this:

“If you want to talk to the fortress, send us a Transmission Control Protocol (TCP)-formatted series of 0’s and 1’s representing appropriate words like ‘GET’ or ‘POST’ and a return address for your castle.

“Send your message to our fortress door #80.

“We’ll send a TCP-formatted series of 0’s and 1’s representing appropriate words like ‘STATUS’ to your return address, letting you know what we thought of your attempt to communicate with us.”

Obviously I’m still hiding a lot of detail.

That’s largely because you can trust your “messenger” (software or a programming language) to take care of the “0’s and 1’s” and the “TCP” formatting.

You can even trust your software or your programming language to take care of ensuring that your messages are properly formatted according to the HTTP(s) protocol … if you look up its documentation on what buttons you’re supposed to be clicking or what commands you’re supposed to be using.

That “if” is why it’s useful to study, at a conceptual level, what a good HTTP(s) request looks like.

Analogy: It’s relatively easy, as an adult, to decide to prepare for a trip to Mexico by consulting a bilingual dictionary and memorizing 50 Spanish words for concepts like “toilet,” “flight,” “breakfast,” etc. Yet you probably wouldn’t bother to give your young toddler the same task, even if they’re coming with you.

  • Q: Why?
  • A: Because as an adult, you’ve already mastered concepts like languages and toilets and air travel and meals.

Similarly, you’re going to find it a little easier to read the instructions for your software or programming language of choice once you have a basic understanding of concepts like these:

  • The HTTP(s) protocol consists of you sending “requests” and receiving “responses”
  • Your requests should be categorized as “GET,” “POST,” “PUT,” etc.
  • Your requests may have a “header” and a “body”
  • What kinds of information about your request you might be expected to put in its header or body
  • The responses you get back should have a “status” and may also have a “header” and a “body

Furthermore, every HTTP(s)-based “API” you meet will have arbitrary rules along the lines of “pink 12x12x12 boxes, the first one painted with green stripes and the last one painted with purple polka dots.”

It’s nice to have “understanding basic HTTP(s) request and response rules” out of the way so that you can focus your attention on the API’s arbitrary rules.

Note: in reality, the “pink 12x12x12 boxes” rules will look more like this:

“Whenever we send you an HTTP(s) response to your request for data, please inspect its body for the word ‘more.’

“If you see it, we didn’t send you everything you asked for.

“Please make another HTTP(s) request, putting a note into the body that indicates where we left off.

“Repeat as necessary until you stop seeing the word ‘more.’”

The programming The trick you’ll be learning in your programming language of choice is how exactly to make i

The “representing appropriate words like ‘GET’ or ‘POST’” bit is what we’ll practice in this series … but reading up on the subject ahead of time would definitely help you follow along.

Codecademy’s article is a nice introduction to HTTP(s) requests.

Why “(s)?”

You might note that I keep calling the HyperText Transfer Protocol “HTTP(s).”

What’s up with the “s” in parentheses?

Well, as you can see, the HTTP part is an abbreviation for the name of the protocol.

However, I add the “(s)” to remind you that, whenever possible, if an API “endpoint” has an address that starts with “http://,” you should try changing it to “https://”.

If it doesn’t work with the extra “s” on the end, you shouldn’t send it any “login credentials” or “authentication keys” and also shouldn’t send it any confidential data, because everyone on the entire internet can read and even modify everything you send to, or receive from, that API endpoint (it’s like communicating by postcard).

Read about APIs

Initially, I wrote my explanation of APIs to differentiate concepts like “hosting an API endpoint” vs. “using someone else’s API endpoint” to Salesforce database administrators responsible for planning data integrations.

However, I was careful to write it so that it didn’t rely upon familiarity with Salesforce, so I recommend reading it before you launch into this series.

It’s not very long, and I’m going to keep referring to API endpoints as “castles” and “fortresses” like I do in that post, so it’s a good scene-setter.

This series is a “how-to” tutorial about the 3rd approach described in that article: knocking on the “castle doors” of other people.

Read about XML & JSON

XML and JSON are, essentially, punctuation standards for formatting plain text that help add structure and meaning to the data within that text.

XML involves a lot of < and > symbols; JSON involves a lot of [, ], {, and } symbols.

If you’re the kind of person who can read something that repeats itself and “follow along” without too much trouble, you might not need to know more than that.

But if you’d like more context, I wrote a 7-part “zero to comfortable” beginner’s tutorial called “Intro to XML and JSON.”

Takeaways & What’s Next

That’s it for today – just some “recommended reading” before we start with hands-on exercises!

The main things to remember about what you’ll learn in this series are:

  1. Talking to another system’s “API” is kind of like knocking on a fortified castle door and talking through the guard’s little eye slot.
  2. Because we’re talking about computer-to-computer communication, everything is done within computer-friendly standards like HTTP(s), XML, & JSON.
    • It’s handy to have a “beginner-intermediate” grasp of them so that you can focus on the additional rules presented by the particular API you’re trying to communicate with.
  3. Security matters. Don’t be careless with passwords and sensitive data. Someone’s always waiting for you to make one false move.
  4. You don’t have to learn to code to take advantage of HTTP(s) APIs – there’s downloadable software such as Postman that lets you point-and-click your way through writing great HTTP(s) requests.
    • This is worth doing for its own sake, even if you don’t have time to learn to code!
    • Many enterprise-level “Extract, Transform, Load” (ETL) tools have a rather point-and-click user interface along the lines of Postman, so you really can learn a useful skill without learning to code.
  5. However, if you are learning to code, or know how to code, you’ll want to learn to leverage HTTP(s) APIs in your programming language of choice.
    • To really leverage APIs, you might need to level up your programming skills with respect to “data structures” and “file I/O.”
    • (That is, if the API’s rule is “use pink 12x12x12 boxes,” you need to know how to cut a box to size and how to color it pink.)

APIs

Interested in learning to code against cloud-hosted services?

Learning to write code against “APIs” to solve Salesforce problems requires mastering some fundamental concepts first.

Here’s a roadmap to teach yourself by:

Step 1: Conceptually understand APIs, XML, & JSON

Start with my “Salesforce REST APIs: A High-Level Primer” post and my series.

I think it’s incredibly important to have a firm grasp of these concepts before you even think about coding.

Step 2: Conceptually understand HTTP

Programming is all about teaching a computer to do stuff on your behalf.

How can you teach a computer what to do if you don’t understand what you want it to do?

Learn what an HTTP(s) request is at a conceptual level first.

Step 3: Get hands-on with HTTP

Cement your understanding of what making an HTTP(s) request does by making a few of your own.

But don’t start doing it with “code” yet.

There’s a wonderful point-and-click tool you can download and run on your computer called Postman that makes it easy to communicate with the “API endpoints” of computers in the cloud.

  • Q: Not sure what cloud-based computer to talk to?
  • A: Try one of these! Choose one that doesn’t require “authentication” (login), because learning to manage passwords properly is a huge topic you’ll need your full attention to master some other time.

Play with using Postman to write and execute “GET” and “POST” HTTP(s) requests against a variety of “APIs.”

  1. Look at the “body” of the data that comes back (the “HTTP(s) response”). Does it contain what you expected it might?
  2. Look at the “status” code of the response. Is it 200? 400? Something else?
  3. Click on the “Headers” tab of the “response.” Is there any interesting information?
    • Running a simple “GET” request against the API endpoint located at https://yesno.wtf/api will send back a response whose headers include a note called “Content-Type” indicating that the text contents of the response’s “body” are structured according to the “JSON” standard and using letters available in the “UTF-8” alphabet.)
  4. Change something and send another “request” to the same “API.” Does the response look like you expected?
  5. Get a feel for the various options that certain APIs expect you to send with your “requests.” See the next 3 steps for examples.
  6. Some API endpoints want you to change the “URI” (address) of the “endpoint” itself by adding “parameters” to the end of it.
    • A “GET” request against http://jsonplaceholder.typicode.com/posts will give you information about lots of fake blog posts.
    • A “GET” request against http://jsonplaceholder.typicode.com/posts/1 will give you the same information about just the first one.
    • A “GET” request against http://www.recipepuppy.com/api/?i=onions,garlic&q=omelet&p=3 will give you a recipe with its text structured according to the “JSON” standard.
    • A “GET” request against http://www.recipepuppy.com/api/?i=onions,garlic&q=omelet&p=3&format=xml will give you the same recipe, but with its text structured according to the “XML” standard.
  7. Some API endpoints refuse to send you a useful response, or change what response they send you, unless you put special data into the “body” of a request.
    • A “POST” request against https://reqres.in/api/register with nothing in the “body” will return an HTTP(s) response with a “Status” code of 400 (“bad request”) and a body whose contents say {"error":"Missing email or username"}.
    • A “POST” request against https://reqres.in/api/register where you set the request’s “body” type to the “raw” radio button, change the “Text” drop-down just to the right of that radio button to “JSON (application/json)”, and fill in the contents of the request “body” with {"email": "[email protected]", "password": "pistol"}, will return an HTTP(s) response with a “Status” code of 200 (“OK”) and a body whose contents say {"id": 4, "token": "QpwL5tke4Pnpja7X4"}, which means it worked.
    • Note: only certain username and password combinations work against this particular API. Once you get it working, try changing the e-mail address and see if you can get it to go back to failing with a response body of {"error": "Note: Only defined users succeed registration"}. So now you can see that sometimes it’s not just a question of figuring out how to send data in the “body” of an HTTP request, but also about figuring out what data the API wants to see.
  8. Some API endpoints refuse to send you a useful response, or change what response they send you, unless you put special data into the “header” of your request.
    • In our previous example, changing the “Text” drop-down to “JSON (application/json)” while setting up the “body” of our request actually made Postman add a header to our request.
    • Click the “Headers” tab of the request you got working against https://reqres.in/api/register. Un-check the checkbox next to the row in the table that indicates a “Key” of “Content-Type” and a “Value” of “application-json” and send your HTTP request again. It should fail with a response body of {"error":"Missing email or username"}. Now check the box so that the header parameter that Postman added for us does go through with the request and re-send the request. It should succeed.
    • The two most common reasons for adding data to the “headers” of an HTTP(s) request are to specify details about the structure of anything you put into the request’s “body” and to accompany your request like authentication information like passwords.
    • Don’t play with passwords just yet. Please. :)

If you’re ever confused while you try to figure out how some API wants to be talked to, come back to Postman and play around until you make it work. Then move on to writing the same request with code. Don’t worry that it’s too “easy” – it’s what all developers do when they get stuck on a difficult API, no matter how much experience they have!

“More, please”

Note that some APIs will refuse to tell you everything they know at once.

For example, if you do a “GET” request against https://api.coinlore.com/api/tickers/, it will only send you price information about 100 different kinds of cryptocurrency.

Its response body looks kind of like this:

{
    "data": [
		{...a currency's price info...},
		...,
        {...a currency's price info...}
    ],
    "info": {
        "coins_num": 2173,
        "time": 1558125575
    }
}

So, we can see at the end of the response body that it actually knows about 2,173 currencies, even though there were only 100 {...a currency's price info...} chunks in the “data” list.

Even if I think I can “trick” it by making a request to https://api.coinlore.com/api/tickers/?start=1&limit=3000, I get the same response. Darnit.

What this API expects me to do is call it 100 chunks at a time, keeping track of where I left off, and taking my own responsibility to stop when I get to the end of the list.

It expects me to do:

  1. https://api.coinlore.com/api/tickers/?start=1&limit=100
  2. https://api.coinlore.com/api/tickers/?start=101&limit=100
  3. https://api.coinlore.com/api/tickers/?start=201&limit=100
  4. https://api.coinlore.com/api/tickers/?start=2101&limit=100

And it expects me to be responsible and not call it with “2201” since it already told me, in the very first response, that it only knew about 2,173 coins.

Obviously, this kind of repetition is going to be a lot easier to write into code (due to the ability to “loop”) than it is to do by hand in Postman.

Still, Postman is perfectly adequate for the kind of tests I just did like whether I could “trick” the API into giving me all of its data at once.

The takeaway is that when programming against APIs, you’ll need to read the documentation, and/or carefully inspect HTTP(s) responses, to determine if a single HTTP(s) request will actually give you everything you want to know.

Step 4: Repeat step 3, but with code

Now it’s time to learn how to use your programming language of choice to send HTTP(s) requests to servers and read the HTTP(s) responses they send back.

Salesforce Apex

Let’s say you want to teach Salesforce Apex code hosted in your org to make an “API call” to someone else’s service (e.g. one that knows things about U.S. postal ZIP codes).

First, practice writing and running Apex code that does the same tasks you previously explored using Postman.

Of course, you’ll want to be comfortable using System.debug(...) to see the textual output of Apex code you run so that you can see what comes back in the HTTP(s) responses from the APIs you talk to!

See these instructions on Trailhead to get started.

Python

Let’s say you want to download all the Prospects in your Pardot instance to your computer.

First, practice writing and running Python code that does the same tasks you previously explored using Postman.

Of course, you’ll want to be comfortable using print(...) to see the textual output of Python code you run so that you can see what comes back in the HTTP(s) responses from the APIs you talk to!

See these links for some documentation on dealing with HTTP(s) requests and responses in Python.

Step 5: Level up your programming skills in general

You won’t get far doing anything useful with a recipe from RecipePuppy or a fake blog post from JSONPlaceholder without knowing some “programming 101” fundamentals.

In your programming language of choice…

  1. Learn about “data structures” like:
    • Lists / Arrays
    • Dictionaries / Maps / Objects
    • Strings (you should understand the difference between a plain-text “string” in your code’s “memory” and a file saved on your hard drive containing a bunch of plain text).
  2. Learn how to select a given item from a list (by position number) or dictionary (by “key” name).
  3. Learn how to “loop” over the contents of a list or dictionary one item at a time.
    • In Python, you may quickly feel pressure to learn about “comprehensions,” a shorthand way to code loops. It’s what all the “cool kids” like to do, so you’ll see a lot of it in answers on StackOverflow.
  4. Learn how to translate back and forth between:
    • The plain-text representations of list/dictionary-like data (such as JSON or XML) that are in HTTP(s) requests/responses
    • The actual lists and dictionaries (and, in the case of Python when dealing with XML, “etree”s) that your programming language can actually process and “loop” over gracefully.
    • This is sometimes called “serialization(data structures to text) and “deserialization(text to data structures) in documentation you find online about your programming language.
  5. Practice doing these translations. Here’s an exercise:
    • Take a plain-text “string” that says {'animal1':'dog','animal2':'cat'} and transform it into a “string” that says {'pet-1':'DOG','pet-2':'CAT'}.
    • Do this by converting it into a dictionary/map/object with “animal1” and “animal2” keys, transforming that dictionary’s contents (either editing it or using it to build a new one), and converting the resulting dictionary back into text.
    • No cheating: don’t just do “find and replace” against the text itself.
  6. Learn how to read and write data to and from files on your hard drive (presuming you’re writing code that runs on your computer, like Python).
    • (Pro tip: never tell your code to overwrite the file you just read data from. Write to a different filename. That way, if you make a mistake, you can start over.)
    • Writing data to/from your hard drive is particularly important when downloading large data sets from other people’s cloud services. It’s rude and slow to re-download the same data every time you test whether your program works. Proper etiquette is to write one program to download the data to your hard drive and run it once. Then play around writing a different program that processes the data from the file on your hard drive. Don’t re-run the first program until the second program is working well enough that you actually need fresh data.
  7. Put it all together
    • Write code that makes an HTTP(s) “GET” request to https://yesno.wtf/api and save the response’s “body” to a file on your hard drive.
    • Write another program that reads the file from your hard drive and turns its contents into a dictionary/map/object with 3 “keys” (answer, forced, and image).
    • All-caps the “value” corresponding to the “answer” key.
    • Turn your dictionary/map/object back into “JSON”-formatted plain text and save it to a different file on your hard drive.
    • Your 2nd file should look just like the 1st file, only the yes/no should be YES/NO.
    • (If you’re doing this exercise in Salesforce Apex, skip the “file” parts of the exercise. You can see the output of your final plain-text String with System.debug(...).)

Step 6: SECURITY SECURITY SECURITY

Be sure you understand how careful you need to be any time you type a password into code.

  1. Promise me you won’t start trying to program against “APIs” that require passwords irresponsibly.
  2. Postman lets you save configuration files for later reuse. Promise me you won’t save passwords, “api keys,” “tokens,” etc. in Postman configuration files.
  3. Promise me you won’t save passwords, “api keys,” “tokens,” etc. in your code when you’re not actively editing it.
  4. Promise me you will never ever ever simply type a password, “api key,” or “token” into code and leave it somewhere on a server so that it can “run automatically.”

Step 7: Find inspiration

Browse my blog posts tagged “API” for inspiration about projects I’ve done like clean up data in Pardot.

Happy progrmaming!

--- ---