Rate-limiting API requests in Python with a decorator

28 Jul 2020 🔖 python tips api integration
💬 EN

The other day, I needed to make sure that a Python script I’m writing doesn’t call a REST API any faster than it’s allowed to.

I wanted “DRY,” readable code, so writing “if-else” code around every call to a function from the requests module was out of the question.

Thanks to side-project work in TinaCMS, I had some idea that the phrase I should Google when seeking out a way to “wrap” someone else’s code in additional functionality was “higher-order.”

That led me to these two tutorials:

• Hacker Earth on higher-order functions and decorators in Python
• Real Python on decorators

I put the following code at the top of my script:

limitLeft = 500 # Although in theory some other code could be hitting the API, start with a presumption that we have all 500.

def __request_rate_limited(request_function):
    def limit_rate(*args, **kwargs):
        global limitLeft
        #if limitLeft < 500: # DEBUG LINE ONLY
            # Because we get our "actual rate limit" from API calls, it will never be 500 -- at most 499 -- in the wild.
            #print('API calls remaining before making call: ' + str(limitLeft)) # DEBUG LINE ONLY
        if limitLeft < 10:
            print('API calls remaining approaching zero (below ten); sleeping for 2 seconds to refresh.')
            sleep(2)
        response = request_function(*args, **kwargs)
        limitLeft = int(response.headers['X-Limit-Left'])
        #print('API calls remaining after making call: ' + str(limitLeft)) # DEBUG LINE ONLY
        return response
    return limit_rate

# Attach __request_rate_limited() to "requests.get" and "requests.post"    
requests.get = __request_rate_limited(requests.get)
requests.post = __request_rate_limited(requests.post)

After adding this chunk of code to the top of my script, all uses of requests.get() and requests.post() worked within the context of my additional code. Pretty neat.

The trick was remembering to attach the __request_rate_limited() wrapper function to both requests.get() and requests.post() before any code using them runs. At first, I only wrapped requests.get(), then couldn’t figure out why my code wasn’t running for certain parts of my script. Turns out those parts were making post requests.

This is probably an inelegant implementation of function wrapping / decorators in Python, but it definitely got the job done for a small command-line script.