Rate-limiting API requests in Python with a decorator
28 Jul 2020
The other day, I needed to make sure that a Python script I’m writing doesn’t call a REST API any faster than it’s allowed to.
I wanted “DRY,” readable code, so writing “if-else” code around every call to a function from the
requests module was out of the question.
Thanks to side-project work in TinaCMS, I had some idea that the phrase I should Google when seeking out a way to “wrap” someone else’s code in additional functionality was “higher-order.”
That led me to these two tutorials:
I put the following code at the top of my script:
limitLeft = 500 # Although in theory some other code could be hitting the API, start with a presumption that we have all 500. def __request_rate_limited(request_function): def limit_rate(*args, **kwargs): global limitLeft #if limitLeft < 500: # DEBUG LINE ONLY # Because we get our "actual rate limit" from API calls, it will never be 500 -- at most 499 -- in the wild. #print('API calls remaining before making call: ' + str(limitLeft)) # DEBUG LINE ONLY if limitLeft < 10: print('API calls remaining approaching zero (below ten); sleeping for 2 seconds to refresh.') sleep(2) response = request_function(*args, **kwargs) limitLeft = int(response.headers['X-Limit-Left']) #print('API calls remaining after making call: ' + str(limitLeft)) # DEBUG LINE ONLY return response return limit_rate # Attach __request_rate_limited() to "requests.get" and "requests.post" requests.get = __request_rate_limited(requests.get) requests.post = __request_rate_limited(requests.post)
After adding this chunk of code to the top of my script, all uses of
requests.post() worked within the context of my additional code. Pretty neat.
The trick was remembering to attach the
__request_rate_limited() wrapper function to both
requests.post() before any code using them runs. At first, I only wrapped
requests.get(), then couldn’t figure out why my code wasn’t running for certain parts of my script. Turns out those parts were making
This is probably an inelegant implementation of function wrapping / decorators in Python, but it definitely got the job done for a small command-line script.<! -- I could get really fun here by including a 2nd link to prev/next if it's not in your language but it has an alt that is. --> <! -- Also, I need to CSS-style these to float left and right, like paginators. -->