Off-by-one errors in Python API pagination

16 Mar 2021 🔖 python tips
💬 EN

If you have json.loads(response.text) in a Python script using requests to perform a loop over API calls structured something like response = requests.get('https://example.com') inside the loop, and you sporadically see an error stack ending in raw_decode mentioning ValueError from None like this, you might have an off-by-one error in your loop condition:

`File ".../json/__init__.py", line ..., in loads
  return _default_decoder.decode(s)
File ".../json/decoder.py", line ..., in decode
  obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File ".../json/decoder.py", line ..., in raw_decode
  raise ValueError(errmsg("Expecting value", s, err.value)) from None

Check to see if you have an off-by-one error that’s making you enter the loop one more time than is actually necessary to fetch all the data.

1. It’s possible that you made a typo and that you just need to fix the condition for continuing your loop.
2. It’s also possible the at the API is lying to you and saying there’s a “next” page when there isn’t really one. In that case, consider adding an extra guard to the loop entry condition that verifies for example, and response is not None.

You’ll often see this problem sporadically – e.g. if the API’s pagination is in multiples of 100, only when there happen to be exactly 800 records in the dataset (or maybe 799 or 801), with your code otherwise working.