16 Mar 2021
If you have
json.loads(response.text) in a Python script using
requests to perform a loop over API calls structured something like
response = requests.get('https://example.com') inside the loop, and you sporadically see an error stack ending in
None like this, you might have an off-by-one error in your loop condition:
`File ".../json/__init__.py", line ..., in loads return _default_decoder.decode(s) File ".../json/decoder.py", line ..., in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File ".../json/decoder.py", line ..., in raw_decode raise ValueError(errmsg("Expecting value", s, err.value)) from None
Check to see if you have an off-by-one error that’s making you enter the loop one more time than is actually necessary to fetch all the data.
- It’s possible that you made a typo and that you just need to fix the condition for continuing your loop.
- It’s also possible the at the API is lying to you and saying there’s a “next” page when there isn’t really one. In that case, consider adding an extra guard to the loop entry condition that verifies for example,
and response is not None.
You’ll often see this problem sporadically – e.g. if the API’s pagination is in multiples of 100, only when there happen to be exactly 800 records in the dataset (or maybe 799 or 801), with your code otherwise working.