Discussions

I'm integrating the FishBase API into a data synchronization service and have run into what appears to be a pagination issue.

My goal is to import the complete species dataset in batches of 500 records. I'm using requests like:

GET /species?limit=500&offset=0
GET /species?limit=500&offset=500
GET /species?limit=500&offset=1000

The import completes without any HTTP errors, but after comparing the results I noticed that some records appear in multiple pages while others are never returned at all. The total number of unique SpecCode values ends up being lower than expected, even though every request returns 500 records.

To rule out an issue in my code, I logged every SpecCode received and compared consecutive batches. I consistently see overlapping IDs between pages, and if I rerun the same sequence a few minutes later, the duplicates are not always the same records.

I also tried reducing the page size to 100 and adding delays between requests, but the behavior didn't change FNF.

Is there a recommended way to page through the entire dataset? Should the requests include an explicit sort order, or is the underlying data updated frequently enough that offset pagination isn't guaranteed to be stable? Any guidance would be appreciated before I redesign the synchronization process.