[librecat-dev] Handling server errors in long running oai-pmh harvests
"Dan Michael O. Heggø"
d.m.heggo at ub.uio.no
Mon Dec 5 14:49:59 CET 2016
Do you have any tips on handling intermittent server errors occuring in
long running harvests?
For a harvest of mine I got around 800.000 records before hitting an error:
all at all@oai_komplett at marc21@3535592670002204 : Server closed connection
without sending any data back
Eliminating random server errors doesn't seem realistic, so I'm
wondering if Catmandu bails out immediately or does multiple retries,
and if this is something that can be configured? If not, is the best way
to handle it to write a small Perl script that catches certain exceptions?
Also, is it possible to pass in the continuation token to the OAI
importer as a flag or something to resume the harvest from where it stopped?
Thanks alot for your feedback!
More information about the librecat-dev