[librecat-dev] Handling server errors in long running oai-pmh harvests

[Vitali Peil]

Hi Dan Michael,

you are hitting a wound point here: There exists an issue which adresses 
the problems you are running into:

https://github.com/LibreCat/Catmandu-OAI/issues/7

We welcome contributions! We can discuss strategies to keep this 
importer stable while running for hours.

Cheers,

Vitali


Am 05.12.2016 um 14:49 schrieb "Dan Michael O. Heggø":
> Hi,
>
> Do you have any tips on handling intermittent server errors occuring 
> in long running harvests?
>
> For a harvest of mine I got around 800.000 records before hitting an 
> error:
>
> ERROR: 
> http://bibsys-k.alma.exlibrisgroup.com/view/oai/47BIBSYS_UBO/request : 
> all at all@oai_komplett at marc21@3535592670002204 : Server closed 
> connection without sending any data back
>
> Eliminating random server errors doesn't seem realistic, so I'm 
> wondering if Catmandu bails out immediately or does multiple retries, 
> and if this is something that can be configured? If not, is the best 
> way to handle it to write a small Perl script that catches certain 
> exceptions?
>
> Also, is it possible to pass in the continuation token to the OAI 
> importer as a flag or something to resume the harvest from where it 
> stopped?
>
> Thanks alot for your feedback!
>
> Dan Michael
> _______________________________________________
> librecat-dev mailing list
> - send list mails to librecat-dev at lists.uni-bielefeld.de
> - to unsubscribe or change options, visit 
> https://lists.uni-bielefeld.de/mailman2/cgi/unibi/listinfo/librecat-dev
> - project website: http://librecat.org/

-- 
Vitali Peil
Office U3-200/E1-144, Tel. +49521-106-4010/6125
Bielefeld University Library