[librecat-dev] Parallel Processing with Catmandu

[Patrick Hochstenbach]

The example that Johann gave in at the end of the blog post would do the trick for any number of CPUs:

$ ls *.mrc | parallel catmandu convert MARC to MARC --fix myfixes.fix  '<' {} '>’  {.}.done '2>' {.}.log

This will transform a bunch of .mrc files using a Catmandu myfixes.fix into an a bunch of  files which .done at the end of the file name. 
If your machine has one CPU it will process everything is sequece. If you have two CPUs it will run at max 2 processes in parallel. Etc.

Patrick

> On 30 Apr 2016, at 10:54, aina at openmailbox.org wrote:
> 
> Dear Patrick,
> 
> we appreciate a lot blog posts like the below.
> 
> Could you please also suggest a way that one could say had only one processor available, or two, how she could load say for example five scripts in a row, and somehow have the next one run, when the previous has just ended?
> 
> Say we have an mrc, and we need to make a number of transformations, or reports out of it. So, each script has as input tis one mrc, but roduces separate output files. For example, catmandu script a, produces mrca, and scripta.log, script#2 produces mrc2, and script2.log, etc.
> 
> Thank you
> 
> On 2016-04-20 14:57, Patrick Hochstenbach wrote:
>> As promised at our Berlin Workshop: here is the blog post
>> https://librecatproject.wordpress.com/2016/04/20/parallel-processing-with-catmandu/
>> Patrick
>> Patrick Hochstenbach - digital architect
>> University Library Ghent
>> Sint-Hubertusstraat 8 - 9000 Ghent - Belgium
>> patrick.hochstenbach at ugent.be
>> +32 (0)9 264 7980
>> _______________________________________________
>> librecat-dev mailing list
>> - send list mails to librecat-dev at lists.uni-bielefeld.de
>> - to unsubscribe or change options, visit
>> https://lists.uni-bielefeld.de/mailman2/cgi/unibi/listinfo/librecat-dev
>> - project website: http://librecat.org/
> 

Patrick Hochstenbach - digital architect
University Library Ghent
Sint-Hubertusstraat 8 - 9000 Ghent - Belgium
patrick.hochstenbach at ugent.be
+32 (0)9 264 7980