<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=Windows-1252"> <style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style> </head> <body dir="ltr"> <div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof"> Dear Martina</div> <div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof"> </div> <div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof"> Hints are added to the latest version of Catmandu v 1.2021 now available on CPAN.</div> <div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof"> </div> <div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof"> BR</div> <div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof"> Patrick</div> <div id="appendonsend"></div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr">From: Siebert, Dr. Martina <Martina.Siebert@sbb.spk-berlin.de> Sent: 03 November 2023 17:09 To: Patrick Hochstenbach <Patrick.Hochstenbach@UGent.be>; librecat-dev@lists.uni-bielefeld.de <librecat-dev@lists.uni-bielefeld.de> Subject: AW: [librecat-dev] Catmandu: field not converted to TSV/CSV output if not present in first (how many?) input records <div> </div> </div> <style>  </style> <div lang="DE" link="#0563C1" vlink="#954F72"> <div class="x_WordSection1"> Hi Patrick, makes 100% sense. Thanks for the heads-up and the ways around the “feature” ;-) Maybe a note in the “Description” that the first record serves as model for the fields exported would help the un-initiated like myself: <a href="https://metacpan.org/pod/Catmandu::Exporter::TSV" originalsrc="https://metacpan.org/pod/Catmandu::Exporter::TSV" shash="SWHO8ivysh4cxh7XcAJ6GDrKfCTw77HHkYMUd6B3WqQ+tX+DshLQ2g+dtnSh3G+xw58zQ4tnMIOgVqDa1O0DMUQ5lOLrKRguMz3CxMXB1s40qSP/XGUhW7sogjsLP/r41ZdkgRgBurHWmm+XSJ5hlWd/u35kSuTQuHJ3i7wmC7U="> https://metacpan.org/pod/Catmandu::Exporter::TSV</a> Best, Martina <div> <div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0cm 0cm 0cm"> Von: Patrick Hochstenbach <Patrick.Hochstenbach@UGent.be> Gesendet: Freitag, 3. November 2023 16:58 An: Siebert, Dr. Martina <Martina.Siebert@sbb.spk-berlin.de>; librecat-dev@lists.uni-bielefeld.de Betreff: Re: [librecat-dev] Catmandu: field not converted to TSV/CSV output if not present in first (how many?) input records </div> </div> Hello Martina, This is a known feature. Catmandu was created to work with very large files in streaming mode. This means that Catmandu doesn’t read the complete input data first to see what kind of fields are available. The software needs to work with the data at hand. The JSON format allows that every record have different fields, and doesn’t care if ‘late-comer’ records have a different field layout. Formats such as TSV and CSV don’t have this feature: from the first record you need to tell the software what fields need to be made available. Catmandu use two procedures: - If no information is provided, the field layout is guessed from the first record it receives. - One can provide information to Catmandu which fields one wants to see in the output. For the latter Catmandu has the `--fields` option: $ catmandu convert JSON to CSV –-fix my.fix –-fields ‘id,name,title,author’ < data.json As I am writing this, I see also a `--collect_fields 1` option, that does what you want. But first load all the data into memory. BR Patrick PS: Do to outlook issues some hyphens and quotes may be lost in the email <div id="x_mail-editor-reference-message-container"> <div> <div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0cm 0cm 0cm"> From: <a href="mailto:librecat-dev-bounces@lists.uni-bielefeld.de">librecat-dev-bounces@lists.uni-bielefeld.de</a> <<a href="mailto:librecat-dev-bounces@lists.uni-bielefeld.de">librecat-dev-bounces@lists.uni-bielefeld.de</a>> on behalf of Siebert, Dr. Martina <<a href="mailto:Martina.Siebert@sbb.spk-berlin.de">Martina.Siebert@sbb.spk-berlin.de</a>> Date: Friday, 3 November 2023 at 15:29 To: <a href="mailto:librecat-dev@lists.uni-bielefeld.de">librecat-dev@lists.uni-bielefeld.de</a> <<a href="mailto:librecat-dev@lists.uni-bielefeld.de">librecat-dev@lists.uni-bielefeld.de</a>> Subject: [librecat-dev] Catmandu: field not converted to TSV/CSV output if not present in first (how many?) input records </div> Hello, It seems not possible to produce a TSV/CSV output that includes fields that appear for the first time only much later in an input file. In the JSON export all is fine, but when exporting to TSV/CSV the “late-comer” fields are missing. When I fake-add the field to the first record the TSV/CSV export is correct. Is this a known bug? Can it be fixed? Best, Martina _____________________________________________ Dr. Martina Siebert Ostasienabteilung | CrossAsia Staatsbibliothek zu Berlin – Preußischer Kulturbesitz <a href="mailto:martina.siebert@sbb.spk-berlin.de">martina.siebert@sbb.spk-berlin.de</a> <a href="http://www.staatsbibliothek-berlin.de/" originalsrc="http://www.staatsbibliothek-berlin.de/" shash="NS0p6I0pmm50FpyuAFgBAPjdapK3MoeO213tOt9rPemHlEyvYRGwrj5FJ3nUUFPLj5KU5F7gqFyoKDHAEnzsOV/372wFgmcWekXOsd1ob6VXUIM4jt6Xs/G59OXZWpxW8uOQqNfgsadL3mMgR5tDhkXxhA9QZLykJJBkzCg6Zic=">www.staatsbibliothek-berlin.de</a> Im Rahmen der E-Mail-Kommunikation werden gegebenenfalls personenbezogene Daten verarbeitet. Unsere Hinweise zum Datenschutz finden Sie hier: <a href="http://sbb.berlin/datenschutz" originalsrc="http://sbb.berlin/datenschutz" shash="aEFzyQET1H74Q6Lxb+n4sfJ/aBRjxhcUV45ukfTl0WFsjY953Zx5hW9uowWqsnsl/xVPZoj+KQiAKhPdtTghl/10c5aJyoymXz5Y8dcKCOirV5BIdaE1a7PTDZZVL2Ov3EpumqJN6+Poq7C0aDVHHl4WOG1AGy9TZbHpdFsP9tE=" target="_short" title="Kurz-URL"> http://sbb.berlin/datenschutz</a> </div> </div> </div> </div> </body> </html>