From b18a2dc375a67e5bb1c629beab34586eafdc3b32 Mon Sep 17 00:00:00 2001 From: Cody Boone Ferguson <53008573+xexyl@users.noreply.github.com> Date: Fri, 19 Jul 2024 08:59:28 -0700 Subject: [PATCH 1/2] Update Makefile for csv2entry and entry2csv The variables CSV2ENTRY and ENTRY2CSV have been added for the new rules that run the new rules, csv2entry and entry2csv, respectively, according to the log in 097dd49cc7e15cdc3458fe0b9e74b09da2e0852f. The rules have been added to .PHONY (thinking about this quickly I believe this is correct but if not it can be corrected) but not make help as I am not sure where in that rule they should be and I am also not sure what should be said even as a summary. The rules have a comment for which tool it runs but it also does not have a summary as I am not sure what it should say and I do not currently have the time to look into it. With this commit it is now easier to run these tools for testing and otherwise. --- Makefile | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index fc647d6b4..5ba00a9d2 100644 --- a/Makefile +++ b/Makefile @@ -66,6 +66,9 @@ GEN_STATUS= bin/gen-status.sh GEN_SITEMAP= bin/gen-sitemap.sh SORT_GITIGNORE= bin/sort.gitignore.sh FIND_MISSING_LINKS= bin/find-missing-links.sh +CSV2ENTRY= bin/csv2entry.sh +ENTRY2CSV= bin/entry2csv.sh + ############# @@ -242,10 +245,9 @@ indent.c: .PHONY: help genpath genfilelist verify_entry_files gen_authors gen_location gen_years \ find_missing_links test entry_index gen_top_html thanks gen_other_html quick_entry_index \ - gen_year_index quick_www www \ - untar_entry_tarball untar_year_tarball \ - form_entry_tarball form_year_tarball tar \ - gen_status gen_sitemap sitemap timestamp update + gen_year_index quick_www www untar_entry_tarball untar_year_tarball \ + form_entry_tarball form_year_tarball tar gen_status gen_sitemap \ + sitemap timestamp update csv2entry entry2csv # Suggest rules in this section # @@ -470,6 +472,21 @@ find_missing_links: ${FIND_MISSING_LINKS} -v 1 @echo '=-=-=-=-= IOCCC complete make $@ =-=-=-=-=' +# csv2entry.sh +# +csv2entry: + @echo '=-=-=-=-= IOCCC begin make $@ =-=-=-=-=' + ${CVS2ENTRY} -v 1 + @echo '=-=-=-=-= IOCCC complete make $@ =-=-=-=-=' + +# entry2csv.sh +# +entry2csv: + @echo '=-=-=-=-= IOCCC begin make $@ =-=-=-=-=' + ${ENTRY2CSV} -v 1 + @echo '=-=-=-=-= IOCCC complete make $@ =-=-=-=-=' + + # mostly harmless tests # test: From 06ceb851e4eede259d86657005192b47f0c31196 Mon Sep 17 00:00:00 2001 From: Cody Boone Ferguson <53008573+xexyl@users.noreply.github.com> Date: Fri, 19 Jul 2024 13:45:24 -0700 Subject: [PATCH 2/2] More updates for entry2csv and csv2entry The Makefile rules now have a comment for what they do. The help rule has not been updated. The bin/README.md has had some format fixes and some typo fixes as well. Rebuilt bin/index.html but these two files will have to be checked more thoroughly later. --- Makefile | 6 ++++-- bin/README.md | 57 ++++++++++++++++++++++++++------------------------ bin/index.html | 57 ++++++++++++++++++++++++++------------------------ 3 files changed, 64 insertions(+), 56 deletions(-) diff --git a/Makefile b/Makefile index 5ba00a9d2..37c1ee8ab 100644 --- a/Makefile +++ b/Makefile @@ -472,14 +472,16 @@ find_missing_links: ${FIND_MISSING_LINKS} -v 1 @echo '=-=-=-=-= IOCCC complete make $@ =-=-=-=-=' -# csv2entry.sh +# convert author_wins.csv, manifest.csv and year_prize.csv CSV files to +# .entry.json files. # csv2entry: @echo '=-=-=-=-= IOCCC begin make $@ =-=-=-=-=' ${CVS2ENTRY} -v 1 @echo '=-=-=-=-= IOCCC complete make $@ =-=-=-=-=' -# entry2csv.sh +# convert .entry.json files to author_wins.csv, manifest.csv and year_prize.csv +# CSV files. # entry2csv: @echo '=-=-=-=-= IOCCC begin make $@ =-=-=-=-=' diff --git a/bin/README.md b/bin/README.md index 027130af3..541b69c2b 100644 --- a/bin/README.md +++ b/bin/README.md @@ -142,9 +142,11 @@ Convert CSV files into `.entry.json` for all entries. This tool takes as input, the following CSV files: -- [author_wins.csv](#author_wins_csv) - author_handle followed by all their entry_ids -- [manifest.csv](#manifest_csv) - information about files under a entry -- [year_prize.csv](#year_prize_csv) - entry_id followed by the entry's award +- [author_wins.csv](#author_wins_csv) - `author_handle` followed by all their +`entry_id`s +- [manifest.csv](#manifest_csv) - information about files under an entry +- [year_prize.csv](#year_prize_csv) - `entry_id` followed by the entry's award +title This tool updates `.entry.json` files for all entries. Only those `.entry.json` files whose content is modified are written. @@ -167,7 +169,7 @@ this tool will modify the CSV file (if needed) in order to restore the CSV order and other canonicalizing processes. This tool will flag as an error, any empty fields, fields that are -an un-quoted _NULL_ or _null_, fields that start with whitespace, +an unquoted `NULL` or `null`, fields that start with whitespace, fields that ends with whitespace, or fields that contain consecutive whitespace characters. @@ -179,40 +181,41 @@ return line feeds" with "newlines. We also make sure that the CSV file ends in a newline. We do this because some spreadsheet applications, when exporting to a CSV file, do not do this. -We also sort the CSV files in the same way that `bin/entry2csv.sh` -sorts its CSV output files. We do this in case the -CSV files were imported into a spreadsheet where their order -was changed before exporting. This means one is free -to order the CSV file content as you wish as this -tool will reset these CSV file. +We also sort the CSV files in the same way that [entry2csv](#entry2csv) sorts +its CSV output files. We do this in case the CSV files were imported into a +spreadsheet where their order was changed before exporting. This means one is +free to order the CSV file content as you wish as this tool will reset these CSV +file. -Next this tool processes the non-CSV comment lines in manifest.csv. -The 1st and 2nd fields of [manifest.csv](#manifest_csv) prefer to entry YYYY and -entry sub-directory (i.e., the YYYY/dir directory under the -root of the git repository). From that list of YYYY/dir -IOCCC entry directories, we will create the `.entry.json` files. -We only modify those `.entry.json` files when their content changes. +Next this tool processes the non-CSV comment lines in +[manifest.csv](#manifest_csv). The 1st and 2nd fields of +[manifest.csv](#manifest_csv) refer to entry YYYY and entry subdirectory (i.e., +the `YYYY/dir` directory under the root of the git repository). From that list of +`YYYY/dir` IOCCC entry directories, we will create the `.entry.json` files. We +only modify those `.entry.json` files when their content changes. **NOTE**: While this tool uses `jparse(1)` to verify that the modified -`.entry.json` contains valid JSON content, this tool does NOT -perform any semantic checks. For example, this tool does NOT +`.entry.json` contains valid JSON content, **this tool does NOT +perform any semantic checks**. For example, this tool does NOT verify that the manifest in the `.entry.json` file matches the -files in the YYYY/dir directory, or even that the `.entry.json` +files in the `YYYY/dir directory`, or even that the `.entry.json` contains a manifest (or any of the other required JSON content). -
+
### [entry2csv.sh](%%REPO_URL%%/bin/entry2csv.sh)
This tool takes as input, all entry `.entry.json` files and updates 3 CSV files: -- [author_wins.csv](#author_wins_csv) - author_handle followed by all their entry_ids +- [author_wins.csv](#author_wins_csv) - `author_handle` followed by all their +`entry_id`s - [manifest.csv](#manifest_csv) - information about files under a entry -- [year_prize.csv](#year_prize_csv) - entry_id followed by the entry's award +- [year_prize.csv](#year_prize_csv) - `entry_id` followed by the entry's award +title The CSV files are written in a canonical UNIX format form. @@ -222,19 +225,19 @@ Only those CSV files files whose content is modified are written. We generate CSV files from the `.entry.json` files from winning IOCCC entries listed under years listed in the `.top` file, -and in sub-directories listed in the `YYYY/.year` file for the +and in subdirectories listed in the `YYYY/.year` file for the given year. Only those entries so listed are processed. All IOCCC entry directories must have a `.path` file that lists -the path of the entry's directory from the TOPDIR. +the path of the entry's directory from the `TOPDIR`. **NOTE**: -When adding a new IOCCC years entries, the `.top` file MUST -be updated, and the new IOCCC year `YYYY/.year` files MUST +When adding new IOCCC years' entries, the `.top` file **MUST** +be updated, and the new IOCCC year `YYYY/.year` files **MUST** reference the directory of the new IOCCC entries. They must also contain a `.path` file that contains the path -of the IOCCC entry directory from the TOPDIR. +of the IOCCC entry directory from the `TOPDIR`.
diff --git a/bin/index.html b/bin/index.html index 1a0d5983b..fe2555098 100644 --- a/bin/index.html +++ b/bin/index.html @@ -465,9 +465,11 @@ the top level Makefile by:

Convert CSV files into .entry.json for all entries.

This tool takes as input, the following CSV files:

This tool updates .entry.json files for all entries. Only those .entry.json files whose content is modified are written.

@@ -483,7 +485,7 @@ modifies the content and final exports back to the CSV file, this tool will modify the CSV file (if needed) in order to restore the CSV order and other canonicalizing processes.

This tool will flag as an error, any empty fields, fields that are -an un-quoted NULL or null, fields that start with whitespace, +an unquoted NULL or null, fields that start with whitespace, fields that ends with whitespace, or fields that contain consecutive whitespace characters.

Internal details of bin/csv2entry.sh

@@ -491,50 +493,51 @@ whitespace characters.

return line feeds” with “newlines. We also make sure that the CSV file ends in a newline. We do this because some spreadsheet applications, when exporting to a CSV file, do not do this.

-

We also sort the CSV files in the same way that bin/entry2csv.sh -sorts its CSV output files. We do this in case the -CSV files were imported into a spreadsheet where their order -was changed before exporting. This means one is free -to order the CSV file content as you wish as this -tool will reset these CSV file.

-

Next this tool processes the non-CSV comment lines in manifest.csv. -The 1st and 2nd fields of manifest.csv prefer to entry YYYY and -entry sub-directory (i.e., the YYYY/dir directory under the -root of the git repository). From that list of YYYY/dir -IOCCC entry directories, we will create the .entry.json files. -We only modify those .entry.json files when their content changes.

+

We also sort the CSV files in the same way that entry2csv sorts +its CSV output files. We do this in case the CSV files were imported into a +spreadsheet where their order was changed before exporting. This means one is +free to order the CSV file content as you wish as this tool will reset these CSV +file.

+

Next this tool processes the non-CSV comment lines in +manifest.csv. The 1st and 2nd fields of +manifest.csv refer to entry YYYY and entry subdirectory (i.e., +the YYYY/dir directory under the root of the git repository). From that list of +YYYY/dir IOCCC entry directories, we will create the .entry.json files. We +only modify those .entry.json files when their content changes.

NOTE:

While this tool uses jparse(1) to verify that the modified -.entry.json contains valid JSON content, this tool does NOT -perform any semantic checks. For example, this tool does NOT +.entry.json contains valid JSON content, this tool does NOT +perform any semantic checks. For example, this tool does NOT verify that the manifest in the .entry.json file matches the -files in the YYYY/dir directory, or even that the .entry.json +files in the YYYY/dir directory, or even that the .entry.json contains a manifest (or any of the other required JSON content).

-
+

This tool takes as input, all entry .entry.json files and updates 3 CSV files:

The CSV files are written in a canonical UNIX format form.

Only those CSV files files whose content is modified are written.

Internal details of bin/entry2csv.sh

We generate CSV files from the .entry.json files from winning IOCCC entries listed under years listed in the .top file, -and in sub-directories listed in the YYYY/.year file for the +and in subdirectories listed in the YYYY/.year file for the given year. Only those entries so listed are processed.

All IOCCC entry directories must have a .path file that lists -the path of the entry’s directory from the TOPDIR.

+the path of the entry’s directory from the TOPDIR.

NOTE:

-

When adding a new IOCCC years entries, the .top file MUST -be updated, and the new IOCCC year YYYY/.year files MUST +

When adding new IOCCC years’ entries, the .top file MUST +be updated, and the new IOCCC year YYYY/.year files MUST reference the directory of the new IOCCC entries. They must also contain a .path file that contains the path -of the IOCCC entry directory from the TOPDIR.

+of the IOCCC entry directory from the TOPDIR.

filelist.entry.json.awk