mirror/ioccc
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add tools to manage .entry.json content

Browse Source 
Added `bin/csv2entry.sh` VERSION "1.0 2024-07-19".

In `bin/entry2csv.sh` changed VERSION from "1.0.2 2024-07-14"
to "1.0.3 2024-07-19".

The `bin/entry2csv.sh` tool converts all entry `.entry.json` files
into canonicalized `author_wins.csv`, `manifest.csv`, and `year_prize.csv`
CSV files.

The `bin/csv2entry.sh` tool converts `author_wins.csv`, `manifest.csv`,
and `year_prize.csv` (canonicalizing them if needed)  `.entry.json`
files for all entries.

To change the manifest for one or more entries:

```sh
    bin/entry2csv.sh -v 1
    # ((import `manifest.csv` into a spreadsheet tool))
    # ((modify in the spreadsheet tool, the content as needed))
    # ((export the spreadsheet contents, as the CSV file  `manifest.csv`))
    bin/csv2entry.sh -v 1
    make www
``

To change the award for one or more entries:

```sh
    bin/entry2csv.sh -v 1
    # ((import `year_prize.csv` into a spreadsheet tool))
    # ((modify in the spreadsheet tool, the content as needed))
    # ((export the spreadsheet contents, as the CSV file  `year_prize.csv`))
    bin/csv2entry.sh -v 1
    make www
``

To change the authorship of one or more entries:

```sh
    bin/entry2csv.sh -v 1
    # ((import `author_wins.csv` into a spreadsheet tool))
    # ((modify in the spreadsheet tool, the content as needed))
    # ((export the spreadsheet contents, as the CSV file  `author_wins.csv`))
    bin/csv2entry.sh -v 1
    # ((modify winning_entry_set of `author/author_handle.json` files as needed))
    make www
``

Performed `make www` to accomplish the above and to test under macOS.

TODO: Review and improve/expand as needed, changes that were made to `bin/README.md`.

TODO: Remove the `tmp` directory (after saving contents into private storage).
This commit is contained in:
Landon Curt Noll committed 2024-07-19 02:15:18 -07:00
1 parent ed866b45e8
commit 097dd49cc7
4 files changed
+1836 -8
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

bin/README.md
+210
View File
@@ -134,6 +134,109 @@ the top level `Makefile` by:
```
<div id="csv2entry">
### [csv2entry.sh](%%REPO_URL%%/bin/csv2entry.sh)
</div>
Convert CSV files into `.entry.json` for all entries.
This tool takes as input, the following CSV files:
- [author_wins.csv](#author_wins_csv) - author_handle followed by all their entry_ids
- [manifest.csv](#manifest_csv) - information about files under a entry
- [year_prize.csv](#year_prize_csv) - entry_id followed by the entry's award
This tool updates `.entry.json` files for all entries.
Only those `.entry.json` files whose content is modified are written.
For example:
``` <!---sh-->
bin/csv2entry.sh -v 1
```
There is no requirement to sort the CSV files, nor convert
them to UNIX format, nor append a final newline to the file.
This tool will canonicalize CSV files before using them
as input. Thus, if one wishes to import the CSV file into
some spreadsheet such as the [macOS](https://www.apple.com/macos)
[Numbers](https://www.apple.com/numbers/) spreadsheet,
modifies the content and final exports back to the CSV file,
this tool will modify the CSV file (if needed) in order
to restore the CSV order and other canonicalizing processes.
This tool will flag as an error, any empty fields, fields that are
an un-quoted _NULL_ or _null_, fields that start with whitespace,
fields that ends with whitespace, or fields that contain consecutive
whitespace characters.
#### Internal details of `bin/csv2entry.sh`
We first canonicalize the CSV files by replacing any "carriage
return line feeds" with "newlines. We also make sure that the CSV
file ends in a newline. We do this because some spreadsheet
applications, when exporting to a CSV file, do not do this.
We also sort the CSV files in the same way that `bin/entry2csv.sh`
sorts its CSV output files. We do this in case the
CSV files were imported into a spreadsheet where their order
was changed before exporting. This means one is free
to order the CSV file content as you wish as this
tool will reset these CSV file.
Next this tool processes the non-CSV comment lines in manifest.csv.
The 1st and 2nd fields of [manifest.csv](#manifest_csv) prefer to entry YYYY and
entry sub-directory (i.e., the YYYY/dir directory under the
root of the git repository). From that list of YYYY/dir
IOCCC entry directories, we will create the `.entry.json` files.
We only modify those `.entry.json` files when their content changes.
**NOTE**:
While this tool uses `jparse(1)` to verify that the modified
`.entry.json` contains valid JSON content, this tool does NOT
perform any semantic checks. For example, this tool does NOT
verify that the manifest in the `.entry.json` file matches the
files in the YYYY/dir directory, or even that the `.entry.json`
contains a manifest (or any of the other required JSON content).
<div id="entry2csv2">
### [entry2csv.sh](%%REPO_URL%%/bin/entry2csv.sh)
</div>
This tool takes as input, all entry `.entry.json` files
and updates 3 CSV files:
- [author_wins.csv](#author_wins_csv) - author_handle followed by all their entry_ids
- [manifest.csv](#manifest_csv) - information about files under a entry
- [year_prize.csv](#year_prize_csv) - entry_id followed by the entry's award
The CSV files are written in a canonical UNIX format form.
Only those CSV files files whose content is modified are written.
#### Internal details of `bin/entry2csv.sh`
We generate CSV files from the `.entry.json` files from winning
IOCCC entries listed under years listed in the `.top` file,
and in sub-directories listed in the `YYYY/.year` file for the
given year. Only those entries so listed are processed.
All IOCCC entry directories must have a `.path` file that lists
the path of the entry's directory from the TOPDIR.
**NOTE**:
When adding a new IOCCC years entries, the `.top` file MUST
be updated, and the new IOCCC year `YYYY/.year` files MUST
reference the directory of the new IOCCC entries. They
must also contain a `.path` file that contains the path
of the IOCCC entry directory from the TOPDIR.
<div id="filelist-entry-json-awk">
### [filelist.entry.json.awk](%%REPO_URL%%/bin/filelist.entry.json.awk)
</div>
@@ -1305,6 +1408,113 @@ A string that identifies the winning entry. The string is of the form:
```
<div id="csv">
# CSV files
</div>
The `bin/csv2entry.sh` and `bin/entry2csv.sh` tools use the following
3 CSV files. In the case of `bin/entry2csv.sh`, these 3 CSV files
are created / updated. In the case of `bin/csv2entry.sh`, these 3
CSV files are used as input.
<div id="author_wins_csv">
## author_wins.csv
</div>
A CSV spreadsheet: one line per `author`.
The first field is an `author_handle`.
The other fields are the `entry_id`'s of all _YYYY/dir_ entry's won by the `author`.
<div id="manifest_csv">
## manifest.csv
</div>
A CSV spreadsheet contains information about files in _YYYY/dir_ entry directories under year directories.
This file has the following fields:
1. year:
IOCCC year as a 4-character string. Normally this would be a 4 digit year string,
however it may also be a string such as "mock".
NOTE: If year begins with "#", then dir is a comment, and the
rest of the row is to be ignored. Rows of this form do NOT
contain manifest information for a file.
2. dir:
Directory name number the IOCCC year.
3. path:
Path under the IOCCC/directory. In a few cases this is a path,
not just a simple filename under the IOCCC/directory.
4. inventory_order:
This number is the rank showing the order
that this file is to be listed for the given entry's in
the winning entry's `index.html` file.
If the number is 9 digits or less, then the given file
is considered a primary file when listed in the inventory
of files for a given winning _YYYY/dir_ entry.
If the number is 10 digits or more, then the given file
is considered a secondary file when listed in the inventory
of files for a given winning _YYYY/dir_ entry We recommend using the
value **4294967295** (2^32-1) for secondary files.
5. OK_to_edit:
If the value is `true`, then the file is one that may be
edited directly.
If the value is `false`, then the file should **NOT** be
edited, because the given file is generated by a tool.
One should **NOT** modify such a file directly, but
instead modify source files (i.e., markdown files,
JSON files, etc.)
6. display_as:
The type of given file.
7. display_via_github
If `true`, then the contents of the file should be viewed
via the GitHub repo.
If `false`, then the contents of the file should be viewed
in the web browser directly. In some cases this may
result in the file being downloaded instead of being displayed.
8. entry_text:
Any text that should be displayed at the end of line in `index.html`
(with a preceding " - "), or null is no such text is to be displayed.
NOTE: Cells containing true or false are JSON booleans.
NOTE: All other cells are JSON strings that need to be double quoted, including the year.
NOTE: Do not put commas, nor quotes, nor newlines in fields as these are bound to cause problems.
<div id="year_prize_csv">
## year_prize.csv
</div>
A CSV spreadsheet: one line per _YYYY/dir_ entry directory.
The first field is a `entry_id`.
The second field is the name of the award for given _YYYY/dir_ entry.
<!--
Copyright © 1984-2024 by Landon Curt Noll. All Rights Reserved.
bin/csv2entry.sh
Executable
+1456
View File
File diff suppressed because it is too large. Load diff
bin/entry2csv.sh
+24 -8
View File
@@ -2,6 +2,19 @@
#
# entry2csv.sh - convert .entry.json data into CSV files
#
# This tool takes as input, all entry `.entry.json` files
# and updates 3 CSV files:
#
# author_wins.csv - author_handle followed by all their entry_ids
# manifest.csv - information about files under a entry
# year_prize.csv - entry_id followed by the entry's award
#
# The CSV files are written in a canonical UNIX format form.
#
# Only those CSV files files whose content is modified are written.
#
# Internal details of entry2csv.sh
#
# We generate CSV files from the `.entry.json` files from winning
# IOCCC entries listed under years listed in the `.top` file,
# and in sub-directories listed in the `YYYY/.year` file for the
@@ -10,11 +23,13 @@
# All IOCCC entry directories must have a `.path` file that lists
# the path of the entry's directory from the TOPDIR.
#
# NOTE: When adding a new IOCCC years entries, the `.top` file MUST
# be updated, and the new IOCCC year `YYYY/.year` files MUST
# reference the directory of the new IOCCC entries. They
# must also contain a `.path` file that contains the path
# of the IOCCC entry directory from the TOPDIR.
# NOTE:
#
# When adding a new IOCCC years entries, the `.top` file MUST
# be updated, and the new IOCCC year `YYYY/.year` files MUST
# reference the directory of the new IOCCC entries. They
# must also contain a `.path` file that contains the path
# of the IOCCC entry directory from the TOPDIR.
#
# NOTE: All IOCCC entries MUST have a valid `.entry.json` file.
#
@@ -42,6 +57,7 @@
#
# Share and enjoy! :-)
# firewall - run only with a bash that is version 5.1.8 or later
#
# The "/usr/bin/env bash" command must result in using a bash that
@@ -100,7 +116,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an
# set variables referenced in the usage message
#
export VERSION="1.0.2 2024-07-14"
export VERSION="1.0.3 2024-07-19"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
@@ -998,12 +1014,12 @@ fi
if [[ -z $NOOP ]]; then
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to: sort -t, -k1,1 -k2d,2 -k4n,4 -k3d,3 -k5d,8 $TMP_MANIFEST_CSV -o $TMP_MANIFEST_CSV" 1>&2
echo "$0: debug[5]: about to: sort -t, -k1,1 -k2d,2 -k4n,4 -k3,3 -k5d,8 $TMP_MANIFEST_CSV -o $TMP_MANIFEST_CSV" 1>&2
fi
sort -t, -k1,1 -k2d,2 -k4g,4 -k3d,3 -k5d,8 "$TMP_MANIFEST_CSV" -o "$TMP_MANIFEST_CSV"
status="$?"
if [[ status -ne 0 ]]; then
echo "$0: ERROR: sort -t, -k1,1 -k2d,2 -k4n,4 -k3d,3 -k5d,8 $TMP_MANIFEST_CSV -o $TMP_MANIFEST_CSV failed," \
echo "$0: ERROR: sort -t, -k1,1 -k2d,2 -k4n,4 -k3,3 -k5d,8 $TMP_MANIFEST_CSV -o $TMP_MANIFEST_CSV failed," \
"error code: $status" 1>&2
EXIT_CODE=8 # exit 8
elif [[ $V_FLAG -ge 3 ]]; then
bin/index.html
+146
View File
@@ -459,6 +459,82 @@ need to do so.</p>
<p>If you wish to run it on all entries, we recommend that this tool be invoked via
the top level <code>Makefile</code> by:</p>
<pre><code> make verify_entry_files</code></pre>
<div id="csv2entry">
<h3 id="csv2entry.sh"><a href="https://github.com/ioccc-src/temp-test-ioccc/blob/master/bin/csv2entry.sh">csv2entry.sh</a></h3>
</div>
<p>Convert CSV files into <code>.entry.json</code> for all entries.</p>
<p>This tool takes as input, the following CSV files:</p>
<ul>
<li><a href="#author_wins_csv">author_wins.csv</a> - author_handle followed by all their entry_ids</li>
<li><a href="#manifest_csv">manifest.csv</a> - information about files under a entry</li>
<li><a href="#year_prize_csv">year_prize.csv</a> - entry_id followed by the entry’s award</li>
</ul>
<p>This tool updates <code>.entry.json</code> files for all entries.
Only those <code>.entry.json</code> files whose content is modified are written.</p>
<p>For example:</p>
<pre><code> bin/csv2entry.sh -v 1</code></pre>
<p>There is no requirement to sort the CSV files, nor convert
them to UNIX format, nor append a final newline to the file.</p>
<p>This tool will canonicalize CSV files before using them
as input. Thus, if one wishes to import the CSV file into
some spreadsheet such as the <a href="https://www.apple.com/macos">macOS</a>
<a href="https://www.apple.com/numbers/">Numbers</a> spreadsheet,
modifies the content and final exports back to the CSV file,
this tool will modify the CSV file (if needed) in order
to restore the CSV order and other canonicalizing processes.</p>
<p>This tool will flag as an error, any empty fields, fields that are
an un-quoted <em>NULL</em> or <em>null</em>, fields that start with whitespace,
fields that ends with whitespace, or fields that contain consecutive
whitespace characters.</p>
<h4 id="internal-details-of-bincsv2entry.sh">Internal details of <code>bin/csv2entry.sh</code></h4>
<p>We first canonicalize the CSV files by replacing any “carriage
return line feeds” with “newlines. We also make sure that the CSV
file ends in a newline. We do this because some spreadsheet
applications, when exporting to a CSV file, do not do this.</p>
<p>We also sort the CSV files in the same way that <code>bin/entry2csv.sh</code>
sorts its CSV output files. We do this in case the
CSV files were imported into a spreadsheet where their order
was changed before exporting. This means one is free
to order the CSV file content as you wish as this
tool will reset these CSV file.</p>
<p>Next this tool processes the non-CSV comment lines in manifest.csv.
The 1st and 2nd fields of <a href="#manifest_csv">manifest.csv</a> prefer to entry YYYY and
entry sub-directory (i.e., the YYYY/dir directory under the
root of the git repository). From that list of YYYY/dir
IOCCC entry directories, we will create the <code>.entry.json</code> files.
We only modify those <code>.entry.json</code> files when their content changes.</p>
<p><strong>NOTE</strong>:</p>
<p>While this tool uses <code>jparse(1)</code> to verify that the modified
<code>.entry.json</code> contains valid JSON content, this tool does NOT
perform any semantic checks. For example, this tool does NOT
verify that the manifest in the <code>.entry.json</code> file matches the
files in the YYYY/dir directory, or even that the <code>.entry.json</code>
contains a manifest (or any of the other required JSON content).</p>
<div id="entry2csv2">
<h3 id="entry2csv.sh"><a href="https://github.com/ioccc-src/temp-test-ioccc/blob/master/bin/entry2csv.sh">entry2csv.sh</a></h3>
</div>
<p>This tool takes as input, all entry <code>.entry.json</code> files
and updates 3 CSV files:</p>
<ul>
<li><a href="#author_wins_csv">author_wins.csv</a> - author_handle followed by all their entry_ids</li>
<li><a href="#manifest_csv">manifest.csv</a> - information about files under a entry</li>
<li><a href="#year_prize_csv">year_prize.csv</a> - entry_id followed by the entry’s award</li>
</ul>
<p>The CSV files are written in a canonical UNIX format form.</p>
<p>Only those CSV files files whose content is modified are written.</p>
<h4 id="internal-details-of-binentry2csv.sh">Internal details of <code>bin/entry2csv.sh</code></h4>
<p>We generate CSV files from the <code>.entry.json</code> files from winning
IOCCC entries listed under years listed in the <code>.top</code> file,
and in sub-directories listed in the <code>YYYY/.year</code> file for the
given year. Only those entries so listed are processed.</p>
<p>All IOCCC entry directories must have a <code>.path</code> file that lists
the path of the entry’s directory from the TOPDIR.</p>
<p><strong>NOTE</strong>:</p>
<p>When adding a new IOCCC years entries, the <code>.top</code> file MUST
be updated, and the new IOCCC year <code>YYYY/.year</code> files MUST
reference the directory of the new IOCCC entries. They
must also contain a <code>.path</code> file that contains the path
of the IOCCC entry directory from the TOPDIR.</p>
<div id="filelist-entry-json-awk">
<h3 id="filelist.entry.json.awk"><a href="https://github.com/ioccc-src/temp-test-ioccc/blob/master/bin/filelist.entry.json.awk">filelist.entry.json.awk</a></h3>
</div>
@@ -1110,6 +1186,76 @@ is directly under a <code>year</code> directory.</p>
</div>
<p>A string that identifies the winning entry. The string is of the form:</p>
<pre><code> year_dir</code></pre>
<div id="csv">
<h1 id="csv-files">CSV files</h1>
</div>
<p>The <code>bin/csv2entry.sh</code> and <code>bin/entry2csv.sh</code> tools use the following
3 CSV files. In the case of <code>bin/entry2csv.sh</code>, these 3 CSV files
are created / updated. In the case of <code>bin/csv2entry.sh</code>, these 3
CSV files are used as input.</p>
<div id="author_wins_csv">
<h2 id="author_wins.csv">author_wins.csv</h2>
</div>
<p>A CSV spreadsheet: one line per <code>author</code>.</p>
<p>The first field is an <code>author_handle</code>.</p>
<p>The other fields are the <code>entry_id</code>’s of all <em>YYYY/dir</em> entry’s won by the <code>author</code>.</p>
<div id="manifest_csv">
<h2 id="manifest.csv">manifest.csv</h2>
</div>
<p>A CSV spreadsheet contains information about files in <em>YYYY/dir</em> entry directories under year directories.
This file has the following fields:</p>
<ol type="1">
<li><p>year:</p>
<p>IOCCC year as a 4-character string. Normally this would be a 4 digit year string,
however it may also be a string such as “mock”.</p>
<p>NOTE: If year begins with “#”, then dir is a comment, and the
rest of the row is to be ignored. Rows of this form do NOT
contain manifest information for a file.</p></li>
<li><p>dir:</p>
<p>Directory name number the IOCCC year.</p></li>
<li><p>path:</p>
<p>Path under the IOCCC/directory. In a few cases this is a path,
not just a simple filename under the IOCCC/directory.</p></li>
<li><p>inventory_order:</p>
<p>This number is the rank showing the order
that this file is to be listed for the given entry’s in
the winning entry’s <code>index.html</code> file.</p>
<p>If the number is 9 digits or less, then the given file
is considered a primary file when listed in the inventory
of files for a given winning <em>YYYY/dir</em> entry.</p>
<p>If the number is 10 digits or more, then the given file
is considered a secondary file when listed in the inventory
of files for a given winning <em>YYYY/dir</em> entry We recommend using the
value <strong>4294967295</strong> (2^32-1) for secondary files.</p></li>
<li><p>OK_to_edit:</p>
<p>If the value is <code>true</code>, then the file is one that may be
edited directly.</p>
<p>If the value is <code>false</code>, then the file should <strong>NOT</strong> be
edited, because the given file is generated by a tool.
One should <strong>NOT</strong> modify such a file directly, but
instead modify source files (i.e., markdown files,
JSON files, etc.)</p></li>
<li><p>display_as:</p>
<p>The type of given file.</p></li>
<li><p>display_via_github</p>
<p>If <code>true</code>, then the contents of the file should be viewed
via the GitHub repo.</p>
<p>If <code>false</code>, then the contents of the file should be viewed
in the web browser directly. In some cases this may
result in the file being downloaded instead of being displayed.</p></li>
<li><p>entry_text:</p>
<p>Any text that should be displayed at the end of line in <code>index.html</code>
(with a preceding ” - “), or null is no such text is to be displayed.</p></li>
</ol>
<p>NOTE: Cells containing true or false are JSON booleans.
NOTE: All other cells are JSON strings that need to be double quoted, including the year.
NOTE: Do not put commas, nor quotes, nor newlines in fields as these are bound to cause problems.</p>
<div id="year_prize_csv">
<h2 id="year_prize.csv">year_prize.csv</h2>
</div>
<p>A CSV spreadsheet: one line per <em>YYYY/dir</em> entry directory.</p>
<p>The first field is a <code>entry_id</code>.</p>
<p>The second field is the name of the award for given <em>YYYY/dir</em> entry.</p>
<!--
Copyright © 1984-2024 by Landon Curt Noll. All Rights Reserved.