mirror/ioccc
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ecabb73bf65bbd304dd7ca78d045cfb8365f89b4
ioccc/tmp/README.md
T
Landon Curt Noll aedb017223 improve markdown to HTML tools, build location.html, authors.html, years.html 
Renamed `.winner.json` to `.entry.json` as per recent decisions made for issue #2156.

JSON element _winner_JSON_format_version_ renamed to _entry_JSON_format_version_
in `.winner.json`.

Changed _entry_JSON_format_version_ from "1.0 2024-01-14" to "1.1 2024-02-11".

JSON element _winner_id_ renamed to _entry_id_ in `.entry.json`
JSON element _winner_text renamed to _entry_text_ in `.entry.json`
JSON element _winner_id_ renamed to _entry_id_ in `author/author_handle.json` files.

Changed _author_JSON_format_version_ from "1.0 2023-06-10" to "1.1 2024-02-11".

Renamed `tmp/gen_winner_json.sh` to `tmp/gen_entry_json.sh`.

Renamed `tmp/example.dot_winner.json` to `tmp/example.dot_entry.json`.

Renamed `tmp/winner_id.txt` to `tmp/entry_id.txt`.

Renamed `tmp/winner_dir.txt` to `tmp/entry_dir.txt`.

Moved outdated `tmp/awards_check.sh` into `tmp/OLD_manifest/awards_check.sh`.
Moved outdated `tmp/awards_check.sh` into `tmp/OLD_manifest/awards_check.sh`.
Moved outdated `tmp/src/winner_name` into `tmp/OLD_manifest/winner_name`.
Moved outdated `tmp/sql` into `tmp/OLD_manifest/sql`.
Moved outdated `tmp/template-footer.html` into `tmp/OLD_manifest/template-footer.html`.
Moved outdated `tmp/template-header.html` into `tmp/OLD_manifest/template-header.html`.
Moved outdated `tmp/template-top.html` into `tmp/OLD_manifest/template-top.html`.
Moved outdated `tmp/test.html` into `tmp/OLD_manifest/test.html`.
Moved outdated `tmp/fake.gen_index.sh` into `tmp/OLD_manifest/fake.gen_index.sh`.
Moved outdated `tmp/old-ioccc.css` into `tmp/OLD_manifest/old-ioccc.css`.

Removed winner_name from `tmp/src/Makefile`.

Updated `tmp/example.dot_entry.json` to reflect the above JSON changes.

Updated `tmp/manifest.numbers` to reflect the above JSON changes.

Updated `tmp/README.md` to reflect the above JSON changes.
Sorted `tmp/README.md` by existing `tmp` filenames.

Updated all `.entry.json` files to reflect the above JSON changes.

Updated all `author/author_handle.json` files to reflect the above JSON changes.

Added new tool `bin/gen-location.sh` to form the top level file `location.html`
from `author/author_handle.json` files!

Added new tool `bin/gen-years.sh` to form the top level file `years.html`
using `.top`, `.year` and `.entry.json' files!

Added new tool `bin/gen-authors.sh` to form a new top level file
`/authors.html` from `author/author_handle.json` and related
`.entry.json' files!

The old `winners.html` file now redirects to the new `authors.html` file.

Renamed `bin/gen-author.sh` to `bin/output-index-author.sh`.
Renamed `bin/subst.winner-index.sh` to `subst.entry-index.sh`.
Renamed `bin/subst.winner-index.awk` to `subst.entry-index.awk`.
Renamed `bin/manifest.entry.json.awk` to `bin/manifest.entry.json.awk`.
Renamed `bin/gen-inventory.sh` to `bin/output-index-inventory.sh`.
Renamed `bin/sort.gitignore` to `bin/sort.gitignore.sh`.
Renamed `bin/sgi` to `bin/sgi.sh`.

The core logic of converting markdown to HTML has been
generalized into a new `bin/md2html.sh` tool.

The `bin/readme2index.sh` tool now calls the
new `bin/md2html.sh` tool to form entry's `index.html` files.

The `bin/quick-readme2index.sh` tool now calls the new `bin/md2html.sh`
tool, unless the `index.html` is newer than both `.entry.json` and
`README.md`, to form entry `index.html` files.

Increased, slightly, the speed of the `bin/quick-readme2index.sh` tool
with the entry's `index.html` file does not need to be updated.

The `bin/quick-readme2index.sh` tool will touch the `index.html` file,
even if the `bin/md2html.sh` tool determines that the `index.html` file
did not change, so that the `index.html` file will not appear
out of date with respect to `.entry.json`.

Slightly improved the look and feel of `index.html` files for entries.

The logic of `bin/all-run.sh` has been improved.  The initial
processing, from first bash version check, thru the -N test, serves
as the model for the `bin/readme2index.sh` tool and the
`bin/quick-readme2index.sh` tool.  There tools now share
a common set of command line options.

Changed the format of `inc/md2html.cfg` so that for a given "_file_glob_"
match, options follow, one per line.  The "_file_glob_" is on a line
by itself.  The option lines that follow must be intended:

```
[12][0-9][0-9][0-9]/README.md
	-H
	topnav=year
	-o
	bin/subst.default.sh
	-o
	bin/subst.year.sh
```

Added "-D ./" to top level markdown files in `inc/md2html.cfg`.

The above intending is performed with 1 or more whitespace
characters, although we use a leading tab "convention"
out of deference to Makefile formats. :-)

Once a new "_file_glob_" is found, the set of any previous
option lines is said to have ended.

The general argument processing logic now makes used of shell arrays.

Added checks in the `bin/md2html.sh` tool to disallow a "_file_glob_"
of . (dot) as this is used internally to help the shell process the
new `inc/md2html.cfg` format.

Changed `bin/all-run.sh` VERSION from "1.1.2 2024-01-20"
to "1.3 2024-02-11".

Changed `bin/output-index-author.sh` VERSION from "1.3.1 2024-01-16"
to "1.4 2024-02-11".

Changed `bin/pandoc-wrapper.sh` VERSION from "1.3.2 2024-01-19"
to "1.4 2024-02-05".

Changed `bin/quick-readme2index.sh` VERSION from "1.0 2024-01-20"
to "1.1 2024-02-11".

Changed `bin/readme2index.sh` VERSION from "1.1.1 2024-01-20"
to "1.2 2024-02-11".

Changed `bin/subst.default.sh` VERSION from "1.1.1 2024-01-16"
to "1.2 2024-02-05".

Changed `bin/subst.entry-index.sh` VERSION from "1.1.1 2024-01-16
to "1.2 2024-02-11".

Changed `bin/output-index-inventory.sh` VERSION from "1.0.1 2024-01-16"
to "1.1 2024-02-11".

Changed `bin/manifest.entry.json.awk` from "1.0 2024-01-14"
to "1.1 2024-02-11".

Changed `bin/sort.gitignore.sh` from "1.0 2024-02-01"
to "1.1 2024-02-11".

Changed substitution symbol _%%DOCROOT_URL_SLASH%%_ to _%%DOCROOT_SLASH%%_
in various "inc/*.html" files.

Moved setting of symbol _%%DOCROOT_SLASH%%_ from `bin/subst.entry-index.sh`
to `bin/subst.default.sh`.

Changed the variable _$DOCROOT_URL_SLASH_ to _$DOCROOT_SLASH_, in
bin tools that use it, to be more descriptive.

Removed `-P pandoc_opts` from `bin/*.sh` command lines.
The PANDOC_WRAPPER_OPTSTR value is only used in `bin/pandoc-wrapper.sh`.

Removed `-B optstr` and `-A optstr` from the `bin/md2html.sh` command line.

Removed `[-O tool=optstr ..]` from the `bin/md2html.sh` command line.

Removed old use of `-U top_url` from command lines.
Add `-w site_url` as sort of a "replacement" for the old `-U top_url`.

Added a new `-U URL` to `bin/gen-author.sh`, `bin/output-index-inventory.sh`,
`bin/md2html.sh`, `bin/subst.default.sh`, and `bin/subst.entry-index.sh`
to allow for the Nu HTML check can use the encoded absolute URL of
the HTML page.

Added `-t tagline` to most command lines.

Added `-T md2html.sh` to most command lines.

Added `-D docroot/` to most command lines to set DOCROOT_SLASH.
The DOCROOT_SLASH value defaults to "../../" for most in tools,
suitable for HTML files under a `YYYY/dir` directory, except for
`bin/gen-location.sh` and `gen-entries-html.sh` (as this produces a
top level HTML file) and `bin/md2html.sh` (forms files at any level)
who use "./" instead.

In the case of `bin/output-index-inventory.sh`, `bin/quick-readme2index.sh`,
`bin/readme2index.sh`, and `bin/all-run.sh`

Helped make the use of exit code in bin tools a bit more consistent.

Fixed a number of typos in the bin tools.

Added comments for lower level tools such as
`manifest.entry.json.awk`, `bin/subst.default.sh`,
`bin/subst.entry-index.sh`, and bin/`manifest.entry.json.awk` to
indicate how they are usually invoked by other tools.

Changed the comments that a new `bin/md2html.sh` tool (that was the
`bin/readme2index.sh` tool) would add to a entry's `index.html` file.
No longer will it add the tool version number to the comment.  Thus,
a change in tool version will not force all entry `index.html`
files to be modified.  Now both the tool adds two comments, one
is the _tagline_ to indicate how the markdown content was generated,
and one for the new `bin/md2html.sh` tool.  For example:

```
<!-- The original markdown content was formed by the tool: bin/quick-readme2index.sh -->
<!-- The HTML was converted from markdown by the tool: bin/md2html.sh -->
```

Improve the debugging, especially in regards to changing important
exported shell variables, in `bin/md2html.sh`.  Other bin tools
have their exported shell variables improved somewhat as well.

The `bin/README.md` now describes the general _converting markdown
to HTML_ procedure including _getopt phases_ and _HTML phases_.
Content from `inc/md2html.cfg` and `bin/readme2index.sh` that related
to this topic has been centralized into the `bin/README.md` file.

Improved formatting of the `bin/README.md` file.  Updated it as needed
according to the above.

Sorted `2011/.year`, `2018/.year`, and `2019/.year` to be in dictionary order.
Adjusted ${WINNERS} in year Makefile to use dictionary order so
that `make genpath` will produce `.year` files in dictionary order.

FYI: We tested this massive set of changes as best as we could.
     Nevertheless we might have missing something in the JSON changes,
     `.entry.json` renames, manifest updates, code comments, etc.
2024-02-11 16:00:31 -08:00

481 lines
15 KiB
Markdown
Raw Blame History

# tmp
The tools under this directory are an alpha release.
The [build-ioccc repo](https://github.com/ioccc-src/build-ioccc/tree/master)
will hold the beta and formal release of tools that will
maintain the [official IOCCC web site](https://www.ioccc.org).
Currently these tools maintain the [experimental web site](https://ioccc-src.github.io/temp-test-ioccc/)
by default, via the [temp-test-ioccc GitHub repo](https://github.com/ioccc-src/temp-test-ioccc).
These tools will redefault to using the temp-test-ioccc repo and the experimental web site
until the [official IOCCC web site](https://www.ioccc.org), via the
[IOCCC winner repo](https://github.com/ioccc-src/winner) is ready for the mass pull request/merge.
**IMPORTANT**: The purpose of the `tmp` directory is to hold temporary files.
The files in the `tmp` directory will go away leaving behind an empty `tmp` directory.
## purpose
This documentation is here to help define terms, files under `tmp/`
and the format of those files.
*NOTE*: The `terms` section may be moved to another file
later on, after the contents of this directory are removed.
## files
### [OLD_manifest](OLD_manifest)
The files in the [OLD_manifest](OLD_manifest) were in the top of
the tmp directory as of 2023 Oct 26. They represent the first
complete manifest that matched the _YYYY/dir_ entry directories.
These files in this directory are kept as a temporary snapshot.
Some obsolete files and tools were moved under [OLD_manifest](OLD_manifest).
### [README.md](README.md)
This file.
### [author.csv](author.csv)
The [author.csv](author.csv) is a CSV file that was exported from the
[author.numbers](author.numbers) file.
In case of conflict, the data in [author.numbers](author.numbers) file is
considered authoritative over the [author.csv](author.csv) file.
The [author.csv](author.csv) is generated from the [author.numbers](author.numbers) file,
via the [macOS](https://www.apple.com/macos/)
[Numbers](https://www.apple.com/numbers/) spreadsheet app, as follows:
0. open [author.csv](author.csv) in numbers: modify if/as needed
1. File -> Export To -> CSV...
* 1a. `[ ]` Include table names (unset)
* 1b. Text Encoding: Unicode (UTF-8)
* 1c. Click ((Save..))
2. Save As: author.csv
* 2a. Select the `tmp` directory
3. Click ((Export))
* 3a. If needed click ((Replace))
4. Execute ./[fix_csv.sh](fix_csv.sh)
### [author.numbers](author.numbers)
A [macOS](https://www.apple.com/macos) [Numbers](https://www.apple.com/numbers/)
spreadsheet contains information about `author`s: one line per `author` with the
following fields:
1. This field is `sort_word` string.
2. This field is the full name of the `author`.
3. This field is the `author_handle` of the `author`.
4. This field is the primary URL of `author` or the string `null` if none was
given.
5. This field is the email address of `author` or the string `null` if none was
given.
6. This field is the location ISO 3166 code of `author` or the string `null` is unknown.
An `author` who does not wish to specify a location ISO 3166 code is encouraged to
use the code `XX`. The location ISO 3166 code consists of two UPPER CASE ASCII letters.
7. This field is the twitter handle of the `author` or the string `null` if none
was given. This field is kept for purely historic record purposes and is
otherwise not used.
8. This field is the mastodon handle of the `author` or the string `null` if
none was given.
9. This field is the alternate URL of `author` or the string `null` if none
was given.
### [author_handle.txt](author_handle.txt)
A text file containing `author_handle`s for each `author`: one `author_handle`
per line.
### [author_names.txt](author_names.txt)
A text file of full names of each `author`: one full name per line.
In the case of anonymous authors the form is:
```
Anonymous YYYY
```
### [author_wins.csv](author_wins.csv)
The [author_wins.csv](author_wins.csv) is a CSV file that was exported from the
[author_wins.numbers](author_wins.numbers) file.
In case of conflict, the data in [author_wins.numbers](author_wins.numbers) file is
considered authoritative over the [author_wins.csv](author_wins.csv) file.
The [author_wins.csv.csv](author_wins.csv) is generated from the [author_wins.numbers](author_wins.numbers) file,
via the [macOS](https://www.apple.com/macos/)
[Numbers](https://www.apple.com/numbers/) spreadsheet app, as follows:
0. open [author_wins.numbers](author_wins.numbers) in numbers: modify if/as needed
1. File -> Export To -> CSV...
* 1a. `[ ]` Include table names (unset)
* 1b. Text Encoding: Unicode (UTF-8)
* 1c. Click ((Save..))
2. Save As: author_wins.csv
* 2a. Select the `tmp` directory
3. Click ((Export))
* 3a. If needed click ((Replace))
4. Execute ./[fix_csv.sh](fix_csv.sh)
### [author_wins.numbers](author_wins.numbers)
A [macOS](https://www.apple.com/macos) [Numbers](https://www.apple.com/numbers/)
spreadsheet: one line per `author`.
The first field is an `author_handle`.
The other fields are the `entry_id`'s of all _YYYY/dir_ entry's won by the `author`.
### [check_path_list.sh](check_path_list.sh)
Look for problems with path_list.required.txt, path_list.manifest.txt,
and path_list.found.txt files. In particular, the files found in
winning directories are compared with the list of required files
as well as the complete manifest list of files.
If all is OK, [check_path_list.sh](check_path_list.sh) exits 0 and prints
nothing. If there are missing required files, or extra files found
then [check_path_list.sh](check_path_list.sh) will exit non-zero and
print information about problems.
### [echo.sh](echo.sh)
Tool that simply echos it arguments: used to test [run_all.sh](run_all.sh).
### [entry_dir.txt](entry_dir.txt)
A list of paths to _YYYY/dir_ entry directories that reside
under year directories.
### [entry_id.txt](entry_id.txt)
A text file containing `entry_id` strings: one `entry_id` per line.
### [example.author.json](example.author.json)
This is a JSON file containing an example `author/author_handle.json` for a
fictitious `author`.
An `author/author_handle.json` file will be derived from the contents of the
[author.csv](author.csv) file and the [author_wins.csv](author_wins.csv) file.
### [example.dot_entry.json](example.dot_entry.json)
This is a JSON file containing an example `year/dir/.entry.json` for a
fictitious winning _YYYY/dir_ entry.
A `year/dir/.entry.json` file will be derived from the contents the `year/dir/.id` file,
the [author_wins.csv](author_wins.csv) file, the `year/dir/.year` file, and the
the [manifest.csv](manifest.csv) file.
The 11 field names in [manifest.numbers](manifest.numbers) match
the JSON member names found in
[example.dot_entry.json](example.dot_entry.json).
### [fix_csv.sh](fix_csv.sh)
This tool will fix CSV files exported by
[macOS](https://www.apple.com/macos)
[Numbers](https://www.apple.com/numbers/) by removing carriage
returns and appending a newline if it was missing.
Fixing involves these actions:
```sh
tr -d '\015' < input.csv > tmp.csv
echo >> tmp.csv
mv -f tmp.csv input.csv
```
This tool also forms [path_list.built.txt](path_list.built.txt),
[path_list.manifest.txt](path_list.manifest.txt), and
[path_list.required.txt](path_list.required.txt) from
[manifest.csv](manifest.csv).
### [format-headers.sh](format-headers.sh)
This tool helped format the `README.md` files for _YYYY/dir_ winning entries.
### [gen_author_json.sh](gen_author_json.sh)
Temporary tool used to generate the `author/author_handle.json` files.
The following command, executed in this directory, created the `author/` directory:
```sh
rm -rf ../author ; ./gen_author_json.sh author_handle.txt author.csv author_wins.csv ../author
```
### [gen_entry_json.sh](gen_entry_json.sh)
Generate a `.entry.json` for a given winning _YYYY/dir_ entry.
### [gen_path_list.found.sh](gen_path_list.found.sh)
The list of paths of files in all winning entries.
After doing a `make clobber` this tool updates, if needed,
form [path_list.found.txt](path_list.found.txt).
### [manifest.csv](manifest.csv)
The [manifest.csv](manifest.csv) is a CSV file that was exported from the
[manifest.numbers](manifest.numbers) file.
In case of conflict, the data in [manifest.numbers](manifest.numbers) file is
considered authoritative over the [manifest.csv](manifest.csv) file.
The [manifest.csv](manifest.csv) is generated from the [manifest.numbers](manifest.numbers) file,
via the [macOS](https://www.apple.com/macos/)
[Numbers](https://www.apple.com/numbers/) spreadsheet app, as follows:
0. open [manifest.numbers](manifest.numbers) in numbers: modify if/as needed
1. File -> Export To -> CSV...
* 1a. `[ ]` Include table names (unset)
* 1b. Text Encoding: Unicode (UTF-8)
* 1c. Click ((Save..))
2. Save As: manifest.csv
* 2a. Select the `tmp` directory
3. Click ((Export))
* 3a. If needed click ((Replace))
4. Execute ./[fix_csv.sh](fix_csv.sh)
### [manifest.numbers](manifest.numbers)
A [macOS](https://www.apple.com/macos) [Numbers](https://www.apple.com/numbers/)
spreadsheet contains information about files in _YYYY/dir_ entry directories under year directories.
This file as the following fields:
1. year:
IOCCC year as a 4-character string. Normally this would be a 4 digit year string,
however it may also be a string such as "mock".
NOTE: If year begins with "#", then dir is a comment, and the
rest of the row is to be ignored. Rows of this form do NOT
contain manifest information for a file.
2. dir:
Directory name number the IOCCC year.
3. path:
Path under the IOCCC/directory. In a few cases this is a path,
not just a simple filename under the IOCCC/directory.
4. inventory_order:
This number is the rank showing the order
that this file is to be listed for the given entry's in
the winning entry's `index.html` file.
If the number is 9 digits or less, then the given file
is considered a primary file when listed in the inventory
of files for a given winning _YYYY/dir_ entry.
If the number is 10 digits or more, then the given file
is considered a secondary file when listed in the inventory
of files for a given winning _YYYY/dir_ entry We recommend using the
value **4294967295** (2^32-1) for secondary files.
5. OK_to_edit:
If the value is `true`, then the file is one that may be
edited directly.
If the value is `false`, then the file should **NOT** be
edited, because the given file is generated by a tool.
One should **NOT** modify such a file directly, but
instead modify source files (i.e., markdown files,
JSON files, etc.)
6. display_as:
The type of given file.
7. display_via_github
If `true`, then the contents of the file should be viewed
via the GitHub repo.
If `false`, then the contents of the file should be viewed
in the web browser directly. In some cases this may
result in the file being downloaded instead of being displayed.
8. entry_text:
Any text that should be displayed at the end of line in `index.html`
(with a preceding " - "), or null is no such text is to be displayed.
The 8 field names in [manifest.numbers](manifest.numbers), must
match the JSON member names found in
[example.dot_entry.json](example.dot_entry.json).
NOTE: Cells containing true or false are JSON booleans.
NOTE: All other cells are JSON strings that need to be double quoted, including the year.
NOTE: Do not put commas, nor quotes, nor newlines in fields as these are bound to cause problems.
In case of conflict, the data in [manifest.numbers](manifest.numbers) file is
considered authoritative over the [manifest.csv](manifest.csv) file.
### [path_list.built.txt](path_list.built.txt)
A list of file paths (from the top directory) of _YYYY/dir_ entry directories,
that are built from other primary content files.
These paths are formed from [manifest.numbers](manifest.numbers)
entries where the "created_by" (5th) field is null. See
[manifest.numbers](manifest.numbers) for details on the fields.
This file is formed by the [fix_csv.sh](fix_csv.sh) tool.
### [path_list.found.txt](path_list.found.txt)
A list of files that exist, or should exist once the tools to generate certain files,
such as the `index.html`, or the `.entry.json` files for each entry, are created.
This file is created by [gen_path_list.found.sh](gen_path_list.found.sh).
### [path_list.manifest.txt](path_list.manifest.txt)
This is the list of file paths, both primary and constructed
files from _YYYY/dir_ entry directories.
This file is formed by the [fix_csv.sh](fix_csv.sh) tool.
### [path_list.required.txt](path_list.required.txt)
A list of file paths (from the top directory) of _YYYY/dir_ entry directories,
that are required to exist.
These paths are formed from [manifest.numbers](manifest.numbers)
entries where the "created_by" (5th) field is non-null. See
[manifest.numbers](manifest.numbers) for details on the fields.
This file is formed by the [fix_csv.sh](fix_csv.sh) tool.
### [run_all.sh](run_all.sh)
A tool to run another tool over all _YYYY/dir_ entry directories.
Examples:
```sh
./run_all.sh -v 3 ./fake.gen_index.sh
./run_all.sh -v 1 ./gen_entry_json.sh
```
### [src](src)
This is a temporary location for potential tools that are used to
build and maintain the www.ioccc.org web site.
Eventually a separate [build-ioccc repo](https://github.com/ioccc-src/build-ioccc/tree/master)
will hold these tools.
### [things-todo.md](things-todo.md)
A todo list of known things to check and/or do.
### [update-manifest.sh](update-manifest.sh)
Interactive script to update the IOCCC manifest step by step post manifest.csv export.
### [year_prize.csv](year_prize.csv)
The [year_prize.csv](year_prize.csv) is a CSV file that was exported from the
[year_prize.numbers](year_prize.numbers) file.
In case of conflict, the data in [year_prize.numbers](year_prize.numbers) file is
considered authoritative over the [year_prize.csv](year_prize.csv) file.
The [year_prize.csv.csv](year_prize.csv) is generated from the [year_prizes.numbers](year_prize.numbers) file,
via the [macOS](https://www.apple.com/macos/)
[Numbers](https://www.apple.com/numbers/) spreadsheet app, as follows:
0. open [year_prize.numbers](year_prize.numbers) in numbers: modify if/as needed
1. File -> Export To -> CSV...
* 1a. `[ ]` Include table names (unset)
* 1b. Text Encoding: Unicode (UTF-8)
* 1c. Click ((Save..))
2. Save As: year_prize.csv
* 2a. Select the `tmp` directory
3. Click ((Export))
* 3a. If needed click ((Replace))
4. Execute ./[fix_csv.sh](fix_csv.sh)
### [year_prize.numbers](year_prize.numbers)
A [macOS](https://www.apple.com/macos) [Numbers](https://www.apple.com/numbers/)
spreadsheet: one line per _YYYY/dir_ entry directory.
The first field is a `entry_id`.
The second field is the name of the award for given _YYYY/dir_ entry.
Reference in new issue View Git Blame Copy Permalink