# bin
> "After 50+ years, the UNIX command line remains very powerful." -- Landon Noll
>
> An answer to (Monty)? Pythonesque question of why these tools
> weren't written in a certain language. :-)
The [bin directory](index.html) holds tools that build files, such as HTML content, for the
[official IOCCC website](https://www.ioccc.org).
For HTML content, the [bin directory](index.html) tools make use of HTML
fragments from the [inc directory](%%REPO_URL%%/inc/index.html) as well as various JSON
files and other content from the [official IOCCC winner repo](https://github.com/ioccc-src/winner).
## [bin/](index.html) tools options
With the exception of the `awk(1)` and `sed(1)` scripts, all the tools in this
directory support a number of options that can be used to get help, diagnose
problems, see progress etc. These options are described below.
### Get help / usage string of a tool
If you need to remember the syntax of the tool or get certain notes about
different options, you can use the `-h` option.
For instance if you want help on the [all-run.sh](index.html#all-run) tool
from the root directory (of the repo/website), you would do:
```
bin/all-run.sh -h
```
### Set verbosity level of a tool
If you want verbosity, say for debugging purposes or to see what is going on
more (than the default), you should use the `-v level` option. For instance if
you wish to see what is going on with the
[quick-readme2index.sh](index.html#quick-readme2index) tool, you might do:
```
bin/quick-readme2index.sh -v 3
```
to set the verbosity level to `3`. The default for verbosity is `0`, no
verbosity, though using the [top level Makefile](%%REPO_URL%%/Makefile) will,
for some tools, set a verbosity level.
### Get version string of a tool
If you just want to know what version the tool is, you can use the `-V` option.
For instance to see what version the [chk-entry.sh](index.html#chk-entry) tool
is, you would do:
```
bin/chk-entry.sh -V
```
### Other notes about some of the common options
These options, and especially `-h` and `-v level`, can be very useful to get
basic usage information and to see what is going on when the tool is running.
For more details on each tool, including the ones mentioned above, see below. As
you go through each tool, if you need to understand more of it, we recommend
that you use the `-h` option on it first.
There are some scripts that are invoked by the
[bin/md2html.cfg](%%REPO_URL%%/bin/md2html.cfg) file but some of these tools can
be directly invoked as well, should you wish to see their output or if you have
some odd need to do so.
### [all-jfmt.sh](%%REPO_URL%%/bin/all-jfmt.sh)
Canonically format all entry and author JSON files
Usage:
```
bin/all-jfmt.sh -v 3
```
Alternate usage:
```
make all_jfmt
```
### [all-years.sh](%%REPO_URL%%/bin/all-years.sh)
Run a command on all IOCCC years.
Usage:
```
bin/all-years.sh -v 1 bin/gen-year-index.sh -v 1
```
If you wish to run instead, for example, [chk-entry.sh](#chk-entry), then do:
```
bin/all-years.sh -v 1 bin/chk-entry.sh
```
### [all-run.sh](%%REPO_URL%%/bin/all-run.sh)
Run a command on all winning entries.
Usage:
```
bin/all-run.sh -v 3 bin/quick-readme2index.sh -v 1
```
If you wish to run instead, for example, [readme2index.sh](#readme2index), then
do:
```
bin/all-run.sh -v 3 bin/readme2index.sh -v 1
```
### [chk-entry.sh](%%REPO_URL%%/bin/chk-entry.sh)
Check an entry directory to verify that all the files in its manifest
(`.entry.json`) exist in the git repo.
Usage:
```
bin/chk-entry.sh 2020/ferguson1
```
If you wish to run it on all entries, we recommend that this tool be invoked via
the top level `Makefile`:
```
make verify_entry_files
```
**NOTE**: see the
FAQ on "[.entry.json files](../faq.html#entry_json)"
for more details on `.entry.json` files.
Combine all author/author_handle.json files as single JSON file.
The purpose of this tool is to make looking for information across all
authors faster by temporarily forming them into a single JSON file.
Because `jsp(1)` open remembers the last copy of a given JSON member name.
As such, we change the JSON member name "winning_entry_set" into
a unique "winning_entry_set.FILENO" where FILENO is the file number.
This the JSON member value will be preserved across all files.
We also convert where FILENO is the file number and FILENAME is the filename:
```
"sort_word" : "data",
```
into:
```
"sort_word.FILENO" : [
{
"sort_word" : "data FILENAME"
}
],
```
This will allow `jsp(1)` to print sort_word values.
We also make sure that the last item from a given file ends in a comma (",)",
due to the bogosity of the so-called JSON spec.
Usage:
```
bin/combine_author_handle.sh > combined_author_handle.json
```
**NOTE**: This tool assume that all JSON files have been formatted with the
`bin/jprint-wrapper.sh` tool. In particular the first line is just "{:,
and the last line is just "}" and each JSON element is on its own line.
**NOTE**: see the
FAQ on "[author_handle.json](../faq.html#author_handle_json)"
for more details on `author_handle.json` files.
### [csv2entry.sh](%%REPO_URL%%/bin/csv2entry.sh)
Convert CSV files into `.entry.json` for all winning IOCCC entries.
This tool takes as input, the following CSV files:
- [author_wins.csv](#author_wins_csv) - `author_handle` followed by all their
`entry_id`s
- [manifest.csv](#manifest_csv) - information about files under an entry
- [year_prize.csv](#year_prize_csv) - `entry_id` followed by the entry's award
title
This tool updates `.entry.json` files entries whose content is modified.
Usage:
```
bin/csv2entry.sh -v 1
```
There is no requirement to sort the CSV files (say through the spreadsheet
application prior to exporting), convert them to UNIX format, or append a final
newline to the file.
This tool will canonicalize the CSV files before using them
as input. Thus, if one wishes to import the CSV file into
some spreadsheet such as the [macOS](https://www.apple.com/macos)
[Numbers](https://www.apple.com/numbers/) spreadsheet application,
modify the content and finally export back to the CSV file,
this tool will modify the CSV file (if needed) in order
to restore the CSV order and other canonicalizing processes.
This tool will flag as an error, any empty fields, fields that are
an unquoted `NULL` or `null`, fields that start with whitespace,
fields that end with whitespace, or fields that contain consecutive
whitespace characters.
**NOTE**: see the
FAQ on "[.entry.json files](../faq.html#entry_json)"
for more details on `.entry.json` files.
#### Internal details of `bin/csv2entry.sh`
We first canonicalize the CSV files by replacing any "carriage
return line feeds" with "newlines". We also make sure that the CSV
files end in a newline. We do this because some spreadsheet
applications, when exporting to a CSV file, do not do this.
We also sort the CSV files in the same way that [entry2csv](#entry2csv) sorts
its CSV output files. We do this in case the CSV files were imported into a
spreadsheet where their order was changed before exporting. This means one is
free to order the CSV file content as you wish as this tool will reset these CSV
files.
Next this tool processes the non-CSV comment lines in
[manifest.csv](#manifest_csv). The 1st and 2nd fields of
[manifest.csv](#manifest_csv) refer to entry YYYY and entry subdirectory (i.e.,
the `YYYY/dir` directory under the root of the git repository). From that list of
`YYYY/dir` IOCCC entry directories, we will create the `.entry.json` files. We
only modify those `.entry.json` files when their content changes.
**IMPORTANT NOTE**: while this tool uses `jparse(1)` to verify that the modified
`.entry.json` contains valid JSON content, **this tool does NOT perform ANY
semantic checks**. For example, this tool does **NOT** verify that the manifest in
the `.entry.json` file matches the files in the `YYYY/dir directory`, or even
that the `.entry.json` contains a manifest (or any of the other required JSON
content). Another tool will be modified to do this, at a later date.
**NOTE**: you can obtain `jparse(1)` from the which can be obtained from the
[mkiocccentry repo](https://github.com/ioccc-src/mkiocccentry)'s copy of the
[jparse repo](https://github.com/xexyl/jparse/).
This tool is used by the [Judges](../judges.html) as part of the final
steps to announce a new set of winning IOCCC entries.
Given a submission (that has won the IOCCC and thus will become an entry), with a `.auth.json`
and `.info.json`, we convert the `.auth.json` and `.info.json` into a new `.entry.json` file.
**NOTE**: the `.auth.json` and `.info.json` files are formed by the
`mkiocccentry(1)` tool from the [mkiocccentry GitHub
repo](https://github.com/ioccc-src/mkiocccentry) as part of packaging
submissions.
We will also form new `author/author_handle.json` (see the
FAQ on "[author_handle.json files](../faq.html#author_json)"
for more information) and update `author/author_handle.json`
for [authors](../authors.html) who previously won. See also the
FAQ on "[author_handle](../faq.html#author_handle_faq)".
See also the
FAQ on "[.entry.json](../faq.html#entry_json)".
Usage:
```
bin/cvt-submission.sh -v 1 YYYY/dir
```
Here, `YYYY/dir` is the path of the submission under the `YYYY` year directory.
This tool will form a compressed tarball of any files under `YYYY/dir`
that could be modified or removed by this tool (unless `-N` is used in which
case this tool does nothing). The compressed tarball, formed by default
under the `/var/tmp` directory, may contain such as:
- `YYYY/dir/.info.json`
- `YYYY/dir/.auth.json`
- `YYYY/dir/remarks.md`
- `YYYY/dir/README.md`
- `YYYY/dir/index.md`
- `YYYY/dir/YYYY_dir.tar.bz2`
- `author/author_handle.json` (could be more than one file)
Only those files that exist will be put into the compressed tarball.
The compressed tarball formed (by default under the `/var/tmp` directory)
is of the following form:
> YYYY_dir.mods.YYYYMMDD.hhmmss.tar.bz2
Here "`YYYYMMDD.hhmmss`" is a date and timestamp from when the tool was executed.
The purpose of this compressed tarball is to allow the files in the
entry directory (that could be modified or removed by this tool)
to be restored as follows:
```
tar -jxvf /var/tmp/YYYY_dir.mods.YYYYMMDD.hhmmss.tar.bz2 YYYY/dir
```
One may also restore modified `author/author_handle.json` files using:
```
tar -jxvf /var/tmp/YYYY_dir.mods.YYYYMMDD.hhmmss.tar.bz2 author
```
This tool will form `YYYY/dir/README.md` if needed, from `YYYY/dir/remarks.md`,
`template/entry/README.md.head`, and `template/entry/README.md.tail`.
**NOTE**: This interactive tool (unless `-i input_data` is used) does
**NOT** perform all of the steps needed to make a directory for a new winning
IOCCC entry. For example, files such as `YYYY/dir/README.md` and/or `YYYY/dir/index.html`
might contain "_triple X_" comments, indicating where the [Judges](../judges.html)
need to add content. Moreover, the `Makefile` and `.gitignore` files
need to be examined for suitability, etc.
**HINT**: Executing this tool on your submission will **NOT** make you an IOCCC winner. :-)
### [entry2csv.sh](%%REPO_URL%%/bin/entry2csv.sh)
This tool takes as input, all entry `.entry.json` files
and updates 3 CSV files:
- [author_wins.csv](#author_wins_csv) - each `author_handle` followed by their `entry_id`s
- [manifest.csv](#manifest_csv) - information about all files in all entries
- [year_prize.csv](#year_prize_csv) - each `entry_id` followed by the entry's award title
The CSV files are written in a canonical UNIX format form.
Only those CSV files files whose content is modified are written.
#### Internal details of `bin/entry2csv.sh`
We generate CSV files from the `.entry.json` files from winning
IOCCC entries listed under years listed in the `.top` file,
and in subdirectories listed in the `YYYY/.year` file for the
given year. Only those entries so listed are processed.
All IOCCC entry directories must have a `.path` file that lists
the path of the entry's directory from the `TOPDIR`.
**IMPORTANT NOTE**: when adding new IOCCC winning entries, the `.top` file
**MUST** be updated, and the new IOCCC year `YYYY/.year` files **MUST**
reference the directory of the new IOCCC entries. Each entry directory **MUST**
also contain a `.path` file that contains the path of the IOCCC entry directory
from the `TOPDIR`.
Generate a list of files in an entry's manifest (the `.entry.json` file).
Usage:
```
awk -f bin/filelist.entry.json.awk 2020/ferguson1/.entry.json
```
In this case the command will list all the files of the
[2020/ferguson1](../2020/ferguson1/index.html) winning entry.
### [html.sed](%%REPO_URL%%/bin/html.sed)
Translates certain characters in their corresponding HTML entities. For example
`<` is converted to `<` and `>` is converted to `>`. This is important to
satisfy html lints.
This script is used in [md2html.sh](#md2html) via
[output-index-inventory.sh](#output-index-inventory) and
[subst.entry-index.sh](#subst-entry-index).
Usage:
```
cmd | sed -f bin/html.sed
sed -f bin/html.sed file > output
sed -i -f bin/html.sed file
```
Find all markdown links to local files that do not exist.
This tool does **NOT** check that a remote URL exists, it checks on
links to local files.
This tool does **NOT** check links in a given place in a file, it
checks that local files linked by markdown actually exist.
This tool does **NOT** check HTML file links, it checks markdown
based links.
Usage:
```
bin/find-missing-links.sh -v 1
```
If no missing links are found, this tool exits 0 with no output
(debug messages notwithstanding), otherwise this tool will exit non-zero.
**NOTE**: If the markdown link is malformed, this tool
might generate an error about a file that does exist.
If this tool claims that a file is missing that does exist,
look for a malformed markdown line and/or use of markdown
that is **NOT** an IOCCC markdown best practice.
See also the [IOCCC markdown best practices](../markdown.html) document for more
details.
We recommend that this tool be invoked via the top level `Makefile`:
```
make find_missing_links
```
This script uses the `jparse(1)` tool, which can be obtained from the
[mkiocccentry repo](https://github.com/ioccc-src/mkiocccentry)'s copy of the
[jparse repo](https://github.com/xexyl/jparse/), and searches for any invalid
JSON files in the tree.
This is important because the IOCCC makes extensive use of JSON.
Usage:
```
bin/find-invalid-json.sh -v 1
```
If no invalid JSON files are found, this tool exits 0 with no output
(debug messages notwithstanding), otherwise this tool will exit non-zero.
We recommend that this tool be invoked via the top level `Makefile`:
```
make find_invalid_json
```
Generate the top level [authors.html](../authors.html) page.
Usage:
```
bin/gen-authors.sh -v 1
```
We recommend that this tool be invoked via the top level `Makefile`:
```
make gen_authors
```
Generate the top level [location.html](../location.html) page.
Usage:
```
bin/gen-location.sh -v 1
```
We recommend that this tool be invoked via the top level `Makefile`:
```
make gen_location
```
Generate the HTML files (other than the README.md to index.html files) from
markdown files, of all entries.
Usage:
```
bin/gen-other-html.sh -v 1
```
We recommend that this tool be invoked via the top level `Makefile`:
```
make gen_other_html
```
Generate a `README.md` file a submission's `remarks.md` and the
template files in [template/entry/README.md.head](%%REPO_URL%%/template/entry/README.md.head)
and [template/entry/README.md.tail](%%REPO_URL%%/template/entry/README.md.tail).
Usage:
```
bin/gen-readme.sh -v 1 YYYY template/entry/README.md.head YYYY/dir/remarks.md template/entry/README.md.tail YYYY/dir/README.md
```
While this tool is used by [cvt-submission](#cvt-submission), it may also
be used by hand by the [Judges](../judges.html) during the final judging rounds.
Generate an XML sitemap for the IOCCC website.
Usage:
```
bin/gen-sitemap.sh -v 1
```
This would generate the [sitemap.xml](../sitemap.xml) file.
We recommend that this tool be invoked via the top level `Makefile`:
```
make gen_sitemap
```
Generate [status.json](../status.json) according to the modification date of
[news.md](%%REPO_URL%%/news.md) and/or whether the `contest_status` is changed
at the command line.
Without an argument, the `contest_status` is unchanged.
Usage:
```
bin/gen-status.sh -v 1
```
To force the `contest_status` to be closed:
```
bin/gen-status.sh -v 1 closed
```
To force the `contest_status` to be open:
```
bin/gen-status.sh -v 1 open
```
To see other valid statuses:
```
bin/gen-status -h
```
We recommend that this tool be invoked via the top level `Makefile`:
```
make gen_status
```
unless the `contest_status` is to be changed, but since only the judges should
do that, that is not a problem.
Generate a number of top level HTML pages for the IOCCC websites.
Usage:
```
bin/gen-top-html.sh -v 1
```
Examples of top level HTML pages built by this tool include:
- [CODE_OF_CONDUCT.html](../CODE_OF_CONDUCT.html)
- [archive/historic/index.html](../archive/historic/index.html)
- [bin/index.html](../bin/index.html)
- [bugs.html](../bugs.html)
- [contact.html](../contact.html)
- [faq.html](../faq.html)
- [inc/index.html](%%REPO_URL%%/inc/index.html)
- [index.html](../index.html)
- [judges.html](../judges.html)
- [news.html](../news.html)
- [thanks-for-help.html](../thanks-for-help.html)
We recommend that this tool be invoked via the top level `Makefile`:
```
make gen_top_html
```
Generate an `index.html` page for a given IOCCC year.
Usage:
```
bin/gen-year-index.sh -v 1 2020
```
This would create the [2020/index.html](../2020/index.html) file.
We recommend that this tool be invoked via the top level `Makefile`:
```
make gen_year_index
```
which will create the `index.html` for every IOCCC year (`1984/index.html`,
`1985/index.html` etc.).
### [gen-years.sh](%%REPO_URL%%/bin/gen-years.sh)
Generate the top level [years.html](../years.html) page.
Usage:
```
bin/gen-years.sh -v 1
```
We recommend that this tool be invoked via the top level `Makefile`:
```
make gen_years
```
The [jfmt-wrapper.sh](%%REPO_URL%%/bin/jfmt-wrapper.sh) tool is a wrapper
tool for `jfmt(1)`, a tool that will format a JSON file into a canonical style.
**NOTE**: As of 2024 Oct 08 the `jfmt(1)` tool has not been written,
so [jfmt-wrapper.sh](%%REPO_URL%%/bin/jfmt-wrapper.sh) uses the
`JSONPath.sh(1)` tool from the recently forked and modified
[JSONPath.sh tool](https://github.com/lcn2/JSONPath.sh).
Usage:
```
bin/jfmt-wrapper.sh .entry.json
```
The [jprint-wrapper.sh](%%REPO_URL%%/bin/jprint-wrapper.sh) tool is a wrapper
tool for `jprint(1)`, a tool that will format JSON in a canonical style
to stdout.
**NOTE**: `jprint(1)` is a planned tool based on `jfmt(1)`.
As of 2024 Oct 08 that tool has not been written, so
[jprint-wrapper.sh](%%REPO_URL%%/bin/jprint-wrapper.sh) uses
the `JSONPath.sh(1)` tool from the recently forked and modified
[JSONPath.sh tool](https://github.com/lcn2/JSONPath.sh).
Usage:
```
bin/jprint-wrapper.sh .entry.json
```
Or use in a pipe:
```
cat .info.json | bin/jprint-wrapper.sh
```
The [jval-wrapper.sh](%%REPO_URL%%/bin/jval-wrapper.sh) tool is a wrapper
tool for `jval(1)`, a tool that will output a JSON value using the
XPath for JSON mechanism. For information in XPath for JSON see:
- [https://goessner.net/articles/JsonPath/](https://goessner.net/articles/JsonPath/)
- [https://jsonpath.com](https://jsonpath.com)
- [https://github.com/lcn2/JSONPath.sh?tab=readme-ov-file#jsonpath-patterns-and-extensions](https://github.com/lcn2/JSONPath.sh?tab=readme-ov-file#jsonpath-patterns-and-extensions)
**NOTE**: As of 2024 Oct 08 the `jval(1)` tool has not been written,
so [jval-wrapper.sh](%%REPO_URL%%/bin/jval-wrapper.sh) uses the
`jsp(1)` tool from [https://github.com/kjozsa/jsp](https://github.com/kjozsa/jsp)
or if not found, the `JSONPath.sh(1)` tool from the recently forked and modified
[JSONPath.sh tool](https://github.com/lcn2/JSONPath.sh).
Usage:
```
bin/jval-wrapper.sh [-b] [-q] [-T] file.json [pattern]
-b print only values (def: print both JSON member and JSON value)
-q remove enclosing double quotes (def: keep any enclosing double quotes)
-T trim all newlines (def: do not)
file.json JSON file to process, - ==> read stdin
pattern Xpath for JSON query pattern
```
For example:
```
bin/jval-wrapper.sh 1984/mullender/.entry.json '$.award'
bin/jval-wrapper.sh 1984/mullender/.entry.json '$..author_handle'
bin/jval-wrapper.sh 1984/mullender/.entry.json '$.manifest[2]'
bin/jval-wrapper.sh -q 1984/mullender/.entry.json '$.manifest[2]'
bin/jval-wrapper.sh -b 1984/mullender/.entry.json '$.manifest[2].entry_text'
bin/jval-wrapper.sh -b -q 1984/mullender/.entry.json '$.manifest[2].inventory_order'
bin/jval-wrapper.sh author/Anton_Algmyr.json '$.full_name'
bin/jval-wrapper.sh -q -T author/Anton_Algmyr.json '$.full_name' | jstrdecode -
```
**NOTE**: With `jstrdecode(1)` version 1.2.3 or better, one can use `jstrdecode -N`
and not have to use `-T` with `bin/jval-wrapper.sh`.
Output manifest table from an entry's `.entry.json` file.
```
awk -v github=REPO_URL -f bin/manifest.entry.json.awk YYYY/dir/.entry.json
```
### [md2html.cfg](%%REPO_URL%%/bin/md2html.cfg)
This is a configuration file for the [md2html](index.html#md2html) tool.
Although a configuration file it is also something of a script file.
To not down descend into the depths of `pandoc(1)` and to have to explain how
this file can change all the HTML files, we advise you to let
[Smaug](https://www.glyphweb.com/arda/s/smaug.html) (the dragon of
[Erebor](https://www.glyphweb.com/arda/e/erebor.html) famously visited by a
[Hobbit](https://www.glyphweb.com/arda/h/hobbits.html) called [Bilbo
Baggins](https://www.glyphweb.com/arda/b/bilbobaggins.html) and 13
[Dwarves](https://www.glyphweb.com/arda/d/dwarves.html) in the
[T.A.](https://www.glyphweb.com/arda/t/thirdage.html) 2941 or 1341
in the [Shire-reckoning](https://www.glyphweb.com/arda/s/shirereckoning.html))), slumber as
"`here be dragons`". If you still need convincing, we wish to remind you of [Bilbo
Baggins](https://www.glyphweb.com/arda/b/bilbobaggins.html)' famous words:
> "Never laugh at live dragons,
[Bilbo](https://www.glyphweb.com/arda/b/bilbobaggins.html) you fool!" he said to himself, and it
became a favourite saying of his later, and passed into a proverb.
### [md2html.sh](%%REPO_URL%%/bin/md2html.sh)
This is the primary tool that forms IOCCC generated HTML content from
markdown files (permanent markdown files or temporarily generated
markdown files) and HTML fragments from the [inc directory](%%REPO_URL%%/inc/index.html).
The [md2html.cfg](index.html#md2html_cfg) configuration file is
used by [md2html.sh](%%REPO_URL%%/bin/md2html.sh) to drive the generation process.
### [new-dir.sh](%%REPO_URL%%/bin/new-dir.sh)
This tool is used by the [Judges](../judges.html) as one of the
steps needed to announce a new winning IOCCC entry.
Usage:
```
mkdir -p YYYY/dir
bin/new-dir.sh YYYY/dir
```
Here, `YYYY/dir` must be the path to the winning IOCCC entry.
**NOTE**: This tool does **NOT** perform all of the steps needed
to make a directory a new winning IOCCC entry. It only starts the
framework for doing so, creating a situation where the
[cvt-submission](#cvt-submission) tool be be run.
This tool can modify the top level `.allyear` file, and `YYYY/Makefile` files.
**NOTE**: This tool assumes that [new-year.sh](#new-year) was executed to create
the prerequisite `YYYY` directory and related files.
**HINT**: Executing this tool on your submission will **NOT** make you an IOCCC winner. :-)
### [new-year.sh](%%REPO_URL%%/bin/new-year.sh)
This tool is used by the [Judges](../judges.html) as part of the final
steps to announce a new set of winning IOCCC entries.
This tool may modify the top level `.top` and `Makefile` files.
It will form `YYYY/Makefile` and/or `YYYY/.year` if needed.
It will form `YYYY/README.md` if needed, from `template/README.md.year`.
**NOTE**: This tool does **NOT** perform all of the steps needed to make a new IOCCC year directory.
For example, files such as `YYYY/README.md` and/or `YYYY/index.html`
might contain "_triple X_" comments indicating where the
[Judges](../judges.html) need to add content.
Usage:
```
bin/new-year.sh -v 1 YYYY
```
Here, `YYYY` must be a new 4-digit (happy :-)) new IOCCC year.
Convert a non-README.md file in an entry to HTML.
Usage:
```
bin/othermd2html.sh YYYY/dir/pathto.md
```
For example:
```
bin/othermd2html.sh -v 1 2020/ferguson1/chocolate-cake.md
```
The [othermd2html.sh](#othermd2html) tool is used by
[gen-other-html.sh](#gen-other-html).
Output an author's or authors' related HTML details for an entry's index.html
page.
For an example, see the [author details in
1984/anonymous](../1984/anonymous/index.html#author).
This tool is used in the [bin/md2html.cfg](%%REPO_URL%%/bin/md2html.cfg) file as part of
the [md2html tool](index.html#md2html).
Output the inventory in HTML form for an entry's index.html page. For an example
inventory, see the [inventory in
1984/anonymous](../1984/anonymous/index.html#inventory).
This tool is used in the [bin/md2html.cfg](%%REPO_URL%%/bin/md2html.cfg) file as part of
the [md2html tool](index.html#md2html).
Output the inventory for a given year's winning entries in HTML form. In other
words in [1984](../1984/index.html) it would list, as links, the four winning
entries, which you can see directly at the [1984
inventory](../1984/index.html#inventory).
This tool is used in the [bin/md2html.cfg](%%REPO_URL%%/bin/md2html.cfg) file as part of
the [md2html tool](index.html#md2html).
Build an entry's `index.html` file if the entry directory
does not have a non-empty `index.hmtl` file, or if either
`.entry.json` or `README.md` is newer than the `index.hmtl` file.
This is useful when only a few entries have been
modified (resulting in an updated `.entry.json` file)
or if the `README.md` of a few entries have been changed: while the
[readme2index.sh](index.html#readme2index) script takes a few seconds
to run for a few entries, when applied to 300+ entries, the extra time can add
up.
If only a few `index.hmtl` files need updating, then
this command will only briefly pause while the
[readme2index.sh](index.html#readme2index) can take much longer.
Usage:
```
# For all entries:
bin/all-run.sh -v 3 bin/quick-readme2index.sh -v 1
# For an individual entry:
bin/quick-readme2index.sh -v 1 2020/ferguson2
```
**NOTE**: This command assumes that the relative
modification times for `README.md`, `.entry.json`,
and `index.html` are correct. If in doubt, use
[readme2index.sh](index.html#readme2index).
We recommend that this tool be invoked via the top level `Makefile`:
```
make quick_readme2index
```
Convert an entry's `README.md` into its `index.html` file.
Usage:
```
bin/readme2index.sh -v 3 1984/mullender
```
To build `index.html` files for all entries:
```
bin/all-run.sh -v 3 bin/readme2index.sh -v 1
```
We recommend that this tool be invoked via the top level `Makefile`:
```
make entry_index
```
### [sgi.sh](%%REPO_URL%%/bin/sgi.sh)
Sort `.gitignore` content from `stdin` to `stdout`.
We sort with lines starting with `#` first.
We sort with lines starting with `*` second.
We sort with lines that do not start with `[#!*]` third.
We sort with lines starting with `!` fourth.
This tool is used by [sort.gitignore.sh](index.html#sort-gitignore).
Sort a `.gitignore` in a entry directory.
Usage:
```
bin/sort.gitignore.sh -v 1 YYYY/dir
```
Suggested usage (for all `.gitignore` files):
```
bin/all-run.sh bin/sort.gitignore.sh -v 1
```
We recommend that this tool be invoked via the top level `Makefile`:
```
make sort_gitignore
```
Convert [status.json](../status.json) into HTML.
This tool is a 'before tool' (`-b tool`) that is intended
to be used by [gen-status.sh](index.html#gen-status).
Output substitutions for `navbar` on behalf of a year level `index.html`.
This tool is used in [subst.year-index.sh](index.html#subst-year-index).
## [tar-entry.sh](%%REPO_URL%%/bin/tar-entry.sh)
Form a compressed tarball for an entry.
Usage:
```
bin/tar-entry.sh -v 1 YYYY/dir
```
Suggested usage:
```
bin/all-run.sh -v 3 bin/tar-entry.sh -v 1 -W
```
## [tar-year.sh](%%REPO_URL%%/bin/tar-year.sh)
Form a compressed tarball for an IOCCC year.
Usage:
```
bin/tar-year.sh -v 1 YYYY
```
Suggested usage:
```
bin/all-years.sh -v 3 bin/tar-year.sh -v 1 -W
```
We recommend that this tool be invoked via the top level `Makefile`:
```
make form_year_tarball
```
Untar an entry's compressed tarball.
Usage:
```
bin/untar-entry.sh -v 1 YYYY/dir
```
Suggested usage:
```
bin/all-run.sh -v 3 bin/untar-entry.sh -v 1
```
We recommend that this tool be invoked via the top level `Makefile`:
```
make untar_entry_tarball
```
Untar an IOCCC year's compressed tarball.
Usage:
```
bin/untar-year.sh -v 1 YYYY
```
Suggested usage:
```
bin/all-years.sh -v 3 bin/untar-year.sh -v 1
```
We recommend that this tool be invoked via the top level `Makefile`:
```
make untar_year_tarball
```
# How IOCCC HTML content is built
The [md2html.sh](index.html#md2html) tool is the primary tool that
is used to form all IOCCC related HTML pages for the
[official IOCCC website](https://www.ioccc.org).
Nearly all IOCCC related HTML pages are built from markdown files,
from either permanent markdown files or temporary generated markdown files,
as well as HTML fragments from the [inc directory](%%REPO_URL%%/inc/index.html).
Most HTML content is built from permanent markdown files, such as a `README.md`
markdown file found in each entry directory. Some HTML content are generated
from temporary markdown files. These temporary markdown files are produced by
tools in the [bin directory](index.html) and exist only while the tool is running.
In addition to converting markdown to HTML, the canonical way that HTML content is
built uses, by default, files from the [inc directory](%%REPO_URL%%/inc/index.html), of
the file name form:
```
*.default.html
```
By using command line options of the form:
```
-H phase=name
```
where `phase` is the name of an HTML phase,
a non-default file may be used. For example:
```
-H navbar=up2index
```
will cause [inc/navbar.up2index.html](%%REPO_URL%%/inc/navbar.up2index.html)
instead of [inc/navbar.default.html](%%REPO_URL%%/inc/navbar.default.html) to be
used during the _navbar_ HTML phase.
If `name` is dot (i.e., `.`), then the given HTML phase is skipped.
For example:
```
-H footer=.
```
will cause no HTML content to be produced during the `footer` HTML phase.
## HTML phases
The following HTML phase files are used to build HTML content:
0. [inc/top.default.html](%%REPO_URL%%/inc/top.default.html)
1. [inc/head.default.html](%%REPO_URL%%/inc/head.default.html)
2. [inc/body.default.html](%%REPO_URL%%/inc/body.default.html)
3. [inc/topbar.default.html](%%REPO_URL%%/inc/topbar.default.html)
4. [inc/header.default.html](%%REPO_URL%%/inc/header.default.html)
5. [inc/navbar.default.html](%%REPO_URL%%/inc/navbar.default.html)
6. [inc/before-content.default.html](%%REPO_URL%%/inc/before-content.default.html)
Phases 7-19 are reserved for future use.
20. ((insert 'before tool' output))
21. ((insert 'pandoc wrapper tool' output))
22. ((insert 'after tool' output))
Phases 23-29 are reserved for future use.
30. [inc/after-content.default.html](%%REPO_URL%%/inc/after-content.default.html)
31. [inc/footer.default.html](%%REPO_URL%%/inc/footer.default.html)
32. [inc/bottom.default.html](%%REPO_URL%%/inc/bottom.default.html)
Phases 33-39 are reserved for future use.
In all of the above HTML phase numbers, symbols of the form `%%TOKEN%%` are
substituted.
Substitutions of the form `%%TOKEN%%` are performed on included HTML content,
content generated by the 'before tool' (`-b tool`),
content generated by the 'after tool' (`-a tool`),
as well as the markdown content given to the `pandoc wrapper tool` (`-p tool`).
It is an error, unless `-S` is given, for any phase, except phases HTML
20-29, to not substitute all `%%TOKEN%%`s. For phases HTML 20-29, any
`%%TOKEN%%` that is not substituted are passed thru without substitution.
See the tool [readme2index.sh](index.html#readme2index) for an example of
how HTML phases are implemented.
## getopt phase processing of the command line
The command options are evaluated in the following `getopt` phases:
### `getopt` phase 0
In `getopt` phase 0 we parse command line options and save all arguments for the
end of the final command line.
Depending on the program, the argument at the end is directory under topdir, or
it is a file under topdir.
`getopt` phase 0 is the only `getopt` phase where `-d topdir` and `-c
md2html.cfg` may be used.
### `getopt` phase 1
In `getopt` phase 1 we execute each `-o "output tool"` from `getopt` phase 0,
parsing the output as a command line.
The `-o "output tool"` may be given `optstr` from `"-O tool=optstr"`, followed
by the phase 0 argument.
The `-o "output tool"` is not allowed to output another `"-o tool"` or a `"-O
tool=optstr"`. Instead a `-o "output tool"` may execute another `-o "output
tool"` and merge the output into its own.
### `getopt` phase 2
In `getopt` phase 2 we parse the `cfg_options` from the first `md2html.cfg` line
matched by a saved argument.
The match is made with the (possibly modified) phase 0 argument.
### `getopt` phase 3
In `getopt` phase 3 we execute each `-o "output tool"` from `getopt` phase 2,
parsing the output as a command line.
The `-o "output tool"` may be given `optstr` from `"-O tool=optstr"`, followed
by the phase 0 argument.
The `-o "output tool"` is not allowed to output another `"-o tool"` or a `"-O
tool=optstr"`. Instead a `-o "output tool"` may execute another `-o "output
tool"` and merge the output into its own.
### Command line option order
Most command line options override earlier copies of the same option. However
in the case of `-H phase=name`, a later `-H phase=name` only overrides an
earlier use of `-H` for the same `phase` _only_. Also in the case of `-s
token=value`, a later `-s token=value` only overrides an earlier use of `-s` for
the same `token` _only_.
### Output tools
An "output tool" may be used to add certain additional command line options to be processed.
**IMPORTANT**: An "output tool" will print each command line option / argument on a separate line.
So for example, if an "output tool" wishes to convey:
```
-s TITLE='IOCCC entry locations' -H navbar=top-of-site
```
The "output tool" would output the following 4 lines:
```
-s
TITLE='IOCCC entry locations'
-H
navbar=top-of-site
```
The command line options printed by an "output tool" are processed
after all command line options, and all options from a matching
line from the [md2html.cfg](%%REPO_URL%%/bin/md2html.cfg) file, and before any filename arguments
on the command line.
For example:
```
command [command_options ..] [cfg_options ..] [output_tool_options ..] [--] [filename_arg ..]
```
Here the `command` first processes any command line options, then it processes
`cfg_options` obtained from this file from a first match of a given `file_glob`,
then it processes `output tool` (i.e., `-o tool`) options found in
`command_options`, and `cfg_options`, then an optional `--` (end of all
options), then zero of more `filename_arg` filename arguments.
The `-o tool` must **NOT** output any `-o tool` options as `-o tool` is **NOT
RECURSIVE**. A `-o tool` is free, however, to execute other `-o tool` output
tools and merge the output from those tools into its own.
### Special `-E exitcode` option
When `-E exitcode` is evaluated, the application will exit with the `exitcode`
value. If one wishes to also output a message to `stderr`, the `-e string` must
come **BEFORE** any `-E exitcode` in the command line.
### Substitution tokens
All tokens (i.e., strings of the form `%%token%%`) **MUST** be substituted (by
some `-s token=value`) in all HTML output, except during phase numbers 20-29
(i.e., except `before tool` output, pandoc wrapper output, and `after tool`
output), or the command will exit non-zero, unless `-S` is given. If `-S` is
given, only a warning about non-substituted tokens will be written to `stderr`.
The command line of tools in the [bin directory](index.html), and perhaps
modified via the [md2html config file](%%REPO_URL%%/bin/md2html.cfg) may change to using a
different filename for a given phase.
For example when forming the HTML from
[2020/ferguson1/chocolate-cake.md](%%REPO_URL%%/2020/ferguson1/chocolate-cake.md),
a different `navbar` navigation bar is needed. So instead of the
usual top navigation bar that normally directs people to the previous
entry for the year, or go up to the year page, or to the next entry
for the year, a top navigation bar to just go up to the entry's
main page is needed. A line in the [md2html config file](%%REPO_URL%%/bin/md2html.cfg)
that refers to
[2020/ferguson1/chocolate-cake.md](%%REPO_URL%%/2020/ferguson1/chocolate-cake.md) may
specify use of `navbar.up2index.html` (as `navbar.up2index`)instead of using the
`navbar.default.html` (`navbar.default.html`) default.
The HTML phase may be skipped resulting in no HTML output during a given phase
and furthermore, forming no HTML content from a given markdown file altogether.
See comments in the [md2html config file](%%REPO_URL%%/bin/md2html.cfg) for details.
See also, the tool [readme2index.sh](index.html#readme2index) for an example of
how such command lines are used.
# Use CAUTION when modifying `inc/` files
Some of the files under this directory are used to form **MOST** of the HTML content
on the [official IOCCC website](https://www.ioccc.org).
... and in particular files under [inc](%%REPO_URL%%/inc/index.html) that are of the form
(called default HTML files) ...:
```
*.default.html
```
... contain default content used to form IOCCC HTML / IOCCC web pages.
Instead of editing the default HTML files in order to fix a special web page,
consider making a copy of the default file and modifying the [md2html config
file](%%REPO_URL%%/bin/md2html.cfg) to refer to the copy instead. That way your special case
situation will not impact **MOST** of the HTML content.
# Why we do not use certain well-known HTML technologies
You may wonder why we need these files in the first place. You may wonder why
we even need the tools in the [bin](index.html) directory when there other
solutions available to form web pages.
## Static web pages only
Here are some reasons why we are using these files and
special tools to create HTML content / IOCCC web pages:
We host the [official IOCCC website](https://www.ioccc.org) via
[GitHub pages](https://pages.github.com). As of the time this file written, **only
static web pages are supported**.
## We cannot use server side include
We use [static web pages](#static-only), so use of "server side include" is not
available to the IOCCC.
For example, Apache SSI `#include` does not work on [GitHub
pages](https://pages.github.com).
## We cannot use a back-end database
We use [static web pages](index.html#static-only), so use of a "back-end database" is not
available to the IOCCC.
## We cannot use non-GitHub web servers
The [GitHub pages](https://pages.github.com) have the distributed server capacity needed to handle
the **huge download volume** or [Slashdot effect](https://en.wikipedia.org/wiki/Slashdot_effect)
that happens when new IOCCC winning entries are published.
Most so-called low cost web hosting sites have a somewhat hidden excessive bandwidth charge
and/or cap bandwidth when the volume gets too high, and/or do not have the infrastructure
that can handle the [Slashdot effect](https://en.wikipedia.org/wiki/Slashdot_effect).
The IOCCC had experience with people offering web mirrors and even free hosting
only to find that the generous benefactor moves on, or their service is purchased
by someone else who is not so generous. While it is possible that GitHub might someday
suffer a similar fate, for the time being we are betting that GitHub will remain
willing to host the IOCCC.
The [official IOCCC website](https://www.ioccc.org) is, after all, primarily C
source code with some supporting documentation (sometimes :-) ). As such it is
a natural fit for GitHub and [GitHub pages](https://pages.github.com).
## We cannot use the HTML `