# bin > "After 50+ years, the UNIX command line remains very powerful." -- Landon Noll > > An answer to (Monty)? Pythonesque question of why these tools > weren't written in a certain language. :-) The [bin directory](index.html) holds tools that build files, such as HTML content, for the [official IOCCC website](https://www.ioccc.org). For HTML content, the [bin directory](index.html) tools make use of HTML fragments from the [inc directory](%%REPO_URL%%/inc/index.html) as well as various JSON files and other content from the [official IOCCC winner repo](https://github.com/ioccc-src/winner). ## [bin/](index.html) tools options With the exception of the `awk(1)` and `sed(1)` scripts, all the tools in this directory support a number of options that can be used to get help, diagnose problems, see progress etc. These options are described below. ### Get help / usage string of a tool If you need to remember the syntax of the tool or get certain notes about different options, you can use the `-h` option. For instance if you want help on the [all-run.sh](index.html#all-run) tool from the root directory (of the repo/website), you would do: ``` bin/all-run.sh -h ``` ### Set verbosity level of a tool If you want verbosity, say for debugging purposes or to see what is going on more (than the default), you should use the `-v level` option. For instance if you wish to see what is going on with the [quick-readme2index.sh](index.html#quick-readme2index) tool, you might do: ``` bin/quick-readme2index.sh -v 3 ``` to set the verbosity level to `3`. The default for verbosity is `0`, no verbosity, though using the [top level Makefile](%%REPO_URL%%/Makefile) will, for some tools, set a verbosity level. ### Get version string of a tool If you just want to know what version the tool is, you can use the `-V` option. For instance to see what version the [chk-entry.sh](index.html#chk-entry) tool is, you would do: ``` bin/chk-entry.sh -V ``` ### Other notes about some of the common options These options, and especially `-h` and `-v level`, can be very useful to get basic usage information and to see what is going on when the tool is running. For more details on each tool, including the ones mentioned above, see below. As you go through each tool, if you need to understand more of it, we recommend that you use the `-h` option on it first. There are some scripts that are invoked by the [bin/md2html.cfg](%%REPO_URL%%/bin/md2html.cfg) file but some of these tools can be directly invoked as well, should you wish to see their output or if you have some odd need to do so.
### [all-jfmt.sh](%%REPO_URL%%/bin/all-jfmt.sh)
Canonically format all entry and author JSON files Usage: ``` bin/all-jfmt.sh -v 3 ``` Alternate usage: ``` make all_jfmt ```
### [all-years.sh](%%REPO_URL%%/bin/all-years.sh)
Run a command on all IOCCC years. Usage: ``` bin/all-years.sh -v 1 bin/gen-year-index.sh -v 1 ``` If you wish to run instead, for example, [chk-entry.sh](#chk-entry), then do: ``` bin/all-years.sh -v 1 bin/chk-entry.sh ```
### [all-run.sh](%%REPO_URL%%/bin/all-run.sh)
Run a command on all winning entries. Usage: ``` bin/all-run.sh -v 3 bin/quick-readme2index.sh -v 1 ``` If you wish to run instead, for example, [readme2index.sh](#readme2index), then do: ``` bin/all-run.sh -v 3 bin/readme2index.sh -v 1 ```
### [chk-entry.sh](%%REPO_URL%%/bin/chk-entry.sh)
Check an entry directory to verify that all the files in its manifest (`.entry.json`) exist in the git repo. Usage: ``` bin/chk-entry.sh 2020/ferguson1 ``` If you wish to run it on all entries, we recommend that this tool be invoked via the top level `Makefile`: ``` make verify_entry_files ``` **NOTE**: see the FAQ on "[.entry.json files](../faq.html#entry_json)" for more details on `.entry.json` files.
### [combine_author_handle.sh](%%REPO_URL%%/bin/combine_author_handle.sh)
Combine all author/author_handle.json files as single JSON file. The purpose of this tool is to make looking for information across all authors faster by temporarily forming them into a single JSON file. Because `jsp(1)` open remembers the last copy of a given JSON member name. As such, we change the JSON member name "winning_entry_set" into a unique "winning_entry_set.FILENO" where FILENO is the file number. This the JSON member value will be preserved across all files. We also convert where FILENO is the file number and FILENAME is the filename: ``` "sort_word" : "data", ``` into: ``` "sort_word.FILENO" : [ { "sort_word" : "data FILENAME" } ], ``` This will allow `jsp(1)` to print sort_word values. We also make sure that the last item from a given file ends in a comma (",)", due to the bogosity of the so-called JSON spec. Usage: ``` bin/combine_author_handle.sh > combined_author_handle.json ``` **NOTE**: This tool assume that all JSON files have been formatted with the `bin/jprint-wrapper.sh` tool. In particular the first line is just "{:, and the last line is just "}" and each JSON element is on its own line. **NOTE**: see the FAQ on "[author_handle.json](../faq.html#author_handle_json)" for more details on `author_handle.json` files.
### [csv2entry.sh](%%REPO_URL%%/bin/csv2entry.sh)
Convert CSV files into `.entry.json` for all winning IOCCC entries. This tool takes as input, the following CSV files: - [author_wins.csv](#author_wins_csv) - `author_handle` followed by all their `entry_id`s - [manifest.csv](#manifest_csv) - information about files under an entry - [year_prize.csv](#year_prize_csv) - `entry_id` followed by the entry's award title This tool updates `.entry.json` files entries whose content is modified. Usage: ``` bin/csv2entry.sh -v 1 ``` There is no requirement to sort the CSV files (say through the spreadsheet application prior to exporting), convert them to UNIX format, or append a final newline to the file. This tool will canonicalize the CSV files before using them as input. Thus, if one wishes to import the CSV file into some spreadsheet such as the [macOS](https://www.apple.com/macos) [Numbers](https://www.apple.com/numbers/) spreadsheet application, modify the content and finally export back to the CSV file, this tool will modify the CSV file (if needed) in order to restore the CSV order and other canonicalizing processes. This tool will flag as an error, any empty fields, fields that are an unquoted `NULL` or `null`, fields that start with whitespace, fields that end with whitespace, or fields that contain consecutive whitespace characters. **NOTE**: see the FAQ on "[.entry.json files](../faq.html#entry_json)" for more details on `.entry.json` files. #### Internal details of `bin/csv2entry.sh` We first canonicalize the CSV files by replacing any "carriage return line feeds" with "newlines". We also make sure that the CSV files end in a newline. We do this because some spreadsheet applications, when exporting to a CSV file, do not do this. We also sort the CSV files in the same way that [entry2csv](#entry2csv) sorts its CSV output files. We do this in case the CSV files were imported into a spreadsheet where their order was changed before exporting. This means one is free to order the CSV file content as you wish as this tool will reset these CSV files. Next this tool processes the non-CSV comment lines in [manifest.csv](#manifest_csv). The 1st and 2nd fields of [manifest.csv](#manifest_csv) refer to entry YYYY and entry subdirectory (i.e., the `YYYY/dir` directory under the root of the git repository). From that list of `YYYY/dir` IOCCC entry directories, we will create the `.entry.json` files. We only modify those `.entry.json` files when their content changes. **IMPORTANT NOTE**: while this tool uses `jparse(1)` to verify that the modified `.entry.json` contains valid JSON content, **this tool does NOT perform ANY semantic checks**. For example, this tool does **NOT** verify that the manifest in the `.entry.json` file matches the files in the `YYYY/dir directory`, or even that the `.entry.json` contains a manifest (or any of the other required JSON content). Another tool will be modified to do this, at a later date. **NOTE**: you can obtain `jparse(1)` from the which can be obtained from the [mkiocccentry repo](https://github.com/ioccc-src/mkiocccentry)'s copy of the [jparse repo](https://github.com/xexyl/jparse/).
### [cvt-submission.sh](%%REPO_URL%%/bin/cvt-submission.sh)
This tool is used by the [Judges](../judges.html) as part of the final steps to announce a new set of winning IOCCC entries. Given a submission (that has won the IOCCC and thus will become an entry), with a `.auth.json` and `.info.json`, we convert the `.auth.json` and `.info.json` into a new `.entry.json` file. **NOTE**: the `.auth.json` and `.info.json` files are formed by the `mkiocccentry(1)` tool from the [mkiocccentry GitHub repo](https://github.com/ioccc-src/mkiocccentry) as part of packaging submissions. We will also form new `author/author_handle.json` (see the FAQ on "[author_handle.json files](../faq.html#author_json)" for more information) and update `author/author_handle.json` for [authors](../authors.html) who previously won. See also the FAQ on "[author_handle](../faq.html#author_handle_faq)". See also the FAQ on "[.entry.json](../faq.html#entry_json)". Usage: ``` bin/cvt-submission.sh -v 1 YYYY/dir ``` Here, `YYYY/dir` is the path of the submission under the `YYYY` year directory. This tool will form a compressed tarball of any files under `YYYY/dir` that could be modified or removed by this tool (unless `-N` is used in which case this tool does nothing). The compressed tarball, formed by default under the `/var/tmp` directory, may contain such as: - `YYYY/dir/.info.json` - `YYYY/dir/.auth.json` - `YYYY/dir/remarks.md` - `YYYY/dir/README.md` - `YYYY/dir/index.md` - `YYYY/dir/YYYY_dir.tar.bz2` - `author/author_handle.json` (could be more than one file) Only those files that exist will be put into the compressed tarball. The compressed tarball formed (by default under the `/var/tmp` directory) is of the following form: > YYYY_dir.mods.YYYYMMDD.hhmmss.tar.bz2 Here "`YYYYMMDD.hhmmss`" is a date and timestamp from when the tool was executed. The purpose of this compressed tarball is to allow the files in the entry directory (that could be modified or removed by this tool) to be restored as follows: ``` tar -jxvf /var/tmp/YYYY_dir.mods.YYYYMMDD.hhmmss.tar.bz2 YYYY/dir ``` One may also restore modified `author/author_handle.json` files using: ``` tar -jxvf /var/tmp/YYYY_dir.mods.YYYYMMDD.hhmmss.tar.bz2 author ``` This tool will form `YYYY/dir/README.md` if needed, from `YYYY/dir/remarks.md`, `template/entry/README.md.head`, and `template/entry/README.md.tail`. **NOTE**: This interactive tool (unless `-i input_data` is used) does **NOT** perform all of the steps needed to make a directory for a new winning IOCCC entry. For example, files such as `YYYY/dir/README.md` and/or `YYYY/dir/index.html` might contain "_triple X_" comments, indicating where the [Judges](../judges.html) need to add content. Moreover, the `Makefile` and `.gitignore` files need to be examined for suitability, etc. **HINT**: Executing this tool on your submission will **NOT** make you an IOCCC winner. :-)
### [entry2csv.sh](%%REPO_URL%%/bin/entry2csv.sh)
This tool takes as input, all entry `.entry.json` files and updates 3 CSV files: - [author_wins.csv](#author_wins_csv) - each `author_handle` followed by their `entry_id`s - [manifest.csv](#manifest_csv) - information about all files in all entries - [year_prize.csv](#year_prize_csv) - each `entry_id` followed by the entry's award title The CSV files are written in a canonical UNIX format form. Only those CSV files files whose content is modified are written. #### Internal details of `bin/entry2csv.sh` We generate CSV files from the `.entry.json` files from winning IOCCC entries listed under years listed in the `.top` file, and in subdirectories listed in the `YYYY/.year` file for the given year. Only those entries so listed are processed. All IOCCC entry directories must have a `.path` file that lists the path of the entry's directory from the `TOPDIR`. **IMPORTANT NOTE**: when adding new IOCCC winning entries, the `.top` file **MUST** be updated, and the new IOCCC year `YYYY/.year` files **MUST** reference the directory of the new IOCCC entries. Each entry directory **MUST** also contain a `.path` file that contains the path of the IOCCC entry directory from the `TOPDIR`.
### [filelist.entry.json.awk](%%REPO_URL%%/bin/filelist.entry.json.awk)
Generate a list of files in an entry's manifest (the `.entry.json` file). Usage: ``` awk -f bin/filelist.entry.json.awk 2020/ferguson1/.entry.json ``` In this case the command will list all the files of the [2020/ferguson1](../2020/ferguson1/index.html) winning entry.
### [html.sed](%%REPO_URL%%/bin/html.sed)
Translates certain characters in their corresponding HTML entities. For example `<` is converted to `<` and `>` is converted to `>`. This is important to satisfy html lints. This script is used in [md2html.sh](#md2html) via [output-index-inventory.sh](#output-index-inventory) and [subst.entry-index.sh](#subst-entry-index). Usage: ``` cmd | sed -f bin/html.sed sed -f bin/html.sed file > output sed -i -f bin/html.sed file ``` Find all markdown links to local files that do not exist. This tool does **NOT** check that a remote URL exists, it checks on links to local files. This tool does **NOT** check links in a given place in a file, it checks that local files linked by markdown actually exist. This tool does **NOT** check HTML file links, it checks markdown based links. Usage: ``` bin/find-missing-links.sh -v 1 ``` If no missing links are found, this tool exits 0 with no output (debug messages notwithstanding), otherwise this tool will exit non-zero. **NOTE**: If the markdown link is malformed, this tool might generate an error about a file that does exist. If this tool claims that a file is missing that does exist, look for a malformed markdown line and/or use of markdown that is **NOT** an IOCCC markdown best practice. See also the [IOCCC markdown best practices](../markdown.html) document for more details. We recommend that this tool be invoked via the top level `Makefile`: ``` make find_missing_links ```
### [find-invalid-json.sh](%%REPO_URL%%/bin/find-invalid-json.sh)
This script uses the `jparse(1)` tool, which can be obtained from the [mkiocccentry repo](https://github.com/ioccc-src/mkiocccentry)'s copy of the [jparse repo](https://github.com/xexyl/jparse/), and searches for any invalid JSON files in the tree. This is important because the IOCCC makes extensive use of JSON. Usage: ``` bin/find-invalid-json.sh -v 1 ``` If no invalid JSON files are found, this tool exits 0 with no output (debug messages notwithstanding), otherwise this tool will exit non-zero. We recommend that this tool be invoked via the top level `Makefile`: ``` make find_invalid_json ```
### [format-headers.sh](%%REPO_URL%%/bin/format-headers.sh)
Format select headers and preceding empty lines in entry README.md files. Usage: ``` bin/format-headers.sh ```
### [gen-authors.sh](%%REPO_URL%%/bin/gen-authors.sh)
Generate the top level [authors.html](../authors.html) page. Usage: ``` bin/gen-authors.sh -v 1 ``` We recommend that this tool be invoked via the top level `Makefile`: ``` make gen_authors ```
### [gen-location.sh](%%REPO_URL%%/bin/gen-location.sh)
Generate the top level [location.html](../location.html) page. Usage: ``` bin/gen-location.sh -v 1 ``` We recommend that this tool be invoked via the top level `Makefile`: ``` make gen_location ```
### [gen-other-html.sh](%%REPO_URL%%/bin/gen-other-html.sh)
Generate the HTML files (other than the README.md to index.html files) from markdown files, of all entries. Usage: ``` bin/gen-other-html.sh -v 1 ``` We recommend that this tool be invoked via the top level `Makefile`: ``` make gen_other_html ```
### [gen-readme.sh](%%REPO_URL%%/bin/gen-readme.sh)
Generate a `README.md` file a submission's `remarks.md` and the template files in [template/entry/README.md.head](%%REPO_URL%%/template/entry/README.md.head) and [template/entry/README.md.tail](%%REPO_URL%%/template/entry/README.md.tail). Usage: ``` bin/gen-readme.sh -v 1 YYYY template/entry/README.md.head YYYY/dir/remarks.md template/entry/README.md.tail YYYY/dir/README.md ``` While this tool is used by [cvt-submission](#cvt-submission), it may also be used by hand by the [Judges](../judges.html) during the final judging rounds.
### [gen-sitemap.sh](%%REPO_URL%%/bin/gen-sitemap.sh)
Generate an XML sitemap for the IOCCC website. Usage: ``` bin/gen-sitemap.sh -v 1 ``` This would generate the [sitemap.xml](../sitemap.xml) file. We recommend that this tool be invoked via the top level `Makefile`: ``` make gen_sitemap ```
### [gen-status.sh](%%REPO_URL%%/bin/gen-status.sh)
Generate [status.json](../status.json) according to the modification date of [news.md](%%REPO_URL%%/news.md) and/or whether the `contest_status` is changed at the command line. Without an argument, the `contest_status` is unchanged. Usage: ``` bin/gen-status.sh -v 1 ``` To force the `contest_status` to be closed: ``` bin/gen-status.sh -v 1 closed ``` To force the `contest_status` to be open: ``` bin/gen-status.sh -v 1 open ``` To see other valid statuses: ``` bin/gen-status -h ``` We recommend that this tool be invoked via the top level `Makefile`: ``` make gen_status ``` unless the `contest_status` is to be changed, but since only the judges should do that, that is not a problem.
### [gen-top-html.sh](%%REPO_URL%%/bin/gen-top-html.sh)
Generate a number of top level HTML pages for the IOCCC websites. Usage: ``` bin/gen-top-html.sh -v 1 ``` Examples of top level HTML pages built by this tool include: - [CODE_OF_CONDUCT.html](../CODE_OF_CONDUCT.html) - [archive/historic/index.html](../archive/historic/index.html) - [bin/index.html](../bin/index.html) - [bugs.html](../bugs.html) - [contact.html](../contact.html) - [faq.html](../faq.html) - [inc/index.html](%%REPO_URL%%/inc/index.html) - [index.html](../index.html) - [judges.html](../judges.html) - [news.html](../news.html) - [thanks-for-help.html](../thanks-for-help.html) We recommend that this tool be invoked via the top level `Makefile`: ``` make gen_top_html ```
### [gen-year-index.sh](%%REPO_URL%%/bin/gen-year-index.sh)
Generate an `index.html` page for a given IOCCC year. Usage: ``` bin/gen-year-index.sh -v 1 2020 ``` This would create the [2020/index.html](../2020/index.html) file. We recommend that this tool be invoked via the top level `Makefile`: ``` make gen_year_index ``` which will create the `index.html` for every IOCCC year (`1984/index.html`, `1985/index.html` etc.).
### [gen-years.sh](%%REPO_URL%%/bin/gen-years.sh)
Generate the top level [years.html](../years.html) page. Usage: ``` bin/gen-years.sh -v 1 ``` We recommend that this tool be invoked via the top level `Makefile`: ``` make gen_years ```
### [jfmt-wrapper.sh](%%REPO_URL%%/bin/jfmt-wrapper.sh)
The [jfmt-wrapper.sh](%%REPO_URL%%/bin/jfmt-wrapper.sh) tool is a wrapper tool for `jfmt(1)`, a tool that will format a JSON file into a canonical style. **NOTE**: As of 2024 Oct 08 the `jfmt(1)` tool has not been written, so [jfmt-wrapper.sh](%%REPO_URL%%/bin/jfmt-wrapper.sh) uses the `JSONPath.sh(1)` tool from the recently forked and modified [JSONPath.sh tool](https://github.com/lcn2/JSONPath.sh). Usage: ``` bin/jfmt-wrapper.sh .entry.json ```
### [jprint-wrapper.sh](%%REPO_URL%%/bin/jprint-wrapper.sh)
The [jprint-wrapper.sh](%%REPO_URL%%/bin/jprint-wrapper.sh) tool is a wrapper tool for `jprint(1)`, a tool that will format JSON in a canonical style to stdout. **NOTE**: `jprint(1)` is a planned tool based on `jfmt(1)`. As of 2024 Oct 08 that tool has not been written, so [jprint-wrapper.sh](%%REPO_URL%%/bin/jprint-wrapper.sh) uses the `JSONPath.sh(1)` tool from the recently forked and modified [JSONPath.sh tool](https://github.com/lcn2/JSONPath.sh). Usage: ``` bin/jprint-wrapper.sh .entry.json ``` Or use in a pipe: ``` cat .info.json | bin/jprint-wrapper.sh ```
### [jval-wrapper.sh](%%REPO_URL%%/bin/jval-wrapper.sh)
The [jval-wrapper.sh](%%REPO_URL%%/bin/jval-wrapper.sh) tool is a wrapper tool for `jval(1)`, a tool that will output a JSON value using the XPath for JSON mechanism. For information in XPath for JSON see: - [https://goessner.net/articles/JsonPath/](https://goessner.net/articles/JsonPath/) - [https://jsonpath.com](https://jsonpath.com) - [https://github.com/lcn2/JSONPath.sh?tab=readme-ov-file#jsonpath-patterns-and-extensions](https://github.com/lcn2/JSONPath.sh?tab=readme-ov-file#jsonpath-patterns-and-extensions) **NOTE**: As of 2024 Oct 08 the `jval(1)` tool has not been written, so [jval-wrapper.sh](%%REPO_URL%%/bin/jval-wrapper.sh) uses the `jsp(1)` tool from [https://github.com/kjozsa/jsp](https://github.com/kjozsa/jsp) or if not found, the `JSONPath.sh(1)` tool from the recently forked and modified [JSONPath.sh tool](https://github.com/lcn2/JSONPath.sh). Usage: ``` bin/jval-wrapper.sh [-b] [-q] [-T] file.json [pattern] -b print only values (def: print both JSON member and JSON value) -q remove enclosing double quotes (def: keep any enclosing double quotes) -T trim all newlines (def: do not) file.json JSON file to process, - ==> read stdin pattern Xpath for JSON query pattern ``` For example: ``` bin/jval-wrapper.sh 1984/mullender/.entry.json '$.award' bin/jval-wrapper.sh 1984/mullender/.entry.json '$..author_handle' bin/jval-wrapper.sh 1984/mullender/.entry.json '$.manifest[2]' bin/jval-wrapper.sh -q 1984/mullender/.entry.json '$.manifest[2]' bin/jval-wrapper.sh -b 1984/mullender/.entry.json '$.manifest[2].entry_text' bin/jval-wrapper.sh -b -q 1984/mullender/.entry.json '$.manifest[2].inventory_order' bin/jval-wrapper.sh author/Anton_Algmyr.json '$.full_name' bin/jval-wrapper.sh -q -T author/Anton_Algmyr.json '$.full_name' | jstrdecode - ``` **NOTE**: With `jstrdecode(1)` version 1.2.3 or better, one can use `jstrdecode -N` and not have to use `-T` with `bin/jval-wrapper.sh`.
### [manifest.csv.entry.awk](%%REPO_URL%%/bin/manifest.csv.entry.awk)
Output `manifest.csv` from a entry's manifest as found in its `.entry.json` file. ``` awk -f bin/manifest.csv.entry.awk YYYY/dir/.entry.json ```
### [bin/manifest.entry.json.awk](%%REPO_URL%%/bin/manifest.entry.json.awk)
Output manifest table from an entry's `.entry.json` file. ``` awk -v github=REPO_URL -f bin/manifest.entry.json.awk YYYY/dir/.entry.json ```
### [md2html.cfg](%%REPO_URL%%/bin/md2html.cfg)
This is a configuration file for the [md2html](index.html#md2html) tool. Although a configuration file it is also something of a script file. To not down descend into the depths of `pandoc(1)` and to have to explain how this file can change all the HTML files, we advise you to let [Smaug](https://www.glyphweb.com/arda/s/smaug.html) (the dragon of [Erebor](https://www.glyphweb.com/arda/e/erebor.html) famously visited by a [Hobbit](https://www.glyphweb.com/arda/h/hobbits.html) called [Bilbo Baggins](https://www.glyphweb.com/arda/b/bilbobaggins.html) and 13 [Dwarves](https://www.glyphweb.com/arda/d/dwarves.html) in the [T.A.](https://www.glyphweb.com/arda/t/thirdage.html) 2941 or 1341 in the [Shire-reckoning](https://www.glyphweb.com/arda/s/shirereckoning.html))), slumber as "`here be dragons`". If you still need convincing, we wish to remind you of [Bilbo Baggins](https://www.glyphweb.com/arda/b/bilbobaggins.html)' famous words: > "Never laugh at live dragons, [Bilbo](https://www.glyphweb.com/arda/b/bilbobaggins.html) you fool!" he said to himself, and it became a favourite saying of his later, and passed into a proverb.
### [md2html.sh](%%REPO_URL%%/bin/md2html.sh)
This is the primary tool that forms IOCCC generated HTML content from markdown files (permanent markdown files or temporarily generated markdown files) and HTML fragments from the [inc directory](%%REPO_URL%%/inc/index.html). The [md2html.cfg](index.html#md2html_cfg) configuration file is used by [md2html.sh](%%REPO_URL%%/bin/md2html.sh) to drive the generation process.
### [new-dir.sh](%%REPO_URL%%/bin/new-dir.sh)
This tool is used by the [Judges](../judges.html) as one of the steps needed to announce a new winning IOCCC entry. Usage: ``` mkdir -p YYYY/dir bin/new-dir.sh YYYY/dir ``` Here, `YYYY/dir` must be the path to the winning IOCCC entry. **NOTE**: This tool does **NOT** perform all of the steps needed to make a directory a new winning IOCCC entry. It only starts the framework for doing so, creating a situation where the [cvt-submission](#cvt-submission) tool be be run. This tool can modify the top level `.allyear` file, and `YYYY/Makefile` files. **NOTE**: This tool assumes that [new-year.sh](#new-year) was executed to create the prerequisite `YYYY` directory and related files. **HINT**: Executing this tool on your submission will **NOT** make you an IOCCC winner. :-)
### [new-year.sh](%%REPO_URL%%/bin/new-year.sh)
This tool is used by the [Judges](../judges.html) as part of the final steps to announce a new set of winning IOCCC entries. This tool may modify the top level `.top` and `Makefile` files. It will form `YYYY/Makefile` and/or `YYYY/.year` if needed. It will form `YYYY/README.md` if needed, from `template/README.md.year`. **NOTE**: This tool does **NOT** perform all of the steps needed to make a new IOCCC year directory. For example, files such as `YYYY/README.md` and/or `YYYY/index.html` might contain "_triple X_" comments indicating where the [Judges](../judges.html) need to add content. Usage: ``` bin/new-year.sh -v 1 YYYY ``` Here, `YYYY` must be a new 4-digit (happy :-)) new IOCCC year.
### [othermd2html.sh](%%REPO_URL%%/bin/othermd2html.sh)
Convert a non-README.md file in an entry to HTML. Usage: ``` bin/othermd2html.sh YYYY/dir/pathto.md ``` For example: ``` bin/othermd2html.sh -v 1 2020/ferguson1/chocolate-cake.md ``` The [othermd2html.sh](#othermd2html) tool is used by [gen-other-html.sh](#gen-other-html).
### [output-index-author.sh](%%REPO_URL%%/bin/output-index-author.sh)
Output an author's or authors' related HTML details for an entry's index.html page. For an example, see the [author details in 1984/anonymous](../1984/anonymous/index.html#author). This tool is used in the [bin/md2html.cfg](%%REPO_URL%%/bin/md2html.cfg) file as part of the [md2html tool](index.html#md2html).
### [output-index-inventory.sh](%%REPO_URL%%/bin/output-index-inventory.sh)
Output the inventory in HTML form for an entry's index.html page. For an example inventory, see the [inventory in 1984/anonymous](../1984/anonymous/index.html#inventory). This tool is used in the [bin/md2html.cfg](%%REPO_URL%%/bin/md2html.cfg) file as part of the [md2html tool](index.html#md2html).
### [output-year-index.sh](%%REPO_URL%%/bin/output-year-index.sh)
Output the inventory for a given year's winning entries in HTML form. In other words in [1984](../1984/index.html) it would list, as links, the four winning entries, which you can see directly at the [1984 inventory](../1984/index.html#inventory). This tool is used in the [bin/md2html.cfg](%%REPO_URL%%/bin/md2html.cfg) file as part of the [md2html tool](index.html#md2html).
### [pandoc-wrapper.sh](%%REPO_URL%%/bin/pandoc-wrapper.sh)
Wrapper tool to run `pandoc(1)`.
### [quick-readme2index.sh](%%REPO_URL%%/bin/quick-readme2index.sh)
Build an entry's `index.html` file if the entry directory does not have a non-empty `index.hmtl` file, or if either `.entry.json` or `README.md` is newer than the `index.hmtl` file. This is useful when only a few entries have been modified (resulting in an updated `.entry.json` file) or if the `README.md` of a few entries have been changed: while the [readme2index.sh](index.html#readme2index) script takes a few seconds to run for a few entries, when applied to 300+ entries, the extra time can add up. If only a few `index.hmtl` files need updating, then this command will only briefly pause while the [readme2index.sh](index.html#readme2index) can take much longer. Usage: ``` # For all entries: bin/all-run.sh -v 3 bin/quick-readme2index.sh -v 1 # For an individual entry: bin/quick-readme2index.sh -v 1 2020/ferguson2 ``` **NOTE**: This command assumes that the relative modification times for `README.md`, `.entry.json`, and `index.html` are correct. If in doubt, use [readme2index.sh](index.html#readme2index). We recommend that this tool be invoked via the top level `Makefile`: ``` make quick_readme2index ```
### [readme2index.sh](%%REPO_URL%%/bin/readme2index.sh)
Convert an entry's `README.md` into its `index.html` file. Usage: ``` bin/readme2index.sh -v 3 1984/mullender ``` To build `index.html` files for all entries: ``` bin/all-run.sh -v 3 bin/readme2index.sh -v 1 ``` We recommend that this tool be invoked via the top level `Makefile`: ``` make entry_index ```
### [sgi.sh](%%REPO_URL%%/bin/sgi.sh)
Sort `.gitignore` content from `stdin` to `stdout`. We sort with lines starting with `#` first. We sort with lines starting with `*` second. We sort with lines that do not start with `[#!*]` third. We sort with lines starting with `!` fourth. This tool is used by [sort.gitignore.sh](index.html#sort-gitignore).
### [sort.gitignore.sh](%%REPO_URL%%/bin/sort.gitignore.sh)
Sort a `.gitignore` in a entry directory. Usage: ``` bin/sort.gitignore.sh -v 1 YYYY/dir ``` Suggested usage (for all `.gitignore` files): ``` bin/all-run.sh bin/sort.gitignore.sh -v 1 ``` We recommend that this tool be invoked via the top level `Makefile`: ``` make sort_gitignore ```
### [status2html.sh](%%REPO_URL%%/bin/status2html.sh)
Convert [status.json](../status.json) into HTML. This tool is a 'before tool' (`-b tool`) that is intended to be used by [gen-status.sh](index.html#gen-status).
### [subst.default.sh](%%REPO_URL%%/bin/subst.default.sh)
Print default substitutions. This tool is used in the [bin/md2html.cfg](%%REPO_URL%%/bin/md2html.cfg) file.
### [subst.entry-index.sh](%%REPO_URL%%/bin/subst.entry-index.sh)
Print substitutions for an entry's `index.html`. This tool is used in the [bin/md2html.cfg](%%REPO_URL%%/bin/md2html.cfg) file.
### [subst.entry-navbar.awk](%%REPO_URL%%/bin/subst.entry-navbar.awk)
Output substitutions for `navbar` on behalf of an entry. This tool is used in [subst.entry-index.sh](index.html#subst-entry-index).
### [subst.year-index.sh](%%REPO_URL%%/bin/subst.year-index.sh)
Print substitutions for a year level `index.html`. This tool is used in the [bin/md2html.cfg](%%REPO_URL%%/bin/md2html.cfg) file.
### [subst.year-navbar.awk](%%REPO_URL%%/bin/subst.year-navbar.awk)
Output substitutions for `navbar` on behalf of a year level `index.html`. This tool is used in [subst.year-index.sh](index.html#subst-year-index).
## [tar-entry.sh](%%REPO_URL%%/bin/tar-entry.sh)
Form a compressed tarball for an entry. Usage: ``` bin/tar-entry.sh -v 1 YYYY/dir ``` Suggested usage: ``` bin/all-run.sh -v 3 bin/tar-entry.sh -v 1 -W ```
## [tar-year.sh](%%REPO_URL%%/bin/tar-year.sh)
Form a compressed tarball for an IOCCC year. Usage: ``` bin/tar-year.sh -v 1 YYYY ``` Suggested usage: ``` bin/all-years.sh -v 3 bin/tar-year.sh -v 1 -W ``` We recommend that this tool be invoked via the top level `Makefile`: ``` make form_year_tarball ```
## [untar-entry.sh](%%REPO_URL%%/bin/untar-entry.sh)
Untar an entry's compressed tarball. Usage: ``` bin/untar-entry.sh -v 1 YYYY/dir ``` Suggested usage: ``` bin/all-run.sh -v 3 bin/untar-entry.sh -v 1 ``` We recommend that this tool be invoked via the top level `Makefile`: ``` make untar_entry_tarball ```
## [untar-year.sh](%%REPO_URL%%/bin/untar-year.sh)
Untar an IOCCC year's compressed tarball. Usage: ``` bin/untar-year.sh -v 1 YYYY ``` Suggested usage: ``` bin/all-years.sh -v 3 bin/untar-year.sh -v 1 ``` We recommend that this tool be invoked via the top level `Makefile`: ``` make untar_year_tarball ```
# How IOCCC HTML content is built
The [md2html.sh](index.html#md2html) tool is the primary tool that is used to form all IOCCC related HTML pages for the [official IOCCC website](https://www.ioccc.org). Nearly all IOCCC related HTML pages are built from markdown files, from either permanent markdown files or temporary generated markdown files, as well as HTML fragments from the [inc directory](%%REPO_URL%%/inc/index.html). Most HTML content is built from permanent markdown files, such as a `README.md` markdown file found in each entry directory. Some HTML content are generated from temporary markdown files. These temporary markdown files are produced by tools in the [bin directory](index.html) and exist only while the tool is running. In addition to converting markdown to HTML, the canonical way that HTML content is built uses, by default, files from the [inc directory](%%REPO_URL%%/inc/index.html), of the file name form: ``` *.default.html ``` By using command line options of the form: ``` -H phase=name ``` where `phase` is the name of an HTML phase, a non-default file may be used. For example: ``` -H navbar=up2index ``` will cause [inc/navbar.up2index.html](%%REPO_URL%%/inc/navbar.up2index.html) instead of [inc/navbar.default.html](%%REPO_URL%%/inc/navbar.default.html) to be used during the _navbar_ HTML phase. If `name` is dot (i.e., `.`), then the given HTML phase is skipped. For example: ``` -H footer=. ``` will cause no HTML content to be produced during the `footer` HTML phase.
## HTML phases
The following HTML phase files are used to build HTML content: 0. [inc/top.default.html](%%REPO_URL%%/inc/top.default.html) 1. [inc/head.default.html](%%REPO_URL%%/inc/head.default.html) 2. [inc/body.default.html](%%REPO_URL%%/inc/body.default.html) 3. [inc/topbar.default.html](%%REPO_URL%%/inc/topbar.default.html) 4. [inc/header.default.html](%%REPO_URL%%/inc/header.default.html) 5. [inc/navbar.default.html](%%REPO_URL%%/inc/navbar.default.html) 6. [inc/before-content.default.html](%%REPO_URL%%/inc/before-content.default.html) Phases 7-19 are reserved for future use. 20. ((insert 'before tool' output)) 21. ((insert 'pandoc wrapper tool' output)) 22. ((insert 'after tool' output)) Phases 23-29 are reserved for future use. 30. [inc/after-content.default.html](%%REPO_URL%%/inc/after-content.default.html) 31. [inc/footer.default.html](%%REPO_URL%%/inc/footer.default.html) 32. [inc/bottom.default.html](%%REPO_URL%%/inc/bottom.default.html) Phases 33-39 are reserved for future use. In all of the above HTML phase numbers, symbols of the form `%%TOKEN%%` are substituted. Substitutions of the form `%%TOKEN%%` are performed on included HTML content, content generated by the 'before tool' (`-b tool`), content generated by the 'after tool' (`-a tool`), as well as the markdown content given to the `pandoc wrapper tool` (`-p tool`). It is an error, unless `-S` is given, for any phase, except phases HTML 20-29, to not substitute all `%%TOKEN%%`s. For phases HTML 20-29, any `%%TOKEN%%` that is not substituted are passed thru without substitution. See the tool [readme2index.sh](index.html#readme2index) for an example of how HTML phases are implemented.
## getopt phase processing of the command line
The command options are evaluated in the following `getopt` phases:
### `getopt` phase 0
In `getopt` phase 0 we parse command line options and save all arguments for the end of the final command line. Depending on the program, the argument at the end is directory under topdir, or it is a file under topdir. `getopt` phase 0 is the only `getopt` phase where `-d topdir` and `-c md2html.cfg` may be used.
### `getopt` phase 1
In `getopt` phase 1 we execute each `-o "output tool"` from `getopt` phase 0, parsing the output as a command line. The `-o "output tool"` may be given `optstr` from `"-O tool=optstr"`, followed by the phase 0 argument. The `-o "output tool"` is not allowed to output another `"-o tool"` or a `"-O tool=optstr"`. Instead a `-o "output tool"` may execute another `-o "output tool"` and merge the output into its own.
### `getopt` phase 2
In `getopt` phase 2 we parse the `cfg_options` from the first `md2html.cfg` line matched by a saved argument. The match is made with the (possibly modified) phase 0 argument.
### `getopt` phase 3
In `getopt` phase 3 we execute each `-o "output tool"` from `getopt` phase 2, parsing the output as a command line. The `-o "output tool"` may be given `optstr` from `"-O tool=optstr"`, followed by the phase 0 argument. The `-o "output tool"` is not allowed to output another `"-o tool"` or a `"-O tool=optstr"`. Instead a `-o "output tool"` may execute another `-o "output tool"` and merge the output into its own.
### Command line option order
Most command line options override earlier copies of the same option. However in the case of `-H phase=name`, a later `-H phase=name` only overrides an earlier use of `-H` for the same `phase` _only_. Also in the case of `-s token=value`, a later `-s token=value` only overrides an earlier use of `-s` for the same `token` _only_.
### Output tools
An "output tool" may be used to add certain additional command line options to be processed. **IMPORTANT**: An "output tool" will print each command line option / argument on a separate line. So for example, if an "output tool" wishes to convey: ``` -s TITLE='IOCCC entry locations' -H navbar=top-of-site ``` The "output tool" would output the following 4 lines: ``` -s TITLE='IOCCC entry locations' -H navbar=top-of-site ``` The command line options printed by an "output tool" are processed after all command line options, and all options from a matching line from the [md2html.cfg](%%REPO_URL%%/bin/md2html.cfg) file, and before any filename arguments on the command line. For example: ``` command [command_options ..] [cfg_options ..] [output_tool_options ..] [--] [filename_arg ..] ``` Here the `command` first processes any command line options, then it processes `cfg_options` obtained from this file from a first match of a given `file_glob`, then it processes `output tool` (i.e., `-o tool`) options found in `command_options`, and `cfg_options`, then an optional `--` (end of all options), then zero of more `filename_arg` filename arguments. The `-o tool` must **NOT** output any `-o tool` options as `-o tool` is **NOT RECURSIVE**. A `-o tool` is free, however, to execute other `-o tool` output tools and merge the output from those tools into its own.
### Special `-E exitcode` option
When `-E exitcode` is evaluated, the application will exit with the `exitcode` value. If one wishes to also output a message to `stderr`, the `-e string` must come **BEFORE** any `-E exitcode` in the command line.
### Substitution tokens
All tokens (i.e., strings of the form `%%token%%`) **MUST** be substituted (by some `-s token=value`) in all HTML output, except during phase numbers 20-29 (i.e., except `before tool` output, pandoc wrapper output, and `after tool` output), or the command will exit non-zero, unless `-S` is given. If `-S` is given, only a warning about non-substituted tokens will be written to `stderr`. The command line of tools in the [bin directory](index.html), and perhaps modified via the [md2html config file](%%REPO_URL%%/bin/md2html.cfg) may change to using a different filename for a given phase. For example when forming the HTML from [2020/ferguson1/chocolate-cake.md](%%REPO_URL%%/2020/ferguson1/chocolate-cake.md), a different `navbar` navigation bar is needed. So instead of the usual top navigation bar that normally directs people to the previous entry for the year, or go up to the year page, or to the next entry for the year, a top navigation bar to just go up to the entry's main page is needed. A line in the [md2html config file](%%REPO_URL%%/bin/md2html.cfg) that refers to [2020/ferguson1/chocolate-cake.md](%%REPO_URL%%/2020/ferguson1/chocolate-cake.md) may specify use of `navbar.up2index.html` (as `navbar.up2index`)instead of using the `navbar.default.html` (`navbar.default.html`) default. The HTML phase may be skipped resulting in no HTML output during a given phase and furthermore, forming no HTML content from a given markdown file altogether. See comments in the [md2html config file](%%REPO_URL%%/bin/md2html.cfg) for details. See also, the tool [readme2index.sh](index.html#readme2index) for an example of how such command lines are used. # Use CAUTION when modifying `inc/` files Some of the files under this directory are used to form **MOST** of the HTML content on the [official IOCCC website](https://www.ioccc.org). ... and in particular files under [inc](%%REPO_URL%%/inc/index.html) that are of the form (called default HTML files) ...: ``` *.default.html ``` ... contain default content used to form IOCCC HTML / IOCCC web pages. Instead of editing the default HTML files in order to fix a special web page, consider making a copy of the default file and modifying the [md2html config file](%%REPO_URL%%/bin/md2html.cfg) to refer to the copy instead. That way your special case situation will not impact **MOST** of the HTML content. # Why we do not use certain well-known HTML technologies You may wonder why we need these files in the first place. You may wonder why we even need the tools in the [bin](index.html) directory when there other solutions available to form web pages.
## Static web pages only
Here are some reasons why we are using these files and special tools to create HTML content / IOCCC web pages: We host the [official IOCCC website](https://www.ioccc.org) via [GitHub pages](https://pages.github.com). As of the time this file written, **only static web pages are supported**. ## We cannot use server side include We use [static web pages](#static-only), so use of "server side include" is not available to the IOCCC. For example, Apache SSI `#include` does not work on [GitHub pages](https://pages.github.com). ## We cannot use a back-end database We use [static web pages](index.html#static-only), so use of a "back-end database" is not available to the IOCCC.
## We cannot use non-GitHub web servers
The [GitHub pages](https://pages.github.com) have the distributed server capacity needed to handle the **huge download volume** or [Slashdot effect](https://en.wikipedia.org/wiki/Slashdot_effect) that happens when new IOCCC winning entries are published. Most so-called low cost web hosting sites have a somewhat hidden excessive bandwidth charge and/or cap bandwidth when the volume gets too high, and/or do not have the infrastructure that can handle the [Slashdot effect](https://en.wikipedia.org/wiki/Slashdot_effect). The IOCCC had experience with people offering web mirrors and even free hosting only to find that the generous benefactor moves on, or their service is purchased by someone else who is not so generous. While it is possible that GitHub might someday suffer a similar fate, for the time being we are betting that GitHub will remain willing to host the IOCCC. The [official IOCCC website](https://www.ioccc.org) is, after all, primarily C source code with some supporting documentation (sometimes :-) ). As such it is a natural fit for GitHub and [GitHub pages](https://pages.github.com). ## We cannot use the HTML `` element The `` HTML element does not work for our needs. HTML elements do not extend into the content that they include. For example, menu bars (see the [ioccc.css](../ioccc.css) stylesheet) will not operate under an HTML element. ## We cannot use the HTML `` element The `` HTML element does not work for our needs. This element wants one to specify the `width` and `height` in pixels. Use of a percentage is not officially supported even if some browsers might do so. Our Responsive Web Design in the [ioccc.css](../ioccc.css) stylesheet needs to be responsive to small-sized cell phone-like screens, mid-sized table-like screens, as well as large-sized desktop-like screens. Specifying a `width` and `height` in pixels will not work well in all of those screen size contexts. ## We cannot use the HTML `