mirror/ioccc
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

improve markdown to HTML tools, build location.html, authors.html, years.html

Browse Source 
Renamed `.winner.json` to `.entry.json` as per recent decisions made for issue #2156.

JSON element _winner_JSON_format_version_ renamed to _entry_JSON_format_version_
in `.winner.json`.

Changed _entry_JSON_format_version_ from "1.0 2024-01-14" to "1.1 2024-02-11".

JSON element _winner_id_ renamed to _entry_id_ in `.entry.json`
JSON element _winner_text renamed to _entry_text_ in `.entry.json`
JSON element _winner_id_ renamed to _entry_id_ in `author/author_handle.json` files.

Changed _author_JSON_format_version_ from "1.0 2023-06-10" to "1.1 2024-02-11".

Renamed `tmp/gen_winner_json.sh` to `tmp/gen_entry_json.sh`.

Renamed `tmp/example.dot_winner.json` to `tmp/example.dot_entry.json`.

Renamed `tmp/winner_id.txt` to `tmp/entry_id.txt`.

Renamed `tmp/winner_dir.txt` to `tmp/entry_dir.txt`.

Moved outdated `tmp/awards_check.sh` into `tmp/OLD_manifest/awards_check.sh`.
Moved outdated `tmp/awards_check.sh` into `tmp/OLD_manifest/awards_check.sh`.
Moved outdated `tmp/src/winner_name` into `tmp/OLD_manifest/winner_name`.
Moved outdated `tmp/sql` into `tmp/OLD_manifest/sql`.
Moved outdated `tmp/template-footer.html` into `tmp/OLD_manifest/template-footer.html`.
Moved outdated `tmp/template-header.html` into `tmp/OLD_manifest/template-header.html`.
Moved outdated `tmp/template-top.html` into `tmp/OLD_manifest/template-top.html`.
Moved outdated `tmp/test.html` into `tmp/OLD_manifest/test.html`.
Moved outdated `tmp/fake.gen_index.sh` into `tmp/OLD_manifest/fake.gen_index.sh`.
Moved outdated `tmp/old-ioccc.css` into `tmp/OLD_manifest/old-ioccc.css`.

Removed winner_name from `tmp/src/Makefile`.

Updated `tmp/example.dot_entry.json` to reflect the above JSON changes.

Updated `tmp/manifest.numbers` to reflect the above JSON changes.

Updated `tmp/README.md` to reflect the above JSON changes.
Sorted `tmp/README.md` by existing `tmp` filenames.

Updated all `.entry.json` files to reflect the above JSON changes.

Updated all `author/author_handle.json` files to reflect the above JSON changes.

Added new tool `bin/gen-location.sh` to form the top level file `location.html`
from `author/author_handle.json` files!

Added new tool `bin/gen-years.sh` to form the top level file `years.html`
using `.top`, `.year` and `.entry.json' files!

Added new tool `bin/gen-authors.sh` to form a new top level file
`/authors.html` from `author/author_handle.json` and related
`.entry.json' files!

The old `winners.html` file now redirects to the new `authors.html` file.

Renamed `bin/gen-author.sh` to `bin/output-index-author.sh`.
Renamed `bin/subst.winner-index.sh` to `subst.entry-index.sh`.
Renamed `bin/subst.winner-index.awk` to `subst.entry-index.awk`.
Renamed `bin/manifest.entry.json.awk` to `bin/manifest.entry.json.awk`.
Renamed `bin/gen-inventory.sh` to `bin/output-index-inventory.sh`.
Renamed `bin/sort.gitignore` to `bin/sort.gitignore.sh`.
Renamed `bin/sgi` to `bin/sgi.sh`.

The core logic of converting markdown to HTML has been
generalized into a new `bin/md2html.sh` tool.

The `bin/readme2index.sh` tool now calls the
new `bin/md2html.sh` tool to form entry's `index.html` files.

The `bin/quick-readme2index.sh` tool now calls the new `bin/md2html.sh`
tool, unless the `index.html` is newer than both `.entry.json` and
`README.md`, to form entry `index.html` files.

Increased, slightly, the speed of the `bin/quick-readme2index.sh` tool
with the entry's `index.html` file does not need to be updated.

The `bin/quick-readme2index.sh` tool will touch the `index.html` file,
even if the `bin/md2html.sh` tool determines that the `index.html` file
did not change, so that the `index.html` file will not appear
out of date with respect to `.entry.json`.

Slightly improved the look and feel of `index.html` files for entries.

The logic of `bin/all-run.sh` has been improved.  The initial
processing, from first bash version check, thru the -N test, serves
as the model for the `bin/readme2index.sh` tool and the
`bin/quick-readme2index.sh` tool.  There tools now share
a common set of command line options.

Changed the format of `inc/md2html.cfg` so that for a given "_file_glob_"
match, options follow, one per line.  The "_file_glob_" is on a line
by itself.  The option lines that follow must be intended:

```
[12][0-9][0-9][0-9]/README.md
	-H
	topnav=year
	-o
	bin/subst.default.sh
	-o
	bin/subst.year.sh
```

Added "-D ./" to top level markdown files in `inc/md2html.cfg`.

The above intending is performed with 1 or more whitespace
characters, although we use a leading tab "convention"
out of deference to Makefile formats. :-)

Once a new "_file_glob_" is found, the set of any previous
option lines is said to have ended.

The general argument processing logic now makes used of shell arrays.

Added checks in the `bin/md2html.sh` tool to disallow a "_file_glob_"
of . (dot) as this is used internally to help the shell process the
new `inc/md2html.cfg` format.

Changed `bin/all-run.sh` VERSION from "1.1.2 2024-01-20"
to "1.3 2024-02-11".

Changed `bin/output-index-author.sh` VERSION from "1.3.1 2024-01-16"
to "1.4 2024-02-11".

Changed `bin/pandoc-wrapper.sh` VERSION from "1.3.2 2024-01-19"
to "1.4 2024-02-05".

Changed `bin/quick-readme2index.sh` VERSION from "1.0 2024-01-20"
to "1.1 2024-02-11".

Changed `bin/readme2index.sh` VERSION from "1.1.1 2024-01-20"
to "1.2 2024-02-11".

Changed `bin/subst.default.sh` VERSION from "1.1.1 2024-01-16"
to "1.2 2024-02-05".

Changed `bin/subst.entry-index.sh` VERSION from "1.1.1 2024-01-16
to "1.2 2024-02-11".

Changed `bin/output-index-inventory.sh` VERSION from "1.0.1 2024-01-16"
to "1.1 2024-02-11".

Changed `bin/manifest.entry.json.awk` from "1.0 2024-01-14"
to "1.1 2024-02-11".

Changed `bin/sort.gitignore.sh` from "1.0 2024-02-01"
to "1.1 2024-02-11".

Changed substitution symbol _%%DOCROOT_URL_SLASH%%_ to _%%DOCROOT_SLASH%%_
in various "inc/*.html" files.

Moved setting of symbol _%%DOCROOT_SLASH%%_ from `bin/subst.entry-index.sh`
to `bin/subst.default.sh`.

Changed the variable _$DOCROOT_URL_SLASH_ to _$DOCROOT_SLASH_, in
bin tools that use it, to be more descriptive.

Removed `-P pandoc_opts` from `bin/*.sh` command lines.
The PANDOC_WRAPPER_OPTSTR value is only used in `bin/pandoc-wrapper.sh`.

Removed `-B optstr` and `-A optstr` from the `bin/md2html.sh` command line.

Removed `[-O tool=optstr ..]` from the `bin/md2html.sh` command line.

Removed old use of `-U top_url` from command lines.
Add `-w site_url` as sort of a "replacement" for the old `-U top_url`.

Added a new `-U URL` to `bin/gen-author.sh`, `bin/output-index-inventory.sh`,
`bin/md2html.sh`, `bin/subst.default.sh`, and `bin/subst.entry-index.sh`
to allow for the Nu HTML check can use the encoded absolute URL of
the HTML page.

Added `-t tagline` to most command lines.

Added `-T md2html.sh` to most command lines.

Added `-D docroot/` to most command lines to set DOCROOT_SLASH.
The DOCROOT_SLASH value defaults to "../../" for most in tools,
suitable for HTML files under a `YYYY/dir` directory, except for
`bin/gen-location.sh` and `gen-entries-html.sh` (as this produces a
top level HTML file) and `bin/md2html.sh` (forms files at any level)
who use "./" instead.

In the case of `bin/output-index-inventory.sh`, `bin/quick-readme2index.sh`,
`bin/readme2index.sh`, and `bin/all-run.sh`

Helped make the use of exit code in bin tools a bit more consistent.

Fixed a number of typos in the bin tools.

Added comments for lower level tools such as
`manifest.entry.json.awk`, `bin/subst.default.sh`,
`bin/subst.entry-index.sh`, and bin/`manifest.entry.json.awk` to
indicate how they are usually invoked by other tools.

Changed the comments that a new `bin/md2html.sh` tool (that was the
`bin/readme2index.sh` tool) would add to a entry's `index.html` file.
No longer will it add the tool version number to the comment.  Thus,
a change in tool version will not force all entry `index.html`
files to be modified.  Now both the tool adds two comments, one
is the _tagline_ to indicate how the markdown content was generated,
and one for the new `bin/md2html.sh` tool.  For example:

```
<!-- The original markdown content was formed by the tool: bin/quick-readme2index.sh -->
<!-- The HTML was converted from markdown by the tool: bin/md2html.sh -->
```

Improve the debugging, especially in regards to changing important
exported shell variables, in `bin/md2html.sh`.  Other bin tools
have their exported shell variables improved somewhat as well.

The `bin/README.md` now describes the general _converting markdown
to HTML_ procedure including _getopt phases_ and _HTML phases_.
Content from `inc/md2html.cfg` and `bin/readme2index.sh` that related
to this topic has been centralized into the `bin/README.md` file.

Improved formatting of the `bin/README.md` file.  Updated it as needed
according to the above.

Sorted `2011/.year`, `2018/.year`, and `2019/.year` to be in dictionary order.
Adjusted ${WINNERS} in year Makefile to use dictionary order so
that `make genpath` will produce `.year` files in dictionary order.

FYI: We tested this massive set of changes as best as we could.
     Nevertheless we might have missing something in the JSON changes,
     `.entry.json` renames, manifest updates, code comments, etc.
This commit is contained in:
Landon Curt Noll committed 2024-02-11 16:00:31 -08:00
1 parent b23ab491ca
commit aedb017223
897 files changed
+29985 -27475
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

bin/pandoc-wrapper.sh
+35 -47
View File
@@ -6,8 +6,8 @@
#
# This tool is intended to be invoked by a tool such as 'readme2index.sh"
# and as such, the leading options will include "-p pandoc_wrapper" and -P "optstr",
# the final arg will be in yyyy/dir form, and will be called from
# the topdir directory under which the yyyy/dir winner directory must be found.
# the final arg will be in YYYY/dir form, and will be called from
# the topdir directory under which the YYYY/dir winner directory must be found.
#
# Copyright (c) 2023,2024 by Landon Curt Noll. All Rights Reserved.
#
@@ -35,7 +35,7 @@
# firewall - must be bash with a version 4.2 or later
#
# We must declare arrays with -ag or -Ag
# We must declare arrays with -ag or -Ag, and we need loops to "export" modified variables.
#
if [[ -z ${BASH_VERSINFO[0]} || ${BASH_VERSINFO[0]} -lt 4 || ${BASH_VERSINFO[0]} -eq 4 && ${BASH_VERSINFO[1]} -lt 2 ]]; then
echo "$0: ERROR: bash version must be >= 4.2: $BASH_VERSION" 1>&2
@@ -56,7 +56,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an
# set variables referenced in the usage message
#
export VERSION="1.3.2 2024-01-19"
export VERSION="1.4 2024-02-05"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
@@ -64,8 +64,8 @@ export PANDOC_TOOL
GIT_TOOL=$(type -P git)
export GIT_TOOL
if [[ -z "$GIT_TOOL" ]]; then
echo "$0: FATAL: git tool is not installed or not in PATH" 1>&2
exit 200
echo "$0: FATAL: git tool is not installed or not in \$PATH" 1>&2
exit 210
fi
"$GIT_TOOL" rev-parse --is-inside-work-tree >/dev/null 2>&1
status="$?"
@@ -76,16 +76,20 @@ PANDOC_TOOL=$(type -P pandoc)
if [[ -z $PANDOC_TOOL ]]; then
PANDOC_TOOL="/opt/homebrew/bin/pandoc"
fi
export PANDOC_ARGS="-f markdown -t html --fail-if-warnings=true"
export PANDOC_WRAPPER="bin/pandoc_wapper.sh"
export PANDOC_WRAPPER_OPTSTR="-f markdown -t html --fail-if-warnings=true"
unset PANDOC_OPTION
declare -ag PANDOC_OPTION
PANDOC_OPTION+=("-f")
PANDOC_OPTION+=("markdown")
PANDOC_OPTION+=("-t")
PANDOC_OPTION+=("html")
PANDOC_OPTION+=("--fail-if-warnings=true")
export REPO_URL="https://github.com/ioccc-src/temp-test-ioccc"
export TOP_URL="https://ioccc-src.github.io/temp-test-ioccc"
# set usage message
#
export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-n] [-N]
[-p pandoc_tool] [-P pandoc_opts] [-u repo_url] [-U top_url] [-e string ..] [-E exitcode]
[-p pandoc_tool] [-u repo_url] [-e string ..] [-E exitcode]
file.md output.html
-h print help message and exit
@@ -98,11 +102,8 @@ export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-n] [-N]
-N do not process file, just parse arguments and ignore the file (def: process the file)
-p pandoc_tool path to the pandoc tool (not the wrapper) (def: $PANDOC_TOOL)
-P pandoc_opts options given to the pandoc tool (def: $PANDOC_ARGS)
NOTE: The 'pandoc_opts' may not contain a single-quote or a double-quote.
-u repo_url Base level URL of target git repo (def: $REPO_URL)
-U top_url Top level URL of web site where HTML files will be viewed (def: $TOP_URL)
-e string output string, followed by newline, to stderr (def: do not)
-E exitcode force exit with exitcode (def: exit based on success or failure of the action)
@@ -121,8 +122,8 @@ Exit codes:
4 bash version is < 4.2
5 file.md file not found or not readable file
6 pandoc tool not found or not executable
>= 10 < 200 ((not used))
>= 200 internal error
>= 10 < 210 ((not used))
>= 210 internal tool error
$NAME version: $VERSION"
@@ -133,7 +134,7 @@ export DO_NOT_PROCESS=
# parse command line
#
while getopts :hv:Vd:nNp:P:u:U:e:E: flag; do
while getopts :hv:Vd:nNp:u:e:E: flag; do
case "$flag" in
h) echo "$USAGE" 1>&2
exit 2
@@ -151,19 +152,15 @@ while getopts :hv:Vd:nNp:P:u:U:e:E: flag; do
;;
p) PANDOC_TOOL="$OPTARG"
;;
P) PANDOC_ARGS="$OPTARG"
;;
u) REPO_URL="$OPTARG"
;;
U) TOP_URL="$OPTARG"
;;
e) echo "$OPTARG" 1>&2
;;
E) exit "$OPTARG"
;;
\?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2
echo 1>&2
print_usage 1>&2
echo "$USAGE" 1>&2
exit 3
;;
:) echo "$0: ERROR: option -$OPTARG requires an argument" 1>&2
@@ -224,16 +221,20 @@ fi
# verify that we have a topdir directory
#
REPO_NAME=$(basename "$REPO_URL")
export REPO_NAME
if [[ -z $TOPDIR ]]; then
echo "$0: ERROR: cannot find top of git repo directory" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_URL; cd $REPO_NAME" 1>&2
exit 201
exit 211
fi
if [[ ! -e $TOPDIR ]]; then
echo "$0: ERROR: TOPDIR does not exist: $TOPDIR" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_URL; cd $REPO_NAME" 1>&2
exit 212
fi
if [[ ! -d $TOPDIR ]]; then
echo "$0: ERROR: TOPDIR is not a directory: $TOPDIR" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_URL; cd $REPO_NAME" 1>&2
exit 202
exit 213
fi
# verify that we have an author subdirectory
@@ -241,7 +242,7 @@ fi
export AUTHOR_PATH="$TOPDIR/author"
if [[ ! -d $AUTHOR_PATH ]]; then
echo "$0: ERROR: author is not a directory under topdir: $AUTHOR_PATH" 1>&2
exit 203
exit 214
fi
export AUTHOR_DIR="author"
@@ -250,7 +251,7 @@ export AUTHOR_DIR="author"
export INC_PATH="$TOPDIR/inc"
if [[ ! -d $INC_PATH ]]; then
echo "$0: ERROR: inc is not a directory under topdir: $INC_PATH" 1>&2
exit 204
exit 215
fi
export INC_DIR="inc"
@@ -259,7 +260,7 @@ export INC_DIR="inc"
export BIN_PATH="$TOPDIR/bin"
if [[ ! -d $BIN_PATH ]]; then
echo "$0: ERROR: bin is not a directory under topdir: $BIN_PATH" 1>&2
exit 205
exit 216
fi
export BIN_DIR="bin"
@@ -267,11 +268,11 @@ export BIN_DIR="bin"
#
if [[ ! -e $TOPDIR ]]; then
echo "$0: ERROR: cannot cd to non-existent path: $TOPDIR" 1>&2
exit 206
exit 217
fi
if [[ ! -d $TOPDIR ]]; then
echo "$0: ERROR: cannot cd to a non-directory: $TOPDIR" 1>&2
exit 207
exit 218
fi
export CD_FAILED
if [[ $V_FLAG -ge 5 ]]; then
@@ -280,7 +281,7 @@ fi
cd "$TOPDIR" || CD_FAILED="true"
if [[ -n $CD_FAILED ]]; then
echo "$0: ERROR: cd $TOPDIR failed" 1>&2
exit 208
exit 219
fi
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: now in directory: $(/bin/pwd)" 1>&2
@@ -292,7 +293,6 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: NAME=$NAME" 1>&2
echo "$0: debug[3]: REPO_URL=$REPO_URL" 1>&2
echo "$0: debug[3]: REPO_NAME=$REPO_NAME" 1>&2
echo "$0: debug[3]: TOP_URL=$TOP_URL" 1>&2
echo "$0: debug[3]: AUTHOR_PATH=$AUTHOR_PATH" 1>&2
echo "$0: debug[3]: AUTHOR_DIR=$AUTHOR_DIR" 1>&2
echo "$0: debug[3]: INC_PATH=$INC_PATH" 1>&2
@@ -316,27 +316,15 @@ fi
#
if [[ -z $NOOP ]]; then
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: will execute pandoc as: $PANDOC_TOOL -o $HTML_OUTPUT $PANDOC_ARGS $MARKDOWN_INPUT" 1>&2
echo "$0: debug[1]: will execute pandoc as: $PANDOC_TOOL ${PANDOC_OPTION[*]} -o $HTML_OUTPUT $MARKDOWN_INPUT" 1>&2
fi
# The pandoc args $PANDOC_ARGS does contain things that must be
# globbed and split (SC2086) for the pandoc tool.
#
# SC2086 (info): Double quote to prevent globbing and word splitting.
# https://www.shellcheck.net/wiki/SC2086
# shellcheck disable=SC2086
"$PANDOC_TOOL" -o "$HTML_OUTPUT" $PANDOC_ARGS "$MARKDOWN_INPUT"
"$PANDOC_TOOL" "${PANDOC_OPTION[@]}" -o "$HTML_OUTPUT" "$MARKDOWN_INPUT"
status="$?"
else
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[3]: will execute pandoc as: $PANDOC_TOOL $PANDOC_ARGS -o /dev/null $MARKDOWN_INPUT" 1>&2
echo "$0: debug[3]: will execute pandoc as: $PANDOC_TOOL ${PANDOC_OPTION[*]} -o /dev/null $MARKDOWN_INPUT" 1>&2
fi
# The pandoc args $PANDOC_ARGS does contain things that must be
# globbed and split (SC2086) for the pandoc tool.
#
# SC2086 (info): Double quote to prevent globbing and word splitting.
# https://www.shellcheck.net/wiki/SC2086
# shellcheck disable=SC2086
"$PANDOC_TOOL" $PANDOC_ARGS -o /dev/null "$MARKDOWN_INPUT"
"$PANDOC_TOOL" "${PANDOC_OPTION[*]}" -o /dev/null "$MARKDOWN_INPUT"
status="$?"
fi
if [[ $status -ne 0 ]]; then