mirror/ioccc
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

use jsp tool, workaround JSON Unicode bugs, misc fixes

Browse Source 
If the [jsp tool](https://github.com/kjozsa/jsp) is
installed, it will be used in favor of the [JSONPath.sh
tool](https://github.com/lcn2/JSONPath.sh).

Updated `bin/jfmt-wrapper.sh`, `bin/jprint-wrapper.sh`, and
`bin/jval-wrapper.sh` to use jsp if found.  These 3 tools may be given
`-t tool` where tool is either "_jsp_" or "_JSONPath.sh_" for force the
use of the tool.  They also use `-Q` to perform a quick check tool using
trivial input.

The `bin/jval-wrapper.sh` tool had been simplified.  The `-w`,
`-i` and `-u` options are gone.  Instead it has:

```
    -b    print only values (def: print both JSON member and JSON value)
    -q    remove enclosing double quotes (def: keep any enclosing double quotes)
```

Add `make all_jfmt` to Makefile to canonically format all entry and
author handle JSON files.

A sed script `bin/unicode-fix.sed` has been added that converts
JSON string "_\u[0-9A-Fa-f]{4}_" encoding that may arise from known
JSON data.  This is done as a work-a-round for [Unicode / UTF-8
issues](https://github.com/xexyl/jparse/issues/13) with `jstrdecode(1)`.

The `bin/unicode-chk.sh` tool, if given on file arguments, scans
all `.entry.json` and `author/author_handle.json` JSON files to be
sure that, after `bin/unicode-fix.sed` processing, no JSON string
"_\u[0-9A-Fa-f]{4}_" encodings remain.

Removed all `-j jparse` args from commands.  Instead the `jparse(1)`
tool is looked for along $PATH.

Added `combine_author_handle.sh` tool to transform all of the
`author/author_handle.json` files into a temporary as a single JSON
file so that `bin/jval-wrapper.sh` tool may be used to extract all
"_sort_word_" or all "_entry_id_" values.

Changed `bin/all-jfmt.sh` VERSION from "1.0 2024-08-31"
to "1.1 2024-09-24".

Changed `bin/csv2entry.sh` VERSION from "1.2.3 2024-08-30"
to "1.3 2024-09-23".

Changed `bin/cvt-submission.sh` VERSION from "1.1 2024-08-31"
to "1.2 2024-09-23".

Changed `bin/entry2csv.sh` VERSION from "1.2.1 2024-08-17"
to "1.3 2024-09-23".

Changed `bin/gen-authors.sh` VERSION from "1.12 2024-08-18"
to "1.13 2024-09-23".

Changed `bin/gen-location.sh` VERSION from "1.6 2024-08-18"
to "1.7 2024-09-23".

Changed `bin/gen-status.sh` VERSION from "1.6 2024-08-17"
to "1.7 2024-09-23".

Changed `bin/gen-years.sh` VERSION from "1.4 2024-08-17"
to "1.5 2024-09-23".

Changed `bin/jfmt-wrapper.sh` VERSION from "1.1 2024-08-20"
to "1.2 2024-09-23".

Changed `bin/jprint-wrapper.sh` VERSION from "1.1 2024-08-20"
to "1.3 2024-09-23".

Changed `bin/jval-wrapper.sh` VERSION from "1.0.4 2024-08-20"
to "1.1 2024-09-23".

Changed `bin/manifest.entry.json.awk` VERSION from "1.1.2 2024-07-28"
to "1.2 2024-09-17".

Changed `bin/output-index-author.sh` VERSION from "1.8 2024-08-17"
to "1.9 2024-09-23".

Changed `bin/output-year-index.sh` VERSION from "1.4.4 2024-08-05"
to "1.5 2024-09-23".

Changed `bin/subst.entry-index.sh` VERSION from "1.5 2024-08-13"
to "1.6 2024-09-23".

Added `bin/combine_author_handle.sh` VERSION "1.0 2024-09-23".

Added `bin/unicode-chk.sh` VERSION "1.0 2024-09-23".

Added `bin/unicode-fix.sed` VERSION "1.0 2024-09-23".

Perform `make www` to perform the above and test under macOS.

FYI: On our MacBook Pro, `make www` took:

```
    make www  631.79s user 264.95s system 94% cpu 15:53.71 total
```

FYI: On our MacBook Pro, `make quick_www` took:

```
    make quick_www  81.43s user 58.26s system 102% cpu 2:16.02 total
```
This commit is contained in:
Landon Curt Noll committed 2024-09-24 04:04:27 -07:00
1 parent e84d6eb095
commit 499be5983d
19 files changed
+3044 -1213
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

bin/all-jfmt.sh
+2 -39
View File
@@ -86,7 +86,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an
# set variables referenced in the usage message
#
export VERSION="1.0 2024-08-31"
export VERSION="1.1 2024-09-24"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
@@ -112,9 +112,6 @@ export SITE_URL="https://ioccc-src.github.io/temp-test-ioccc"
export NOOP=
export DO_NOT_PROCESS=
export EXIT_CODE="0"
#
JPARSE_TOOL=$(type -P jparse)
export JPARSE_TOOL
# clear options we will add to tools
@@ -125,7 +122,7 @@ declare -ag TOOL_OPTION
# usage
#
export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-D docroot/] [-j jparse] [-n] [-N]
export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-D docroot/] [-n] [-N]
-h print help message and exit
-v level set verbosity level (def level: 0)
@@ -136,8 +133,6 @@ export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-D docroot/] [-j jpars
NOTE: The '-D docroot/' is passed as leading options on tool command lines.
NOTE: 'docroot' must end in a slash
-j jparse path to the jparse tool (def: $JPARSE_TOOL)
-n go thru the actions, but do not update any files (def: do the action)
-N do not process anything, just parse arguments (def: process something)
@@ -182,8 +177,6 @@ while getopts :hv:Vd:D:j:nN flag; do
TOOL_OPTION+=("-D")
TOOL_OPTION+=("$DOCROOT_SLASH")
;;
j) JPARSE_TOOL="$OPTARG"
;;
n) NOOP="-n"
;;
N) DO_NOT_PROCESS="-N"
@@ -326,22 +319,6 @@ if [[ ! -s $TOP_FILE ]]; then
fi
# verify jparse tool
#
if [[ -z "$JPARSE_TOOL" ]]; then
echo "$0: ERROR: cannot find an executable jparse tool" 1>&2
exit 5
fi
if [[ ! -f "$JPARSE_TOOL" ]]; then
echo "$0: ERROR: jparse is not a file: $JPARSE_TOOL" 1>&2
exit 5
fi
if [[ ! -x "$JPARSE_TOOL" ]]; then
echo "$0: ERROR: jparse is not an executable file: $JPARSE_TOOL" 1>&2
exit 5
fi
# print running info if verbose
#
# If -v 3 or higher, print exported variables in order that they were exported.
@@ -371,7 +348,6 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2
echo "$0: debug[3]: JFMT_WRAPPER=$JFMT_WRAPPER" 1>&2
echo "$0: debug[3]: TOP_FILE=$TOP_FILE" 1>&2
echo "$0: debug[3]: JPARSE_TOOL=$JPARSE_TOOL" 1>&2
fi
@@ -569,19 +545,6 @@ for YYYY in $(< "$TOP_FILE"); do
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: about to run: $JPARSE_TOOL -q -- $ENTRY_JSON" 1>&2
fi
if "$JPARSE_TOOL" -q -- "$ENTRY_JSON"; then
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: valid JSON: $ENTRY_JSON" 1>&2
fi
else
echo "$0: ERROR: .entry.json is not valid JSON: $ENTRY_JSON" 1>&2
EXIT_CODE="7" # exit 7
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
# canonically format YYYY/dir/.entry.json unless -n
#
bin/combine_author_handle.sh
Executable
+447
View File
@@ -0,0 +1,447 @@
#!/usr/bin/env bash
#
# combine_author_handle.sh - output all author/author_handle.JSON files as single JSON file
#
# We form a single JSON file out of all author/author_handle.JSON files.
# We cannot simply cat the author/author_handle.JSON files because
# each author/author_handle.JSON file opens and closes the JSON.
# So this tool removes the intermediate JSON document opening "{"
# and closing "}".
#
# Because jsp open remembers the last copy of a given JSON member name.
# As such, we change the JSON member name "winning_entry_set" into
# a unique "winning_entry_set.FILENO" where FILENO is the file number.
# This the JSON member value will be preserved across all files.
#
# We also convert where FILENO is the file number and FILENAME is the filename:
#
# "sort_word" : "data",
#
# into:
#
# "sort_word.FILENO" : [
# {
# "sort_word" : "data FILENAME"
# }
# ],
#
# This will allow jsp to print sort_word values.
#
# We also make sure that the last item from a given file ends in a comma (",)",
# due to the bogosity of the so-called JSON spec.
#
# NOTE: We assume that all JSON files have been formatted with the
# bin/jprint-wrapper.sh tool. In particular the first line is just "{:,
# and the last line is just "}" and each JSON element is on its own line.
#
# Copyright (c) 2024 by Landon Curt Noll. All Rights Reserved.
#
# Permission to use, copy, modify, and distribute this software and
# its documentation for any purpose and without fee is hereby granted,
# provided that the above copyright, this permission notice and text
# this comment, and the disclaimer below appear in all of the following:
#
# supporting documentation
# source copies
# source works derived from this source
# binaries derived from this source or from derived source
#
# LANDON CURT NOLL DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE,
# INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO
# EVENT SHALL LANDON CURT NOLL BE LIABLE FOR ANY SPECIAL, INDIRECT OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF
# USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
# OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
# PERFORMANCE OF THIS SOFTWARE.
#
# chongo (Landon Curt Noll, http://www.isthe.com/chongo/index.html) /\oo/\
#
# Share and enjoy! :-)
# firewall - run only with a bash that is version 5.1.8 or later
#
# The "/usr/bin/env bash" command must result in using a bash that
# is version 5.1.8 or later.
#
# We could relax this version and insist on version 4.2 or later. Versions
# of bash between 4.2 and 5.1.7 might work. However, to be safe, we will require
# bash version 5.1.8 or later.
#
# WHY 5.1.8 and not 4.2? This safely is done because macOS Homebrew bash we
# often use is "version 5.2.26(1)-release" or later, and the RHEL Linux bash we
# use often use is "version 5.1.8(1)-release" or later. These versions are what
# we initially tested. We recommend you either upgrade bash or install a newer
# version of bash and adjust your $PATH so that "/usr/bin/env bash" finds a bash
# that is version 5.1.8 or later.
#
# NOTE: The macOS shipped, as of 2024 March 15, a version of bash is something like
# bash "version 3.2.57(1)-release". That macOS shipped version of bash
# will NOT work. For users of macOS we recommend you install Homebrew,
# (see https://brew.sh), and then run "brew install bash" which will
# typically install it into /opt/homebrew/bin/bash, and then arrange your $PATH
# so that "/usr/bin/env bash" finds "/opt/homebrew/bin" (or whatever the
# Homebrew bash is).
#
# NOTE: And while MacPorts might work, we noticed a number of subtle differences
# with some of their ported tools to suggest you might be better off
# with installing Homebrew (see https://brew.sh). No disrespect is intended
# to the MacPorts team as they do a commendable job. Nevertheless we ran
# into enough differences with MacPorts environments to suggest you
# might find a better experience with this tool under Homebrew instead.
#
if [[ -z ${BASH_VERSINFO[0]} ||
${BASH_VERSINFO[0]} -lt 5 ||
${BASH_VERSINFO[0]} -eq 5 && ${BASH_VERSINFO[1]} -lt 1 ||
${BASH_VERSINFO[0]} -eq 5 && ${BASH_VERSINFO[1]} -eq 1 && ${BASH_VERSINFO[2]} -lt 8 ]]; then
echo "$0: ERROR: bash version needs to be >= 5.1.8: $BASH_VERSION" 1>&2
echo "$0: Warning: bash version >= 4.2 might work but 5.1.8 was the minimum we tested" 1>&2
echo "$0: Notice: For macOS users: install Homebrew (see https://brew.sh), then run" \
""brew install bash" and then modify your \$PATH so that \"#!/usr/bin/env bash\"" \
"finds the Homebrew installed (usually /opt/homebrew/bin/bash) version of bash" 1>&2
exit 4
fi
# setup bash file matching
#
# We must declare arrays with -ag or -Ag, and we need loops to "export" modified variables.
# This requires a bash with a version 4.2 or later. See the larger comment above about bash versions.
#
shopt -s nullglob # enable expanded to nothing rather than remaining unexpanded
shopt -u failglob # disable error message if no matches are found
shopt -u dotglob # disable matching files starting with .
shopt -u nocaseglob # disable strict case matching
shopt -u extglob # enable extended globbing patterns
shopt -s globstar # enable ** to match all files and zero or more directories and subdirectories
# set variables referenced in the usage message
#
export VERSION="1.0 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
GIT_TOOL=$(type -P git)
export GIT_TOOL
if [[ -z "$GIT_TOOL" ]]; then
echo "$0: FATAL: git tool is not installed or not in \$PATH" 1>&2
exit 5
fi
"$GIT_TOOL" rev-parse --is-inside-work-tree >/dev/null 2>&1
status="$?"
if [[ $status -eq 0 ]]; then
TOPDIR=$("$GIT_TOOL" rev-parse --show-toplevel)
fi
export TOPDIR
export DOCROOT_SLASH="../../"
export TAGLINE="unspecified"
export REPO_TOP_URL="https://github.com/ioccc-src/temp-test-ioccc"
# GitHub puts individual files under the "blob/master" sub-directory.
export REPO_URL="$REPO_TOP_URL/blob/master"
export SITE_URL="https://ioccc-src.github.io/temp-test-ioccc"
#
export NOOP=
export DO_NOT_PROCESS=
export EXIT_CODE="0"
# usage
#
export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-D docroot/] [-n] [-N]
-h print help message and exit
-v level set verbosity level (def level: 0)
-V print version string and exit
-d topdir set topdir (def: $TOPDIR)
-D docroot/ set the document root path followed by slash (def: $DOCROOT_SLASH)
NOTE: The '-D docroot/' is passed as leading options on tool command lines.
NOTE: 'docroot' must end in a slash
-n go thru the actions, but do not update any files (def: do the action)
-N do not process anything, just parse arguments (def: process something)
Exit codes:
0 all OK
1 found unexpected \\uHexHexHexHex strings in JSON files
2 -h and help string printed or -V and version string printed
3 command line error
4 bash version is too old
5 some internal tool is not found or not an executable file
6 problems found with or in the topdir or topdir/YYYY directory
7 problems found with or in the entry topdir/YYYY/dir directory
8 file.json not a readable file
>= 10 internal error
$NAME version: $VERSION"
# parse command line
#
while getopts :hv:Vd:D:j:nN flag; do
case "$flag" in
h) echo "$USAGE" 1>&2
exit 2
;;
v) V_FLAG="$OPTARG"
;;
V) echo "$VERSION"
exit 2
;;
d) TOPDIR="$OPTARG"
;;
D) # parse -D docroot/
case "$OPTARG" in
*/) ;;
*) echo "$0: ERROR: in -D docroot/, the docroot must end in /" 1>&2
echo 1>&2
print_usage 1>&2
exit 3
;;
esac
DOCROOT_SLASH="$OPTARG"
;;
n) NOOP="-n"
;;
N) DO_NOT_PROCESS="-N"
;;
\?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2
echo 1>&2
print_usage 1>&2
exit 3
;;
:) echo "$0: ERROR: option -$OPTARG requires an argument" 1>&2
echo 1>&2
echo "$USAGE" 1>&2
exit 3
;;
*) echo "$0: ERROR: unexpected value from getopts: $flag" 1>&2
echo 1>&2
echo "$USAGE" 1>&2
exit 3
;;
esac
done
#
# remove the options
#
shift $(( OPTIND - 1 ));
#
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: file argument count: $#" 1>&2
fi
if [[ $# -ne 0 ]]; then
echo "$0: ERROR: expected 0 args, found: $#" 1>&2
echo 1>&2
echo "$USAGE" 1>&2
exit 3
fi
# verify that we have a topdir directory
#
REPO_NAME=$(basename "$REPO_TOP_URL")
export REPO_NAME
if [[ -z $TOPDIR ]]; then
echo "$0: ERROR: cannot find top of git repo directory" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
if [[ ! -e $TOPDIR ]]; then
echo "$0: ERROR: TOPDIR does not exist: $TOPDIR" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
if [[ ! -d $TOPDIR ]]; then
echo "$0: ERROR: TOPDIR is not a directory: $TOPDIR" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
# cd to topdir
#
if [[ ! -e $TOPDIR ]]; then
echo "$0: ERROR: cannot cd to non-existent path: $TOPDIR" 1>&2
exit 6
fi
if [[ ! -d $TOPDIR ]]; then
echo "$0: ERROR: cannot cd to a non-directory: $TOPDIR" 1>&2
exit 6
fi
export CD_FAILED
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to: cd $TOPDIR" 1>&2
fi
cd "$TOPDIR" || CD_FAILED="true"
if [[ -n $CD_FAILED ]]; then
echo "$0: ERROR: cd $TOPDIR failed" 1>&2
exit 6
fi
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: now in directory: $(/bin/pwd)" 1>&2
fi
# verify that we have an author subdirectory
#
export AUTHOR_PATH="$TOPDIR/author"
if [[ ! -d $AUTHOR_PATH ]]; then
echo "$0: ERROR: author is not a directory under topdir: $AUTHOR_PATH" 1>&2
exit 6
fi
export AUTHOR_DIR="author"
# verify we have a non-empty readable .top file
#
export TOP_FILE=".top"
if [[ ! -e $TOP_FILE ]]; then
echo "$0: ERROR: .top does not exist: $TOP_FILE" 1>&2
exit 6
fi
if [[ ! -f $TOP_FILE ]]; then
echo "$0: ERROR: .top is not a regular file: $TOP_FILE" 1>&2
exit 6
fi
if [[ ! -r $TOP_FILE ]]; then
echo "$0: ERROR: .top is not an readable file: $TOP_FILE" 1>&2
exit 6
fi
if [[ ! -s $TOP_FILE ]]; then
echo "$0: ERROR: .top is not a non-empty readable file: $TOP_FILE" 1>&2
exit 6
fi
# print running info if verbose
#
# If -v 3 or higher, print exported variables in order that they were exported.
#
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: VERSION=$VERSION" 1>&2
echo "$0: debug[3]: NAME=$NAME" 1>&2
echo "$0: debug[3]: V_FLAG=$V_FLAG" 1>&2
echo "$0: debug[3]: GIT_TOOL=$GIT_TOOL" 1>&2
echo "$0: debug[3]: TOPDIR=$TOPDIR" 1>&2
echo "$0: debug[3]: DOCROOT_SLASH=$DOCROOT_SLASH" 1>&2
echo "$0: debug[3]: TAGLINE=$TAGLINE" 1>&2
echo "$0: debug[3]: REPO_TOP_URL=$REPO_TOP_URL" 1>&2
echo "$0: debug[3]: REPO_URL=$REPO_URL" 1>&2
echo "$0: debug[3]: SITE_URL=$SITE_URL" 1>&2
echo "$0: debug[3]: NOOP=$NOOP" 1>&2
echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2
echo "$0: debug[3]: EXIT_CODE=$EXIT_CODE" 1>&2
echo "$0: debug[3]: REPO_NAME=$REPO_NAME" 1>&2
echo "$0: debug[3]: CD_FAILED=$CD_FAILED" 1>&2
echo "$0: debug[3]: AUTHOR_PATH=$AUTHOR_PATH" 1>&2
echo "$0: debug[3]: AUTHOR_DIR=$AUTHOR_DIR" 1>&2
fi
# determine the number of author_handle.json files
#
AUTHOR_COUNT=$(find "$AUTHOR_DIR" -type f -name '*.json' -print | wc -l)
export AUTHOR_COUNT
if [[ -z $AUTHOR_COUNT || $AUTHOR_COUNT -le 0 ]]; then
echo "$0: ERROR: found no author_handle.json files under: $AUTHOR_DIR" 1>&2
exit 10
fi
if [[ $AUTHOR_COUNT -le 1 ]]; then
echo "$0: ERROR: expected more than 1 author_handle.json files under: $AUTHOR_DIR" 1>&2
exit 11
fi
# -N stops early before any processing is performed
#
if [[ -n $DO_NOT_PROCESS ]]; then
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: arguments parsed, -N given, exiting 0" 1>&2
fi
exit 0
fi
# create a list of all author/author_handle files
#
export TMP_AUTHOR_HANDLE_LIST=".tmp.$NAME.AUTHOR_HANDLE_LIST.$$.tmp"
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: list pf author_handle JSON files: $TMP_AUTHOR_HANDLE_LIST" 1>&2
fi
trap 'rm -f $TMP_AUTHOR_HANDLE_LIST; exit' 0 1 2 3 15
rm -f "$TMP_AUTHOR_HANDLE_LIST"
if [[ -e $TMP_AUTHOR_HANDLE_LIST ]]; then
echo "$0: ERROR: cannot remove list pf author_handle JSON files: $TMP_AUTHOR_HANDLE_LIST" 1>&2
exit 12
fi
find "$AUTHOR_DIR" -type f -name '*.json' -print | LC_ALL=C sort -t / -d > "$TMP_AUTHOR_HANDLE_LIST"
if [[ ! -e $TMP_AUTHOR_HANDLE_LIST ]]; then
echo "$0: ERROR: cannot create list pf author_handle JSON files: $TMP_AUTHOR_HANDLE_LIST" 1>&2
exit 13
fi
# combine all author_handle.json files
#
export SED_PART1='s;^(\s*)("sort_word" : )"(.*)",$;\1"sort_word.'
export SED_PART2='" : [\n\1 {\n\1 \2"\3 '
export SED_PART3='"\n\1 }\n\1],;'
export count=0
echo "{"
for AUTHOR_HANDLE_JSON in $(< "$TMP_AUTHOR_HANDLE_LIST"); do
# count file
#
((++count))
# case: not the last file
#
# Remove 1st and last lines, and add , (comma) to the end of the last line.
#
# We also replace "winning_entry_set" with "winning_entry_set.FILENO" where
# FILENO is the file number.
#
# We also convert where FILENO is the file number and FILENAME is the filename:
#
# "sort_word" : "data",
#
# into:
#
# "sort_word.FILENO" : [
# {
# "sort_word" : "data FILENAME"
# }
# ],
#
if [[ $count -lt $AUTHOR_COUNT ]]; then
tail -n +2 "$AUTHOR_HANDLE_JSON" |
head -n -1 |
sed -E -e 's/"winning_entry_set"/"winning_entry_set.'"$count"'"/' \
-e "$SED_PART1$count$SED_PART2$AUTHOR_HANDLE_JSON$SED_PART3" \
-e '$s/$/,/'
# case: last file
#
else
tail -n +2 "$AUTHOR_HANDLE_JSON" |
head -n -1 |
sed -E -e 's/"winning_entry_set"/"winning_entry_set.'"$count"'"/' \
-e "$SED_PART1$count$SED_PART2$AUTHOR_HANDLE_JSON$SED_PART3"
echo '}'
fi
done
# All Done!!! All Done!!! -- Jessica Noll, Age 2
#
if [[ $EXIT_CODE -ne 0 ]]; then
echo "$0: Warning: about to exit non-zero: $EXIT_CODE" 1>&2
elif [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: about to exit $EXIT_CODE" 1>&2
fi
exit "$EXIT_CODE"
bin/csv2entry.sh
+114 -155
View File
@@ -142,7 +142,7 @@ shopt -s lastpipe # explicitly run the last command of the pipe line in the curr
# set variables referenced in the usage message
#
export VERSION="1.2.3 2024-08-30"
export VERSION="1.3 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
@@ -169,8 +169,6 @@ export NOOP=
export DO_NOT_PROCESS=
export EXIT_CODE="0"
#
JPARSE_TOOL=$(type -P jparse)
export JPARSE_TOOL
export AUTHOR_WINS_CSV="author_wins.csv"
export MANIFEST_CSV="manifest.csv"
export YEAR_PRIZE_CSV="year_prize.csv"
@@ -289,7 +287,7 @@ function canonicalize_csv
# usage
#
export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-n] [-N] [-j jparse]
export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-n] [-N]
[author_wins.csv [manifest.csv [year_prize.csv]]]
-h print help message and exit
@@ -301,8 +299,6 @@ export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-n] [-N] [-j jparse]
-n go thru the actions, but do not update any files (def: do the action)
-N do not process anything, just parse arguments (def: process something)
-j jparse path to the jparse tool (def: $JPARSE_TOOL)
author_wins.csv where to form author_wins.csv (def: $AUTHOR_WINS_CSV)
manifest.csv where to form manifest.csv (def: $MANIFEST_CSV)
year_prize.csv where to form year_prize.csv (def: $YEAR_PRIZE_CSV)
@@ -325,7 +321,7 @@ $NAME version: $VERSION"
# parse command line
#
while getopts :hv:Vd:D:nNj: flag; do
while getopts :hv:Vd:D:nN flag; do
case "$flag" in
h) echo "$USAGE" 1>&2
exit 2
@@ -341,8 +337,6 @@ while getopts :hv:Vd:D:nNj: flag; do
;;
N) DO_NOT_PROCESS="-N"
;;
j) JPARSE_TOOL="$OPTARG"
;;
\?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2
echo 1>&2
print_usage 1>&2
@@ -490,7 +484,7 @@ export BIN_DIR="bin"
# verify that the bin/jval-wrapper.sh tool is executable
#
JVAL_WRAPPER="$BIN_DIR/jval-wrapper.sh"
export JVAL_WRAPPER="$BIN_DIR/jval-wrapper.sh"
if [[ ! -e $JVAL_WRAPPER ]]; then
echo "$0: ERROR: bin/jval-wrapper.sh does not exist: $JVAL_WRAPPER" 1>&2
exit 5
@@ -505,18 +499,36 @@ if [[ ! -x $JVAL_WRAPPER ]]; then
fi
# verify jparse tool
# verify that the bin/jval-wrapper.sh tool is executable
#
if [[ -z "$JPARSE_TOOL" ]]; then
echo "$0: ERROR: cannot find an executable jparse tool" 1>&2
export UNICODE_CHK="$BIN_DIR/unicode-chk.sh"
if [[ ! -e $UNICODE_CHK ]]; then
echo "$0: ERROR: bin/unicode-chk.sh does not exist: $UNICODE_CHK" 1>&2
exit 5
fi
if [[ ! -f "$JPARSE_TOOL" ]]; then
echo "$0: ERROR: jparse is not a file: $JPARSE_TOOL" 1>&2
if [[ ! -f $UNICODE_CHK ]]; then
echo "$0: ERROR: bin/unicode-chk.sh is not a regular file: $UNICODE_CHK" 1>&2
exit 5
fi
if [[ ! -x "$JPARSE_TOOL" ]]; then
echo "$0: ERROR: jparse is not an executable file: $JPARSE_TOOL" 1>&2
if [[ ! -x $UNICODE_CHK ]]; then
echo "$0: ERROR: bin/unicode-chk.sh is not an executable file: $UNICODE_CHK" 1>&2
exit 5
fi
# verify that the bin/combine_author_handle.sh tool is executable
#
export COMBINE_AUTHOR="$BIN_DIR/combine_author_handle.sh"
if [[ ! -e $COMBINE_AUTHOR ]]; then
echo "$0: ERROR: bin/unicode-chk.sh does not exist: $COMBINE_AUTHOR" 1>&2
exit 5
fi
if [[ ! -f $COMBINE_AUTHOR ]]; then
echo "$0: ERROR: bin/unicode-chk.sh is not a regular file: $COMBINE_AUTHOR" 1>&2
exit 5
fi
if [[ ! -x $COMBINE_AUTHOR ]]; then
echo "$0: ERROR: bin/unicode-chk.sh is not an executable file: $COMBINE_AUTHOR" 1>&2
exit 5
fi
@@ -534,7 +546,6 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: NOOP=$NOOP" 1>&2
echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2
echo "$0: debug[3]: EXIT_CODE=$EXIT_CODE" 1>&2
echo "$0: debug[3]: JPARSE_TOOL=$JPARSE_TOOL" 1>&2
echo "$0: debug[3]: AUTHOR_WINS_CSV=$AUTHOR_WINS_CSV" 1>&2
echo "$0: debug[3]: MANIFEST_CSV=$MANIFEST_CSV" 1>&2
echo "$0: debug[3]: YEAR_PRIZE_CSV=$YEAR_PRIZE_CSV" 1>&2
@@ -546,6 +557,8 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: BIN_PATH=$BIN_DIR" 1>&2
echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2
echo "$0: debug[3]: JVAL_WRAPPER=$JVAL_WRAPPER" 1>&2
echo "$0: debug[3]: UNICODE_CHK=$UNICODE_CHK" 1>&2
echo "$0: debug[3]: COMBINE_AUTHOR=$COMBINE_AUTHOR" 1>&2
echo "$0: debug[3]: DOT_ENTRY_JSON_BASENAME=$DOT_ENTRY_JSON_BASENAME" 1>&2
echo "$0: debug[3]: ENTRY_JSON_FORMAT_VERSION=$ENTRY_JSON_FORMAT_VERSION" 1>&2
echo "$0: debug[3]: NO_COMMENT=$NO_COMMENT" 1>&2
@@ -705,26 +718,46 @@ if [[ ! -e $TMP_EXIT_CODE ]]; then
fi
# create a temporary author_handle inventory file
# create a temporary sorted list of author_handles from author/author_handle.json files
#
export TMP_AUTHOR_HANDLE_INVENTORY=".tmp.$NAME.AUTHOR_HANDLE_INVENTORY.$$.tmp"
export TMP_AUTHOR_HANDLE_FROM_FILES=".tmp.$NAME.AUTHOR_HANDLE_FROM_FILES.$$.tmp"
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: temporary author_handle inventory file: $TMP_AUTHOR_HANDLE_INVENTORY" 1>&2
echo "$0: debug[3]: temporary sorted list of author_handles: $TMP_AUTHOR_HANDLE_FROM_FILES" 1>&2
fi
trap 'rm -f $TMP_AUTHOR_WINS_CSV $TMP_MANIFEST_CSV $TMP_YEAR_PRIZE_CSV $TMP_YYYY_DIR_INV $TMP_YYYY_DIR_INV \
$TMP_2ND_YYYY_DIR_INV $TMP_ENTRY_JSON $TMP_EXIT_CODE $TMP_AUTHOR_HANDLE_INVENTORY; exit' 0 1 2 3 15
rm -f "$TMP_AUTHOR_HANDLE_INVENTORY"
if [[ -e $TMP_AUTHOR_HANDLE_INVENTORY ]]; then
echo "$0: ERROR: cannot remove temporary author_handle inventory file: $TMP_AUTHOR_HANDLE_INVENTORY" 1>&2
$TMP_2ND_YYYY_DIR_INV $TMP_ENTRY_JSON $TMP_EXIT_CODE $TMP_AUTHOR_HANDLE_FROM_FILES; exit' 0 1 2 3 15
rm -f "$TMP_AUTHOR_HANDLE_FROM_FILES"
if [[ -e $TMP_AUTHOR_HANDLE_FROM_FILES ]]; then
echo "$0: ERROR: cannot remove temporary sorted list of author_handles: $TMP_AUTHOR_HANDLE_FROM_FILES" 1>&2
exit 24
fi
: > "$TMP_AUTHOR_HANDLE_INVENTORY"
if [[ ! -e $TMP_AUTHOR_HANDLE_INVENTORY ]]; then
echo "$0: ERROR: cannot create temporary author_handle inventory file: $TMP_AUTHOR_HANDLE_INVENTORY" 1>&2
: > "$TMP_AUTHOR_HANDLE_FROM_FILES"
if [[ ! -e $TMP_AUTHOR_HANDLE_FROM_FILES ]]; then
echo "$0: ERROR: cannot create temporary sorted list of author_handles: $TMP_AUTHOR_HANDLE_FROM_FILES" 1>&2
exit 25
fi
# create a temporary of author_handles from CSV file
#
export TMP_AUTHOR_HANDLE_FROM_CSV=".tmp.$NAME.AUTHOR_HANDLE_FROM_CSV.$$.tmp"
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: temporary sorted list of author_handles: $TMP_AUTHOR_HANDLE_FROM_CSV" 1>&2
fi
trap 'rm -f $TMP_AUTHOR_WINS_CSV $TMP_MANIFEST_CSV $TMP_YEAR_PRIZE_CSV $TMP_YYYY_DIR_INV $TMP_YYYY_DIR_INV \
$TMP_2ND_YYYY_DIR_INV $TMP_ENTRY_JSON $TMP_EXIT_CODE $TMP_AUTHOR_HANDLE_FROM_CSV; exit' 0 1 2 3 15
rm -f "$TMP_AUTHOR_HANDLE_FROM_CSV"
if [[ -e $TMP_AUTHOR_HANDLE_FROM_CSV ]]; then
echo "$0: ERROR: cannot remove temporary sorted list of author_handles: $TMP_AUTHOR_HANDLE_FROM_CSV" 1>&2
exit 26
fi
: > "$TMP_AUTHOR_HANDLE_FROM_CSV"
if [[ ! -e $TMP_AUTHOR_HANDLE_FROM_CSV ]]; then
echo "$0: ERROR: cannot create temporary sorted list of author_handles: $TMP_AUTHOR_HANDLE_FROM_CSV" 1>&2
exit 26
fi
# canonicalize author_wins.csv into temporary author_wins.csv file
#
if [[ -z $NOOP ]]; then
@@ -812,142 +845,70 @@ elif [[ $V_FLAG -ge 3 ]]; then
fi
# scan author_wins.csv file for unknown author handles and
# author handles w/o author/author_handle.json file and
# author/author_handle.json file that do not reference the entry id
# form sorted list of author handles from author/author_handle.json files
#
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: about to: scan for unknown author handles" 1>&2
echo "$0: debug[1]: forming list of author handles from author/author_handle.json files" 1>&2
fi
PATTERN='$..entry_id'
"$COMBINE_AUTHOR" | "$JVAL_WRAPPER" -b -q - "$PATTERN" | LC_ALL=C sort -t _ -d -u > "$TMP_AUTHOR_HANDLE_FROM_FILES"
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] ]]; then
echo "$0: ERROR: $COMBINE_AUTHOR | $JVAL_WRAPPER -b -q '$PATTERN' | sort ... failed", \
"error codes: ${status_codes[*]}" 1>&2
EXIT_CODE="9" # exit 9
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
# NOTE: We do NOT want to stop processing via continue - proceeding instead
fi
if [[ ! -s $TMP_AUTHOR_HANDLE_FROM_FILES ]]; then
echo "$0: ERROR: unable to form sorted list of author handles from author/author_handle.json files" 1>&2
EXIT_CODE="9" # exit 9
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
# NOTE: We do NOT want to stop processing via continue - proceeding instead
fi
export line=0
sed -e 's/,/ /g' "$AUTHOR_WINS_CSV" | while read -r AUTHOR_HANDLE ENTRY_ID_SET; do
# skip comment lines
#
((++line))
if [[ $AUTHOR_HANDLE =~ '#' ]]; then
continue;
fi
# look for unknown author handles
#
export AUTHOR_HANDLE_JSON="$AUTHOR_DIR/$AUTHOR_HANDLE.json"
if [[ ! -f $AUTHOR_DIR/$AUTHOR_HANDLE.json ]]; then
echo "$0: ERROR: entry_id $ENTRY_ID refers to an unknown author_handle: $AUTHOR_HANDLE" 1>&2
echo "$0: Warning: no such author_handle file: $AUTHOR_HANDLE_JSON" 1>&2
echo "$0: Warning: line: $line for: $AUTHOR_WINS_CSV flagging unknown author_handle as an error: continuing" 1>&2
EXIT_CODE="9" # exit 9
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
# NOTE: We do NOT want to stop processing via continue - proceeding instead
# look for lines with no entry ids
#
elif [[ -z $ENTRY_ID_SET ]]; then
echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV as no entry ids on the line" 1>&2
EXIT_CODE="8" # exit 8
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
# NOTE: We do NOT want to stop processing via continue - proceeding instead
# look for unknown entry_id's
#
else
# verify the JSON of author/author_handle.json
#
if [[ $V_FLAG -ge 7 ]]; then
echo "$0: debug[7]: about to run: $JPARSE_TOOL -q -- $AUTHOR_HANDLE_JSON" 1>&2
fi
echo "$AUTHOR_HANDLE_JSON" >> "$TMP_AUTHOR_HANDLE_INVENTORY"
if "$JPARSE_TOOL" -q -- "$AUTHOR_HANDLE_JSON"; then
# case: author/author_handle.json file is valid JSON
#
if [[ $V_FLAG -ge 9 ]]; then
echo "$0: debug[9]: valid JSON for: $AUTHOR_HANDLE_JSON" 1>&2
fi
# check each entry id on the line
#
echo "$ENTRY_ID_SET" | sed -e 's/ /\n/g' | while read -r ENTRY_ID; do
# convert ENTRY_ID into YYYY_DIR
#
export YEAR_DIR=${ENTRY_ID%%_*}
export ENTRY_DIR=${ENTRY_ID#*_}
export YYYY_DIR="${YEAR_DIR}/${ENTRY_DIR}"
# verify that the entry exists
#
if [[ ! -d $YYYY_DIR ]]; then
# case: entry does not exist
#
echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV" \
"refers to unknown entry id: $ENTRY_ID" 1>&2
EXIT_CODE="7" # exit 7
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
# NOTE: We do NOT want to stop processing via continue - proceeding instead
# verify that the author/author_handle.json file refers to the entry id
#
else
# check if the author/author_handle.json file refers to the entry id
#
FOUND=$("$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON" '$..entry_id')
if [[ -z $FOUND ]]; then
echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV" \
"entry id: $ENTRY_ID not in winning_entry_set of: $AUTHOR_HANDLE_JSON" 1>&2
EXIT_CODE="9" # exit 9
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
# NOTE: We do NOT want to stop processing via continue - proceeding instead
fi
fi
done
# case: author/author_handle.json is not valid JSON
#
else
echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV" \
"is not valid JSON for: $AUTHOR_HANDLE_JSON" 1>&2
EXIT_CODE="1" # exit 1
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
# NOTE: We do NOT want to stop processing via continue - proceeding instead
fi
fi
done
# form sorted list of author handles from the author_wins.csv file
#
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: forming list of author handles from the author_wins.csv file" 1>&2
fi
sed -e '/^#/d' -e 's/^[^,]*,//' -e 's/,/\n/g' "$AUTHOR_WINS_CSV" | LC_ALL=C sort -t _ -d -u > "$TMP_AUTHOR_HANDLE_FROM_CSV"
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] ]]; then
echo "$0: ERROR: sed -e ... | sort ... failed", \
"error codes: ${status_codes[*]}" 1>&2
EXIT_CODE="9" # exit 9
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
# NOTE: We do NOT want to stop processing via continue - proceeding instead
fi
if [[ ! -s $TMP_AUTHOR_HANDLE_FROM_CSV ]]; then
echo "$0: ERROR: unable to form sorted list of author handles from the author_wins.csv file" 1>&2
EXIT_CODE="9" # exit 9
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
# NOTE: We do NOT want to stop processing via continue - proceeding instead
fi
# verify that the author directory contains only author_handle.json files referenced by author_wins.csv
#
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to: sort $TMP_AUTHOR_HANDLE_INVENTORY -o $TMP_AUTHOR_HANDLE_INVENTORY" 1>&2
fi
LC_ALL=C sort "$TMP_AUTHOR_HANDLE_INVENTORY" -o "$TMP_AUTHOR_HANDLE_INVENTORY"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: LC_ALL=C sort $TMP_AUTHOR_HANDLE_INVENTORY -o $TMP_AUTHOR_HANDLE_INVENTORY failed," \
"error code: $status" 1>&2
exit 9
fi
AUTHOR_DIFF=$(find "$AUTHOR_DIR" -mindepth 1 -maxdepth 1 -type f -name '*.json' 2>&1 |
LC_ALL=C sort |
diff -u - "$TMP_AUTHOR_HANDLE_INVENTORY")
AUTHOR_DIFF=$(diff "$TMP_AUTHOR_HANDLE_FROM_FILES" "$TMP_AUTHOR_HANDLE_FROM_CSV" 2>/dev/null)
export AUTHOR_DIFF
if [[ -n $AUTHOR_DIFF ]]; then
echo "$0: ERROR: author handles in author_wins.csv: $AUTHOR_WINS_CSV differs from author directory: $AUTHOR_DIR" 1>&2
echo "$0: Warning: author directory differences start below" 1>&2
echo "$AUTHOR_DIFF" 1>&2
echo "$0: Warning: author directory differences end above" 1>&2
EXIT_CODE="9" # exit 1
EXIT_CODE="9" # exit 9
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
# NOTE: We do NOT want to stop processing via continue - proceeding instead
elif [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: all author handles are accounted for" 1>&2
fi
@@ -1478,17 +1439,15 @@ sed -e '/^#/d' -e 's/,/ /g' "$MANIFEST_CSV" |
printf "}\n"
} >> "$TMP_ENTRY_JSON"
# verify JSON of temporary .entry.json file
# check the JSON file
#
if [[ $V_FLAG -ge 7 ]]; then
echo "$0: debug[7]: about to run: $JPARSE_TOOL -q -- $TMP_ENTRY_JSON" 1>&2
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to check: $PREV_ENTRY_JSON" 1>&2
fi
if "$JPARSE_TOOL" -q -- "$TMP_ENTRY_JSON"; then
if [[ $V_FLAG -ge 9 ]]; then
echo "$0: debug[9]: valid JSON for $PREV_ENTRY_DIR: $TMP_ENTRY_JSON" 1>&2
fi
else
echo "$0: ERROR: .entry.json is not valid JSON for $PREV_ENTRY_DIR: $TMP_ENTRY_JSON" 1>&2
"$UNICODE_CHK" "$TMP_ENTRY_JSON"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: JSON file failed check: $TMP_ENTRY_JSON" 1>&2
exit 1
fi
bin/cvt-submission.sh
+159 -100
View File
@@ -94,7 +94,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an
# set variables referenced in the usage message
#
export VERSION="1.1 2024-08-31"
export VERSION="1.2 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
@@ -155,8 +155,6 @@ export AUTHOR_JSON_FORMAT_VERSION="1.1 2024-02-11"
export NOOP=
export DO_NOT_PROCESS=
#
unset TAR_FILE_SET
declare -ag TAR_FILE_SET
export TARBALL_DIR="/var/tmp"
#
export X_3="X""X""X"
@@ -1003,29 +1001,6 @@ export TEMPLATE_README_MD_HEAD="$TEMPLATE_ENTRY_DIR/README.md.head"
export TEMPLATE_README_MD_TAIL="$TEMPLATE_ENTRY_DIR/README.md.tail"
# initialize the list of files that we might tar
#
for FILE in "$INFO_JSON" "$AUTH_JSON" "$REMARKS_MD" "$README_MD" "$INDEX_HTML" "$TAR_BZ2"; do
if [[ -f $FILE ]]; then
TAR_FILE_SET+=("$FILE")
fi
done
#
# add each author/author_handle.json file referenced by .auth.json file that exists
#
export PATTERN
PATTERN="\$authors..author_handle"
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: about to run: $JVAL_WRAPPER -b -- $AUTH_JSON $PATTERN" 1>&2
fi
for AUTHOR_HANDLE in $("$JVAL_WRAPPER" -b -- "$AUTH_JSON" "$PATTERN"); do
AUTHOR_HANDLE_JSON="author/$AUTHOR_HANDLE.json"
if [[ -f $AUTHOR_HANDLE_JSON ]]; then
TAR_FILE_SET+=("$AUTHOR_HANDLE_JSON")
fi
done
# determine TARBALL filename
#
NOW=$(date '+%Y%m%d.%H%M%S')
@@ -1084,9 +1059,6 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: TEMPLATE_GITIGNORE=$TEMPLATE_GITIGNORE" 1>&2
echo "$0: debug[3]: TEMPLATE_README_MD_HEAD=$TEMPLATE_README_MD_HEAD" 1>&2
echo "$0: debug[3]: TEMPLATE_README_MD_TAIL=$TEMPLATE_README_MD_TAIL" 1>&2
for index in "${!TAR_FILE_SET[@]}"; do
echo "$0: debug[3]: TAR_FILE_SET[$index]=${TAR_FILE_SET[$index]}" 1>&2
done
echo "$0: debug[3]: TARBALL_DIR=$TARBALL_DIR" 1>&2
echo "$0: debug[3]: NOW=$NOW" 1>&2
echo "$0: debug[3]: TARBALL=$TARBALL" 1>&2
@@ -1206,6 +1178,56 @@ if [[ ! -s $AUTH_JSON ]]; then
fi
# initialize the list of files that we might tar
#
unset TAR_FILE_SET
declare -ag TAR_FILE_SET
#
for FILE in "$INFO_JSON" "$AUTH_JSON" "$REMARKS_MD" "$README_MD" "$INDEX_HTML" "$TAR_BZ2"; do
if [[ -f $FILE ]]; then
TAR_FILE_SET+=("$FILE")
fi
done
#
# add each author/author_handle.json file referenced by .auth.json file that exists
#
unset AUTHOR_HANDLE_SET
declare -ag AUTHOR_HANDLE_SET
#
export PATTERN
PATTERN='$..author_handle'
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $AUTH_JSON '$PATTERN'" 1>&2
fi
export FOUND_AUTHOR_HANDLES
FOUND_AUTHOR_HANDLES=$("$JVAL_WRAPPER" -b -q -- "$AUTH_JSON" "$PATTERN")
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JVAL_WRAPPER -b -q -- $AUTH_JSON '$PATTERN' failed," \
"error code: $status" 1>&2
exit 10
fi
if [[ -z $FOUND_AUTHOR_HANDLES ]]; then
echo "$0: ERROR: found no author_handle's in: $AUTH_JSON" 1>&2
exit 11
fi
for AUTHOR_HANDLE in $FOUND_AUTHOR_HANDLES; do
# add referenced by .auth.json if it exists
AUTHOR_HANDLE_JSON="author/$AUTHOR_HANDLE.json"
if [[ -f $AUTHOR_HANDLE_JSON ]]; then
TAR_FILE_SET+=("$AUTHOR_HANDLE_JSON")
fi
# remember the author_handle for later even if it doesn't exist
AUTHOR_HANDLE_SET+=("$AUTHOR_HANDLE")
done
#
if [[ $V_FLAG -ge 3 ]]; then
for index in "${!TAR_FILE_SET[@]}"; do
echo "$0: debug[3]: TAR_FILE_SET[$index]=${TAR_FILE_SET[$index]}" 1>&2
done
fi
# perform the semantic check on .info.json and .auth.json
#
if [[ $V_FLAG -ge 1 ]]; then
@@ -1213,7 +1235,7 @@ if [[ $V_FLAG -ge 1 ]]; then
"$CHKENTRY_TOOL" -v 1 -- "$INFO_JSON" "$AUTH_JSON"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: $CHKENTRY_TOOL -v 1 -- $INFO_JSON $AUTH_JSON failed," \
echo "$0: ERROR: $CHKENTRY_TOOL -v 1 -- $INFO_JSON $AUTH_JSON failed," \
"error code: $status" 1>&2
exit 7
fi
@@ -1252,7 +1274,7 @@ if [[ -z $NOOP ]]; then
echo "$0: ERROR:" \
"TZ=UTC LC_ALL=C $GTAR_TOOL --sort=name" \
"--totals -jcvf $TARBALL ${TAR_FILE_SET[*]} failed, error: $status" 1>&2
exit 10
exit 12
fi
# case: form the tarball silently
@@ -1265,7 +1287,7 @@ if [[ -z $NOOP ]]; then
echo "$0: ERROR:" \
"TZ=UTC LC_ALL=C $GTAR_TOOL --sort=name" \
"-jcf $TARBALL ${TAR_FILE_SET[*]} failed, error: $status" 1>&2
exit 11
exit 13
fi
fi
elif [[ $V_FLAG -ge 3 ]]; then
@@ -1284,12 +1306,12 @@ if [[ -z $NOOP ]]; then
rm -f "$TMP_ENTRY_JSON"
if [[ -e $TMP_ENTRY_JSON ]]; then
echo "$0: ERROR: cannot remove temporary .entry.json file: $TMP_ENTRY_JSON" 1>&2
exit 12
exit 14
fi
:> "$TMP_ENTRY_JSON"
if [[ ! -e $TMP_ENTRY_JSON ]]; then
echo "$0: ERROR: cannot create temporary .entry.json file: $TMP_ENTRY_JSON" 1>&2
exit 13
exit 15
fi
elif [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: because of -n, temporary .entry.json file is not used: $TMP_ENTRY_JSON" 1>&2
@@ -1307,12 +1329,12 @@ if [[ -z $NOOP ]]; then
rm -f "$TMP_AUTHOR_HANDLE_JSON"
if [[ -e $TMP_AUTHOR_HANDLE_JSON ]]; then
echo "$0: ERROR: cannot remove temporary author_handle file: $TMP_AUTHOR_HANDLE_JSON" 1>&2
exit 14
exit 16
fi
:> "$TMP_AUTHOR_HANDLE_JSON"
if [[ ! -e $TMP_AUTHOR_HANDLE_JSON ]]; then
echo "$0: ERROR: cannot create temporary author_handle file: $TMP_AUTHOR_HANDLE_JSON" 1>&2
exit 15
exit 17
fi
elif [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: because of -n, temporary author_handle file is not used: $TMP_AUTHOR_HANDLE_JSON" 1>&2
@@ -1330,12 +1352,12 @@ if [[ -z $NOOP ]]; then
rm -f "$TMP_MANIFEST_CSV"
if [[ -e $TMP_MANIFEST_CSV ]]; then
echo "$0: ERROR: cannot remove temporary manifest csv file: $TMP_MANIFEST_CSV" 1>&2
exit 16
exit 18
fi
:> "$TMP_MANIFEST_CSV"
if [[ ! -e $TMP_MANIFEST_CSV ]]; then
echo "$0: ERROR: cannot create temporary manifest csv file: $TMP_MANIFEST_CSV" 1>&2
exit 17
exit 19
fi
elif [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: because of -n, temporary manifest csv file is not used: $TMP_MANIFEST_CSV" 1>&2
@@ -1395,7 +1417,7 @@ if [[ -z $NOOP ]]; then
# SC2129 (style): Consider using { cmd1; cmd2; } >> file instead of individual redirects
# https://www.shellcheck.net/wiki/SC2129
# shellcheck disable=SC2129
for AUTHOR_HANDLE in $("$JVAL_WRAPPER" -w -b -- "$AUTH_JSON" "\$authors..author_handle"); do
for AUTHOR_HANDLE in "${AUTHOR_HANDLE_SET[@]}"; do
# case: not the first author_handle
#
@@ -1414,12 +1436,22 @@ if [[ -z $NOOP ]]; then
# parse an element value of the this particular author in the authors JSON array
#
PATTERN='$.authors['"$i"']'
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: $JVAL_WRAPPER -u $AUTH_JSON \$.authors[$i]" 1>&2
echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $AUTH_JSON '$PATTERN'" 1>&2
fi
AUTHOR_DATA=$("$JVAL_WRAPPER" -u -- "$AUTH_JSON" "\$.authors[$i]" |
sed -e 's/^\["//' -e 's/^\([^"]*\)"]/\1/')
export AUTHOR_DATA
AUTHOR_DATA=$("$JVAL_WRAPPER" -b -q -- "$AUTH_JSON" "$PATTERN")
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JVAL_WRAPPER -b -q -- $AUTH_JSON '$PATTERN' failed," \
"error code: $status" 1>&2
exit 20
fi
if [[ -z $AUTHOR_DATA ]]; then
echo "$0: ERROR: found nothing for author[$i] in: $AUTH_JSON" 1>&2
exit 21
fi
export AUTHOR_PAST_WINNING_AUTHOR=
export AUTHOR_DEFAULT_HANDLE=
export AUTHOR_AUTHOR_NUMBER=
@@ -1461,20 +1493,20 @@ if [[ -z $NOOP ]]; then
author_handle)
if [[ \"$AUTHOR_HANDLE\" != "$VALUE" ]]; then
echo "$0: ERROR: AUTHOR_HANDLE: \"$AUTHOR_HANDLE\" != author_handle value: $VALUE" 1>&2
exit 18
exit 22
fi
AUTHOR_INFO[author_handle]="$VALUE"
;;
author_number)
if [[ $i != "$VALUE" ]]; then
echo "$0: ERROR: AUTHOR_HANDLE: $AUTHOR_HANDLE i: $i != author_number: $VALUE" 1>&2
exit 19
exit 23
fi
AUTHOR_AUTHOR_NUMBER="$VALUE"
;;
*)
echo "$0: ERROR: AUTHOR_HANDLE: $AUTHOR_HANDLE unknown name: $NAME" 1>&2
exit 20
exit 24
;;
esac
done <<< "$AUTHOR_DATA"
@@ -1484,20 +1516,20 @@ if [[ -z $NOOP ]]; then
for NAME in name location_code email url alt_url mastodon github affiliation author_handle; do
if [[ -z ${AUTHOR_INFO[${NAME}]} ]]; then
echo "$0: ERROR: AUTHOR_HANDLE[$i]: $AUTHOR_HANDLE missing $NAME value from: $AUTH_JSON" 1>&2
exit 21
exit 25
fi
done
if [[ -z $AUTHOR_PAST_WINNING_AUTHOR ]]; then
echo "$0: ERROR: AUTHOR_HANDLE[$i]: $AUTHOR_HANDLE missing past_winning_author value from: $AUTH_JSON" 1>&2
exit 22
exit 26
fi
if [[ -z $AUTHOR_DEFAULT_HANDLE ]]; then
echo "$0: ERROR: AUTHOR_HANDLE[$i]: $AUTHOR_HANDLE missing default_handle value from: $AUTH_JSON" 1>&2
exit 23
exit 27
fi
if [[ -z $AUTHOR_AUTHOR_NUMBER ]]; then
echo "$0: ERROR: AUTHOR_HANDLE[$i]: $AUTHOR_HANDLE missing author_number value from: $AUTH_JSON" 1>&2
exit 24
exit 28
fi
# determine sort_word from the Author's last name
@@ -1506,7 +1538,7 @@ if [[ -z $NOOP ]]; then
export SORT_WORD
if [[ -z $SORT_WORD ]]; then
echo "$0: ERROR: AUTHOR_HANDLE: $AUTHOR_HANDLE unable to determine the sort_word for: ${AUTHOR_INFO[name]}" 1>&2
exit 25
exit 29
fi
case "$SORT_WORD" in
[a-z]*) ;;
@@ -1545,22 +1577,35 @@ if [[ -z $NOOP ]]; then
#
if [[ ! -f $AUTHOR_HANDLE_JSON ]]; then
echo "$0: ERROR: author_handle JSON is not a file: $AUTHOR_HANDLE_JSON" 1>&2
exit 26
exit 30
fi
if [[ ! -r $AUTHOR_HANDLE_JSON ]]; then
echo "$0: ERROR: author_handle JSON is not a readable file: $AUTHOR_HANDLE_JSON" 1>&2
exit 27
exit 31
fi
if [[ ! -s $AUTHOR_HANDLE_JSON ]]; then
echo "$0: ERROR: author_handle JSON is not a non-empty readable file: $AUTHOR_HANDLE_JSON" 1>&2
exit 28
exit 32
fi
# parse author/author_handle.json file other than the winning_entry_set
#
AUTHOR_DATA=$("$JVAL_WRAPPER" -u "$AUTHOR_HANDLE_JSON" |
grep -v winning_entry_set |
sed -e 's/^\["//' -e 's/^\([^"]*\)"]/\1/')
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: $JVAL_WRAPPER -q -- $AUTHOR_HANDLE_JSON | sed -E -e ..." 1>&2
fi
export AUTHOR_DATA
AUTHOR_DATA=$("$JVAL_WRAPPER" -q -- "$AUTHOR_HANDLE_JSON" |
sed -E -e '/^entry_id\t/d')
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] ]]; then
echo "$0: ERROR: $JVAL_WRAPPER -q -- $AUTHOR_HANDLE_JSON | sed -E -e ... failed," \
"error codes: ${status_codes[*]}" 1>&2
exit 33
fi
if [[ -z $AUTHOR_DATA ]]; then
echo "$0: ERROR: found entry_id's in: $AUTH_JSON" 1>&2
exit 34
fi
while read -r NAME VALUE; do
# parse name value got non
@@ -1570,21 +1615,21 @@ if [[ -z $NOOP ]]; then
if [[ $VALUE != \"$NO_COMMENT\" ]]; then
echo "$0: ERROR: author_handle file: $AUTHOR_HANDLE_JSON" \
"invalid no_comment: $VALUE != \"$NO_COMMENT\"" 1>&2
exit 29
exit 35
fi
;;
author_JSON_format_version)
if [[ $VALUE != \"$AUTHOR_JSON_FORMAT_VERSION\" ]]; then
echo "$0: ERROR: author_handle file: $AUTHOR_HANDLE_JSON" \
"invalid author_JSON_format_version: $VALUE != \"$AUTHOR_JSON_FORMAT_VERSION\"" 1>&2
exit 30
exit 36
fi
;;
author_handle)
if [[ $VALUE != "${AUTHOR_INFO[author_handle]}" ]]; then
echo "$0: ERROR: author_handle file: $AUTHOR_HANDLE_JSON" \
"invalid author_handle: $VALUE != ${AUTHOR_INFO[author_handle]}" 1>&2
exit 31
exit 37
fi
;;
full_name)
@@ -1663,17 +1708,31 @@ if [[ -z $NOOP ]]; then
*)
echo "$0: ERROR: author_handle file: $AUTHOR_HANDLE_JSON" \
"unknown $NAME : $VALUE" 1>&2
exit 32
exit 38
;;
esac
done <<< "$AUTHOR_DATA"
# collect the entry_ids from the winning_entry_set and add the new entry_id
#
ENTRY_IDS=$( { "$JVAL_WRAPPER" -w -b -- "$AUTHOR_HANDLE_JSON" "\$..entry_id" |
sed -e 's/^\["//' -e 's/^\([^"]*\)"]/\1/'
PATTERN='$..entry_id'
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $AUTHOR_HANDLE_JSON '$PATTERN'" 1>&2
fi
export ENTRY_IDS
ENTRY_IDS=$( { "$JVAL_WRAPPER" -b -q -- "$AUTHOR_HANDLE_JSON" "$PATTERN";
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JVAL_WRAPPER -b -q -- $AUTHOR_HANDLE_JSON '$PATTERN' failed," \
"error code: $status" 1>&2
exit 39
fi
echo "$ENTRY_ID"
} | LC_ALL=C sort -d -u )
if [[ -z $AUTHOR_DATA ]]; then
echo "$0: ERROR: found no entry_id's in: $AUTHOR_HANDLE_JSON" 1>&2
exit 40
fi
while read -r ENTRY_ID_SET_VALUE; do
ENTRY_ID_SET+=("$ENTRY_ID_SET_VALUE")
done <<< "$ENTRY_IDS"
@@ -1692,7 +1751,7 @@ if [[ -z $NOOP ]]; then
echo "$0: ERROR: write_author_handle_file AUTHOR_INFO ENTRY_ID_SET" \
"$TMP_AUTHOR_HANDLE_JSON $AUTHOR_HANDLE_JSON failed," \
"error code: $status" 1>&2
exit 33
exit 41
fi
# output the author_handle information for .entry.json
@@ -1718,17 +1777,17 @@ if [[ -z $NOOP ]]; then
# leave out the trailing comma.
#
((++count))
PATTERN="\$manifest..c_src"
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: about to run: $JVAL_WRAPPER -b -- $INFO_JSON $PATTERN" 1>&2
PATTERN='$..c_src'
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $INFO_JSON '$PATTERN'" 1>&2
fi
export FILE_PATH
FILE_PATH=$("$JVAL_WRAPPER" -b -- "$INFO_JSON" "$PATTERN")
FILE_PATH=$("$JVAL_WRAPPER" -b -q -- "$INFO_JSON" "$PATTERN")
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JVAL_WRAPPER -b -- $INFO_JSON $PATTERN failed," \
echo "$0: ERROR: $JVAL_WRAPPER -b -q -- $INFO_JSON '$PATTERN' failed," \
"error code: $status" 1>&2
exit 34
exit 42
fi
if [[ -z $FILE_PATH ]]; then
echo "$0: ERROR: cannot determine c_src in manifest from: $INFO_JSON" 1>&2
@@ -1736,11 +1795,11 @@ if [[ -z $NOOP ]]; then
fi
if [[ ! -e $YYYY_DIR/$FILE_PATH ]]; then
echo "$0: ERROR: manifest file for c_src does not exist: $YYYY_DIR/$FILE_PATH" 1>&2
exit 35
exit 43
fi
if [[ ! -f $YYYY_DIR/$FILE_PATH ]]; then
echo "$0: ERROR: manifest file for c_src is not a file: $YYYY_DIR/$FILE_PATH" 1>&2
exit 36
exit 44
fi
manifest_csv "$FILE_PATH" 20 true c true "entry source code" >> "$TMP_MANIFEST_CSV"
export PROG_C="$FILE_PATH"
@@ -1748,29 +1807,29 @@ if [[ -z $NOOP ]]; then
# write manifest csv line for entry Makefile
#
((++count))
PATTERN="\$manifest..Makefile"
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: about to run: $JVAL_WRAPPER -b -- $INFO_JSON $PATTERN" 1>&2
PATTERN='$..Makefile'
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $INFO_JSON '$PATTERN'" 1>&2
fi
export FILE_PATH
FILE_PATH=$("$JVAL_WRAPPER" -b -- "$INFO_JSON" "$PATTERN")
FILE_PATH=$("$JVAL_WRAPPER" -b -q -- "$INFO_JSON" "$PATTERN")
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JVAL_WRAPPER -b -- $INFO_JSON $PATTERN failed," \
echo "$0: ERROR: $JVAL_WRAPPER -b -q -- $INFO_JSON '$PATTERN' failed," \
"error code: $status" 1>&2
exit 37
exit 45
fi
if [[ -z $FILE_PATH ]]; then
echo "$0: ERROR: cannot determine Makefile in manifest from: $INFO_JSON" 1>&2
exit 38
exit 46
fi
if [[ ! -e $YYYY_DIR/$FILE_PATH ]]; then
echo "$0: ERROR: manifest file for Makefile does not exist: $YYYY_DIR/$FILE_PATH" 1>&2
exit 39
exit 47
fi
if [[ ! -f $YYYY_DIR/$FILE_PATH ]]; then
echo "$0: ERROR: manifest file for Makefile is not a file: $YYYY_DIR/$FILE_PATH" 1>&2
exit 40
exit 48
fi
manifest_csv "$FILE_PATH" 30 true makefile true "entry Makefile" >> "$TMP_MANIFEST_CSV"
@@ -1798,7 +1857,7 @@ if [[ -z $NOOP ]]; then
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: cp -f -p -v $YYYY_DIR/$PROG_C $YYYY_DIR/$ORIG_C failed," \
"error code: $status" 1>&2
exit 41
exit 49
fi
else
cp -f -p "$YYYY_DIR/$PROG_C" "$YYYY_DIR/$ORIG_C"
@@ -1806,7 +1865,7 @@ if [[ -z $NOOP ]]; then
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: cp -f -p $YYYY_DIR/$PROG_C $YYYY_DIR/$ORIG_C failed," \
"error code: $status" 1>&2
exit 42
exit 50
fi
fi
fi
@@ -1819,17 +1878,17 @@ if [[ -z $NOOP ]]; then
# collect manifest for extra_file(s)
#
export PATTERN
PATTERN="\$manifest..extra_file"
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: about to run: $JVAL_WRAPPER -b -- $INFO_JSON $PATTERN" 1>&2
PATTERN='$..extra_file'
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $INFO_JSON '$PATTERN' | ,,, sort ..." 1>&2
fi
export EXTRA_FILE_SET
EXTRA_FILE_SET=$("$JVAL_WRAPPER" -b -- "$INFO_JSON" "$PATTERN" | LC_ALL=C sort -d -u)
EXTRA_FILE_SET=$("$JVAL_WRAPPER" -b -q -- "$INFO_JSON" "$PATTERN" | LC_ALL=C sort -d -u)
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] ]]; then
echo "$0: ERROR: $JVAL_WRAPPER -b -- $INFO_JSON $PATTERN failed," \
echo "$0: ERROR: $JVAL_WRAPPER -b -q -- $INFO_JSON '$PATTERN' | ... sort ... failed," \
"error code: ${status_codes[*]}" 1>&2
exit 43
exit 51
fi
# perform some validation checks on the extra_file set
@@ -1841,7 +1900,7 @@ if [[ -z $NOOP ]]; then
case "$EXTRA_FILE" in
prog.c|Makefile|prog.orig.c|index.html|README.md|remarks.md|\.*)
echo "$0: ERROR: invalid extra_file filename: $EXTRA_FILE" 1>&2
exit 44
exit 52
;;
*) ;;
esac
@@ -1850,11 +1909,11 @@ if [[ -z $NOOP ]]; then
#
if [[ ! -e $YYYY_DIR/$EXTRA_FILE ]]; then
echo "$0: ERROR: manifest file for extra_file does not exist: $YYYY_DIR/$EXTRA_FILE" 1>&2
exit 45
exit 53
fi
if [[ ! -f $YYYY_DIR/$EXTRA_FILE ]]; then
echo "$0: ERROR: manifest file for extra_file is not a file: $YYYY_DIR/$EXTRA_FILE" 1>&2
exit 46
exit 54
fi
done
@@ -2007,7 +2066,7 @@ if [[ -z $NOOP ]]; then
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: cp -f -p -v $TEMPLATE_GITIGNORE $DOT_GITIGNORE failed," \
"error code: $status" 1>&2
exit 47
exit 55
fi
else
cp -f -p "$TEMPLATE_GITIGNORE" "$DOT_GITIGNORE"
@@ -2015,7 +2074,7 @@ if [[ -z $NOOP ]]; then
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: cp -f -p $TEMPLATE_GITIGNORE $DOT_GITIGNORE failed," \
"error code: $status" 1>&2
exit 48
exit 56
fi
fi
fi
@@ -2075,7 +2134,7 @@ if [[ -z $NOOP ]]; then
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: LC_ALL=C sort -t, -k2n -k1,1 -k3,6 $TMP_MANIFEST_CSV -o $TMP_MANIFEST_CSV failed," \
"error code: $status" 1>&2
exit 49
exit 57
elif [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: sorted temporary manifest csv file: $TMP_MANIFEST_CSV" 1>&2
fi
@@ -2118,7 +2177,7 @@ if [[ -z $NOOP ]]; then
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JPARSE_TOOL -q -- $TMP_ENTRY_JSON filed," \
"error code: $status" 1>&2
exit 50
exit 58
elif [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: valid JSON: $TMP_ENTRY_JSON" 1>&2
fi
@@ -2150,13 +2209,13 @@ if [[ -z $NOOP ]]; then
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: mv -f -- $TMP_ENTRY_JSON $ENTRY_JSON filed," \
"error code: $status" 1>&2
exit 51
exit 59
elif [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: updated .entry.json: $ENTRY_JSON" 1>&2
fi
if [[ ! -s $ENTRY_JSON ]]; then
echo "$0: ERROR: not a non-empty .entry.json: $ENTRY_JSON" 1>&2
exit 52
exit 60
fi
fi
@@ -2191,7 +2250,7 @@ for EXTRA_FILE in $EXTRA_FILE_SET; do
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $OTHERMD2HTML_SH -v $V_FLAG -- $YYYY_DIR/$EXTRA_FILE filed," \
"error code: $status" 1>&2
exit 53
exit 61
fi
elif [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: -n disabled forming HTML: $YYYY_DIR/$HTML_FILE" 1>&2
@@ -2219,7 +2278,7 @@ if [[ -z $NOOP ]]; then
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $MD2HTML_SH -v $V_FLAG -- $README_MD $INDEX_HTML filed," \
"error code: $status" 1>&2
exit 54
exit 62
fi
fi
@@ -2240,7 +2299,7 @@ status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: rm -f -v -- $INFO_JSON $AUTH_JSON $REMARKS_MD filed," \
"error code: $status" 1>&2
exit 55
exit 63
fi
bin/entry2csv.sh
+67 -87
View File
@@ -118,7 +118,7 @@ shopt -s lastpipe # explicitly run the last command of the pipe line in the curr
# set variables referenced in the usage message
#
export VERSION="1.2.1 2024-08-17"
export VERSION="1.3 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
@@ -139,8 +139,6 @@ export NOOP=
export DO_NOT_PROCESS=
export EXIT_CODE="0"
#
JPARSE_TOOL=$(type -P jparse)
export JPARSE_TOOL
export AUTHOR_WINS_CSV="author_wins.csv"
export MANIFEST_CSV="manifest.csv"
export YEAR_PRIZE_CSV="year_prize.csv"
@@ -161,6 +159,7 @@ export AUTHOR_DIR="author"
function output_award
{
local ENTRY_JSON_PATH; # the .entry.json path
local PATTERN; # XPath for JSON pattern
# parse args
#
@@ -184,7 +183,17 @@ function output_award
# obtain the award string
#
"$JVAL_WRAPPER" -w -b "$ENTRY_JSON_PATH" '$..award'
PATTERN='$..award'
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN'" 1>&2
fi
"$JVAL_WRAPPER" -b -q "$ENTRY_JSON_PATH" "$PATTERN" | sed -f "$UNICODE_FIX_SED"
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] ]]; then
echo "$0: ERROR: in output_award: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN' failed," \
"error codes: ${status_codes[*]}" 1>&2
return 5
fi
return 0
}
@@ -203,6 +212,7 @@ function output_award
function output_author_handles
{
local ENTRY_JSON_PATH; # the .entry.json path
local PATTERN; # XPath for JSON pattern
# parse args
#
@@ -226,14 +236,24 @@ function output_author_handles
# extract author handles from .entry.json
#
"$JVAL_WRAPPER" -w -b "$ENTRY_JSON_PATH" '$..author_handle'
PATTERN='$..author_handle'
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN'" 1>&2
fi
"$JVAL_WRAPPER" -b -q "$ENTRY_JSON_PATH" "$PATTERN"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: in output_award: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN' failed," \
"error code: $status" 1>&2
return 5
fi
return 0
}
# usage
#
export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-n] [-N] [-j jparse]
export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-n] [-N]
[author_wins.csv [manifest.csv [year_prize.csv]]]
-h print help message and exit
@@ -245,8 +265,6 @@ export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-n] [-N] [-j jparse]
-n go thru the actions, but do not update any files (def: do the action)
-N do not process anything, just parse arguments (def: process something)
-j jparse path to the jparse tool (def: $JPARSE_TOOL)
author_wins.csv where to form author_wins.csv (def: $AUTHOR_WINS_CSV)
manifest.csv where to form manifest.csv (def: $MANIFEST_CSV)
year_prize.csv where to form year_prize.csv (def: $YEAR_PRIZE_CSV)
@@ -269,7 +287,7 @@ $NAME version: $VERSION"
# parse command line
#
while getopts :hv:Vd:D:nNj: flag; do
while getopts :hv:Vd:D:nN flag; do
case "$flag" in
h) echo "$USAGE" 1>&2
exit 2
@@ -285,8 +303,6 @@ while getopts :hv:Vd:D:nNj: flag; do
;;
N) DO_NOT_PROCESS="-N"
;;
j) JPARSE_TOOL="$OPTARG"
;;
\?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2
echo 1>&2
print_usage 1>&2
@@ -471,30 +487,31 @@ fi
export MANIFEST_ENTRY_CSV_ENTRY_AWK="$BIN_DIR/manifest.csv.entry.awk"
if [[ ! -e $MANIFEST_ENTRY_CSV_ENTRY_AWK ]]; then
echo "$0: ERROR: bin/manifest.csv.entry.awk does not exist: $MANIFEST_ENTRY_CSV_ENTRY_AWK" 1>&2
exit 6
exit 5
fi
if [[ ! -f $MANIFEST_ENTRY_CSV_ENTRY_AWK ]]; then
echo "$0: ERROR: bin/manifest.csv.entry.awk is not a regular file: $MANIFEST_ENTRY_CSV_ENTRY_AWK" 1>&2
exit 6
exit 5
fi
if [[ ! -r $MANIFEST_ENTRY_CSV_ENTRY_AWK ]]; then
echo "$0: ERROR: bin/manifest.csv.entry.awk is not a readable file: $MANIFEST_ENTRY_CSV_ENTRY_AWK" 1>&2
exit 6
exit 5
fi
# verify jparse tool
# verify that the bin/unicode-fix.sed tool is executable
#
if [[ -z "$JPARSE_TOOL" ]]; then
echo "$0: ERROR: cannot find an executable jparse tool" 1>&2
export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed"
if [[ ! -e $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -f "$JPARSE_TOOL" ]]; then
echo "$0: ERROR: jparse is not a file: $JPARSE_TOOL" 1>&2
if [[ ! -f $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -x "$JPARSE_TOOL" ]]; then
echo "$0: ERROR: jparse is not an executable file: $JPARSE_TOOL" 1>&2
if [[ ! -r $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
@@ -512,7 +529,6 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: NOOP=$NOOP" 1>&2
echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2
echo "$0: debug[3]: EXIT_CODE=$EXIT_CODE" 1>&2
echo "$0: debug[3]: JPARSE_TOOL=$JPARSE_TOOL" 1>&2
echo "$0: debug[3]: AUTHOR_WINS_CSV=$AUTHOR_WINS_CSV" 1>&2
echo "$0: debug[3]: MANIFEST_CSV=$MANIFEST_CSV" 1>&2
echo "$0: debug[3]: YEAR_PRIZE_CSV=$YEAR_PRIZE_CSV" 1>&2
@@ -524,6 +540,7 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: TOP_FILE=$TOP_FILE" 1>&2
echo "$0: debug[3]: BIN_PATH=$BIN_DIR" 1>&2
echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2
echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2
echo "$0: debug[3]: JVAL_WRAPPER=$JVAL_WRAPPER" 1>&2
echo "$0: debug[3]: MANIFEST_ENTRY_CSV_ENTRY_AWK=$MANIFEST_ENTRY_CSV_ENTRY_AWK" 1>&2
fi
@@ -859,21 +876,6 @@ for YYYY in $(< "$TOP_FILE"); do
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
continue
fi
#
if [[ $V_FLAG -ge 7 ]]; then
echo "$0: debug[7]: about to run: $JPARSE_TOOL -q -- $ENTRY_JSON" 1>&2
fi
if "$JPARSE_TOOL" -q -- "$ENTRY_JSON"; then
if [[ $V_FLAG -ge 9 ]]; then
echo "$0: debug[9]: valid JSON: $ENTRY_JSON" 1>&2
fi
else
echo "$0: ERROR: .entry.json is not valid JSON: $ENTRY_JSON" 1>&2
EXIT_CODE="1" # exit 1
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
continue
fi
# process the year_prize.csv update
#
@@ -1136,71 +1138,49 @@ sed -e 's/,/ /g' "$AUTHOR_WINS_CSV" | while read -r AUTHOR_HANDLE ENTRY_ID_SET;
#
else
# verify the JSON of author/author_handle.json
# check each entry id on the line
#
if [[ $V_FLAG -ge 7 ]]; then
echo "$0: debug[7]: about to run: $JPARSE_TOOL -q -- $AUTHOR_HANDLE_JSON" 1>&2
fi
echo "$AUTHOR_HANDLE_JSON" >> "$TMP_AUTHOR_HANDLE_INVENTORY"
if "$JPARSE_TOOL" -q -- "$AUTHOR_HANDLE_JSON"; then
echo "$ENTRY_ID_SET" | sed -e 's/ /\n/g' | while read -r ENTRY_ID; do
# case: author/author_handle.json file is valid JSON
# convert ENTRY_ID into YYYY_DIR
#
if [[ $V_FLAG -ge 9 ]]; then
echo "$0: debug[9]: valid JSON for: $AUTHOR_HANDLE_JSON" 1>&2
fi
export YEAR_DIR=${ENTRY_ID%%_*}
export ENTRY_DIR=${ENTRY_ID#*_}
export YYYY_DIR="${YEAR_DIR}/${ENTRY_DIR}"
# check each entry id on the line
# verify that the entry exists
#
echo "$ENTRY_ID_SET" | sed -e 's/ /\n/g' | while read -r ENTRY_ID; do
if [[ ! -d $YYYY_DIR ]]; then
# convert ENTRY_ID into YYYY_DIR
# case: entry does not exist
#
export YEAR_DIR=${ENTRY_ID%%_*}
export ENTRY_DIR=${ENTRY_ID#*_}
export YYYY_DIR="${YEAR_DIR}/${ENTRY_DIR}"
echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV" \
"refers to unknown entry id: $ENTRY_ID" 1>&2
EXIT_CODE="7" # exit 7
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
# NOTE: We do NOT want to stop processing via continue - proceeding instead
# verify that the entry exists
# verify that the author/author_handle.json file refers to the entry id
#
else
# check if the author/author_handle.json file refers to the entry id
#
if [[ ! -d $YYYY_DIR ]]; then
# case: entry does not exist
#
PATTERN='$..entry_id'
FOUND=$("$JVAL_WRAPPER" -b -q "$AUTHOR_HANDLE_JSON" "$PATTERN")
status="$?"
if [[ $status -ne 0 || -z $FOUND ]]; then
echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV" \
"refers to unknown entry id: $ENTRY_ID" 1>&2
EXIT_CODE="7" # exit 7
"entry id: $ENTRY_ID not in winning_entry_set of: $AUTHOR_HANDLE_JSON" 1>&2
EXIT_CODE="9" # exit 9
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
# NOTE: We do NOT want to stop processing via continue - proceeding instead
# verify that the author/author_handle.json file refers to the entry id
#
else
# check if the author/author_handle.json file refers to the entry id
#
FOUND=$("$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON" '$..entry_id')
if [[ -z $FOUND ]]; then
echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV" \
"entry id: $ENTRY_ID not in winning_entry_set of: $AUTHOR_HANDLE_JSON" 1>&2
EXIT_CODE="9" # exit 9
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
# NOTE: We do NOT want to stop processing via continue - proceeding instead
fi
fi
done
# case: author/author_handle.json is not valid JSON
#
else
echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV" \
"is not valid JSON for: $AUTHOR_HANDLE_JSON" 1>&2
EXIT_CODE="1" # exit 1
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
echo "$EXIT_CODE" > "$TMP_EXIT_CODE"
# NOTE: We do NOT want to stop processing via continue - proceeding instead
fi
fi
done
fi
done
bin/gen-authors.sh
+204 -548
View File
@@ -88,7 +88,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an
# set variables referenced in the usage message
#
export VERSION="1.12 2024-08-18"
export VERSION="1.13 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
@@ -200,175 +200,6 @@ Exit codes:
$NAME version: $VERSION"
# output_entry_ids
#
# Write the entry ids from an author/author_handle.json file to standard output (stdout)
#
# usage:
# output_entry_ids author/author_handle.json
#
# returns:
# 0 ==> no errors detected, but output may be empty
# > 0 ==> function error number
#
function output_entry_ids
{
local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path
local entry_id; # a found entry id
# parse args
#
if [[ $# -ne 1 ]]; then
echo "$0: ERROR: in output_entry_ids: expected 1 arg, found $#" 1>&2
return 1
fi
AUTHOR_HANDLE_JSON_PATH="$1"
if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_entry_ids: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 2
fi
if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_entry_ids: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 3
fi
if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_entry_ids: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 4
fi
# extract entry ids from the author/author_handle.json file
#
"$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..entry_id'
return 0
}
# output_author_handle
#
# Write the author handles from an author/author_handle.json file to standard output (stdout)
#
# usage:
# output_author_handle author/author_handle.json
#
# returns:
# 0 ==> no errors detected, but output may be empty
# > 0 ==> function error number
#
function output_author_handle
{
local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path
# parse args
#
if [[ $# -ne 1 ]]; then
echo "$0: ERROR: in output_author_handle: expected 1 arg, found $#" 1>&2
return 1
fi
AUTHOR_HANDLE_JSON_PATH="$1"
if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_author_handle: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 2
fi
if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_author_handle: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 3
fi
if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_author_handle: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 4
fi
# extract entry ids from the author/author_handle.json file
#
"$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..author_handle'
return 0
}
# output_full_name
#
# Write the Full Name from an author/author_handle.json file to standard output (stdout)
#
# usage:
# output_full_name author/author_handle.json
#
# returns:
# 0 ==> no errors detected, but output may be empty
# > 0 ==> function error number
#
function output_full_name
{
local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path
# parse args
#
if [[ $# -ne 1 ]]; then
echo "$0: ERROR: in output_full_name: expected 1 arg, found $#" 1>&2
return 1
fi
AUTHOR_HANDLE_JSON_PATH="$1"
if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_full_name: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 2
fi
if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_full_name: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 3
fi
if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_full_name: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 4
fi
# extract Full Name from the author/author_handle.json file
#
"$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..full_name'
return 0
}
# output_location_code
#
# Write the Location Code from an author/author_handle.json file to standard output (stdout)
#
# usage:
# output_location_code author/author_handle.json
#
# returns:
# 0 ==> no errors detected, but output may be empty
# > 0 ==> function error number
#
function output_location_code
{
local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path
# parse args
#
if [[ $# -ne 1 ]]; then
echo "$0: ERROR: in output_location_code: expected 1 arg, found $#" 1>&2
return 1
fi
AUTHOR_HANDLE_JSON_PATH="$1"
if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_location_code: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 2
fi
if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_location_code: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 3
fi
if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_location_code: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 4
fi
# extract Location Code from the author/author_handle.json file
#
"$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..location_code'
return 0
}
# output_award
#
# Write the award name to standard output (stdout)
@@ -383,6 +214,7 @@ function output_location_code
function output_award
{
local ENTRY_JSON_PATH; # the .entry.json path
local PATTERN; # XPath for JSON pattern
# parse args
#
@@ -406,283 +238,17 @@ function output_award
# obtain the award string
#
"$JVAL_WRAPPER" -w -b "$ENTRY_JSON_PATH" '$..award'
return 0
}
# output_url
#
# Write the url from an author/author_handle.json file to standard output (stdout)
#
# NOTE: If the url for the author is null, nothing is written to standard output.
#
# usage:
# output_url author/author_handle.json
#
# returns:
# 0 ==> no errors detected, but output may be empty
# > 0 ==> function error number
#
function output_url
{
local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path
# parse args
#
if [[ $# -ne 1 ]]; then
echo "$0: ERROR: in output_url: expected 1 arg, found $#" 1>&2
return 1
PATTERN='$..award'
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN'" 1>&2
fi
AUTHOR_HANDLE_JSON_PATH="$1"
if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_url: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 2
"$JVAL_WRAPPER" -b -q "$ENTRY_JSON_PATH" "$PATTERN"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: in output_award: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN' failed," \
"error code: $status" 1>&2
return 5
fi
if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_url: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 3
fi
if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_url: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 4
fi
# extract url from the author/author_handle.json file
#
# NOTE: We output nothing if the JSON member value is null.
#
"$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..url' | grep -F -v null
return 0
}
# output_alt_url
#
# Write the alternate url from an author/author_handle.json file to standard output (stdout)
#
# NOTE: If the alternate url for the author is null, nothing is written to standard output.
#
# usage:
# output_alt_url author/author_handle.json
#
# returns:
# 0 ==> no errors detected, but output may be empty
# > 0 ==> function error number
#
function output_alt_url
{
local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path
# parse args
#
if [[ $# -ne 1 ]]; then
echo "$0: ERROR: in output_alt_url: expected 1 arg, found $#" 1>&2
return 1
fi
AUTHOR_HANDLE_JSON_PATH="$1"
if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_alt_url: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 2
fi
if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_alt_url: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 3
fi
if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_alt_url: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 4
fi
# extract alternate url from the author/author_handle.json file
#
# NOTE: We output nothing if the JSON member value is null.
#
"$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..alt_url' | grep -F -v null
return 0
}
# output_mastodon
#
# Write the mastodon handle from an author/author_handle.json file to standard output (stdout)
#
# NOTE: If the mastodon handle for the author is null, nothing is written to standard output.
#
# usage:
# output_mastodon author/author_handle.json
#
# returns:
# 0 ==> no errors detected, but output may be empty
# > 0 ==> function error number
#
function output_mastodon
{
local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path
# parse args
#
if [[ $# -ne 1 ]]; then
echo "$0: ERROR: in output_mastodon: expected 1 arg, found $#" 1>&2
return 1
fi
AUTHOR_HANDLE_JSON_PATH="$1"
if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_mastodon: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 2
fi
if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_mastodon: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 3
fi
if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_mastodon: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 4
fi
# extract mastodon handle from the author/author_handle.json file
#
# NOTE: We output nothing if the JSON member value is null.
#
"$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..mastodon' | grep -F -v null
return 0
}
# output_mastodon_url
#
# Write the mastodon handle from an author/author_handle.json file to standard output (stdout)
#
# NOTE: If the mastodon URL for the author is null, nothing is written to standard output.
#
# usage:
# output_mastodon_url author/author_handle.json
#
# returns:
# 0 ==> no errors detected, but output may be empty
# > 0 ==> function error number
#
function output_mastodon_url
{
local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path
# parse args
#
if [[ $# -ne 1 ]]; then
echo "$0: ERROR: in output_mastodon_url: expected 1 arg, found $#" 1>&2
return 1
fi
AUTHOR_HANDLE_JSON_PATH="$1"
if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_mastodon_url: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 2
fi
if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_mastodon_url: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 3
fi
if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_mastodon_url: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 4
fi
# extract mastodon URL from the author/author_handle.json file
#
# NOTE: We output nothing if the JSON member value is null.
#
"$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..mastodon_url' | grep -F -v null
return 0
}
# output_github
#
# Write the GitHub handle from an author/author_handle.json file to standard output (stdout)
#
# NOTE: If the GitHub handle for the author is null, nothing is written to standard output.
#
# usage:
# output_github author/author_handle.json
#
# returns:
# 0 ==> no errors detected, but output may be empty
# > 0 ==> function error number
#
function output_github
{
local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path
# parse args
#
if [[ $# -ne 1 ]]; then
echo "$0: ERROR: in output_github: expected 1 arg, found $#" 1>&2
return 1
fi
AUTHOR_HANDLE_JSON_PATH="$1"
if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_github: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 2
fi
if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_github: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 3
fi
if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_github: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 4
fi
# extract GitHub handle from the author/author_handle.json file
#
# NOTE: We output nothing if the JSON member value is null.
#
"$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..github' | grep -F -v null
return 0
}
# output_affiliation
#
# Write the affiliation from an author/author_handle.json file to standard output (stdout)
#
# NOTE: If the affiliation for the author is null, nothing is written to standard output.
#
# usage:
# output_affiliation author/author_handle.json
#
# returns:
# 0 ==> no errors detected, but output may be empty
# > 0 ==> function error number
#
function output_affiliation
{
local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path
# parse args
#
if [[ $# -ne 1 ]]; then
echo "$0: ERROR: in output_affiliation: expected 1 arg, found $#" 1>&2
return 1
fi
AUTHOR_HANDLE_JSON_PATH="$1"
if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_affiliation: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 2
fi
if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_affiliation: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 3
fi
if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then
echo "$0: ERROR: in output_affiliation: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2
return 4
fi
# extract affiliation from the author/author_handle.json file
#
# NOTE: We output nothing if the JSON member value is null.
#
"$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..affiliation' | grep -F -v null
return 0
}
@@ -919,6 +485,40 @@ fi
export AUTHORS_HTML="authors.html"
# verify that the bin/combine_author_handle.sh tool is executable
#
export COMBINE_AUTHOR="$BIN_DIR/combine_author_handle.sh"
if [[ ! -e $COMBINE_AUTHOR ]]; then
echo "$0: ERROR: bin/unicode-chk.sh does not exist: $COMBINE_AUTHOR" 1>&2
exit 5
fi
if [[ ! -f $COMBINE_AUTHOR ]]; then
echo "$0: ERROR: bin/unicode-chk.sh is not a regular file: $COMBINE_AUTHOR" 1>&2
exit 5
fi
if [[ ! -x $COMBINE_AUTHOR ]]; then
echo "$0: ERROR: bin/unicode-chk.sh is not an executable file: $COMBINE_AUTHOR" 1>&2
exit 5
fi
# verify that the bin/unicode-fix.sed tool is executable
#
export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed"
if [[ ! -e $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -f $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -r $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
# print running info if verbose
#
# If -v 3 or higher, print exported variables in order that they were exported.
@@ -938,8 +538,8 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: SITE_URL=$SITE_URL" 1>&2
echo "$0: debug[3]: NOOP=$NOOP" 1>&2
echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2
echo "$0: debug[3]: EXIT_CODE=$EXIT_CODE" 1>&2
echo "$0: debug[3]: QUICK_MODE=$QUICK_MODE" 1>&2
echo "$0: debug[3]: EXIT_CODE=$EXIT_CODE" 1>&2
for index in "${!IS_FOR[@]}"; do
echo "$0: debug[3]: IS_FOR[$index]=${IS_FOR[$index]}" 1>&2
done
@@ -955,6 +555,8 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: JVAL_WRAPPER=$JVAL_WRAPPER" 1>&2
echo "$0: debug[3]: LOCATION_TOOL=$LOCATION_TOOL" 1>&2
echo "$0: debug[3]: AUTHORS_HTML=$AUTHORS_HTML" 1>&2
echo "$0: debug[3]: COMBINE_AUTHOR=$COMBINE_AUTHOR" 1>&2
echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2
fi
@@ -1026,31 +628,32 @@ elif [[ $V_FLAG -ge 3 ]]; then
fi
# load initial letter . lines into the temporary sort word list file
#
for letter in "${!IS_FOR[@]}"; do
echo "$letter ."
done > "$TMP_SORT_WORD"
# add author sort_word filename to the temporary sort word list file
#
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: Forming temporary sort word list file: $TMP_SORT_WORD"
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: Forming temporary sort word list file: $TMP_SORT_WORD"
fi
find "$AUTHOR_DIR" -mindepth 1 -maxdepth 1 -type f -name '*.json' 2>/dev/null | while read -r json_file; do
echo "$(bin/jval-wrapper.sh -w -b "$json_file" '$..sort_word') $json_file"
done >> "$TMP_SORT_WORD"
export PATTERN='$..sort_word'
{
# load initial letter . lines
#
for letter in "${!IS_FOR[@]}"; do
echo "$letter ."
done
# sort the temporary sort word list file
#
LC_ALL=C sort -f -d "$TMP_SORT_WORD" -o "$TMP_SORT_WORD"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: LC_ALL=C sort -f -d $TMP_SORT_WORD -o $TMP_SORT_WORD failed, error: $status" 1>&2
exit 14
fi
# add sort_word filenames lines
#
"$COMBINE_AUTHOR" | "$JVAL_WRAPPER" -b -q -- - "$PATTERN"
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] ]]; then
echo "$0: ERROR: $COMBINE_AUTHOR | $JVAL_WRAPPER -b -q -- - '$PATTERN' |" \
"LC_ALL=C sort -f -d > $TMP_SORT_WORD failed," \
"error codes: ${status_codes[*]}" 1>&2
exit 14
fi
} | LC_ALL=C sort -f -d > "$TMP_SORT_WORD"
# generate the temporary entry markdown file
@@ -1114,9 +717,7 @@ EOF
echo '<hr style="width:10%;text-align:left;margin-left:0">'
echo
if [[ $V_FLAG -ge 1 ]]; then
echo 1>&2
echo "$0: debug[1]: Processing author_handles that begin with $l for: $AUTHORS_HTML" 1>&2
echo 1>&2
fi
# case: we have a "sort_word file" line
@@ -1130,91 +731,145 @@ EOF
exit 15
fi
FULL_NAME=$(output_full_name "$filename")
export FULL_NAME
if [[ -z $FULL_NAME ]]; then
echo "$0: ERROR: full name is empty in: $filename" 1>&2
# parse an element value of the this particular author in the authors JSON array
#
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: $JVAL_WRAPPER -q -- - | sed ... | LC_ALL=C sort -d)" 1>&2
fi
export AUTHOR_DATA
AUTHOR_DATA=$("$JVAL_WRAPPER" -q -- "$filename" |
sed -f "$UNICODE_FIX_SED" |
LC_ALL=C sort -d)
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] ]]; then
echo "$0: ERROR: $JVAL_WRAPPER -q -- $filename | sed ... | LC_ALL=C sort -d failed," \
"error codes: ${status_codes[*]}" 1>&2
exit 16
fi
LOCATION_CODE=$(output_location_code "$filename")
export LOCATION_CODE
if [[ -z $LOCATION_CODE ]]; then
echo "$0: ERROR: location code is empty in: $filename" 1>&2
if [[ -z $AUTHOR_DATA ]]; then
echo "$0: ERROR: found nothing for author in: $filename" 1>&2
exit 17
fi
export FULL_NAME=
export LOCATION_CODE=
export AUTHOR_URL=
export AUTHOR_ALT_URL=
export AUTHOR_MASTODON_HANDLE=
export AUTHOR_MASTODON_URL=
export AUTHOR_GITHUB=
export AUTHOR_AFFILIATION=
export AUTHOR_HANDLE=
unset ENTRY_ID_SET
declare -ag ENTRY_ID_SET
while read -r NAME VALUE; do
# parse name value
#
case "$NAME" in
full_name)
FULL_NAME="$VALUE"
;;
location_code)
LOCATION_CODE="$VALUE"
;;
url)
AUTHOR_URL="$VALUE"
;;
alt_url)
AUTHOR_ALT_URL="$VALUE"
;;
mastodon)
AUTHOR_MASTODON_HANDLE="$VALUE"
;;
mastodon_url)
AUTHOR_MASTODON_URL="$VALUE"
;;
github)
AUTHOR_GITHUB="$VALUE"
;;
affiliation)
AUTHOR_AFFILIATION="$VALUE"
;;
author_handle)
AUTHOR_HANDLE="$VALUE"
;;
entry_id)
ENTRY_ID_SET+=("$VALUE")
;;
*) # ignore all other fields
;;
esac
done <<< "$AUTHOR_DATA"
# validate fields values
#
if [[ -z $FULL_NAME ]]; then
echo "$0: ERROR: cannot find name in: $filename" 1>&2
exit 18
fi
if [[ -z $LOCATION_CODE ]]; then
echo "$0: ERROR: cannot find location_code in: $filename" 1>&2
exit 19
fi
if [[ -z $AUTHOR_URL ]]; then
echo "$0: ERROR: cannot find url in: $filename" 1>&2
exit 20
fi
if [[ -z $AUTHOR_ALT_URL ]]; then
echo "$0: ERROR: cannot find alt_url in: $filename" 1>&2
exit 21
fi
if [[ -z $AUTHOR_MASTODON_HANDLE ]]; then
echo "$0: ERROR: cannot find mastodon in: $filename" 1>&2
exit 22
fi
if [[ -z $AUTHOR_MASTODON_URL ]]; then
echo "$0: ERROR: cannot find mastodon_url in: $filename" 1>&2
exit 23
fi
if [[ -z $AUTHOR_GITHUB ]]; then
echo "$0: ERROR: cannot find github in: $filename" 1>&2
exit 24
fi
if [[ -z $AUTHOR_AFFILIATION ]]; then
echo "$0: ERROR: cannot find affiliation in: $filename" 1>&2
exit 25
fi
if [[ -z $AUTHOR_HANDLE ]]; then
echo "$0: ERROR: cannot find author_handle in: $filename" 1>&2
exit 26
fi
# use the location tool to obtain the location_name
#
LOCATION_NAME=$("$LOCATION_TOOL" "$LOCATION_CODE" 2>/dev/null)
status="$?"
export LOCATION_NAME
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: cannot determine location name" \
"for location ISO 3166 code: $LOCATION_CODE" 1>&2
exit 18
fi
if [[ -z $LOCATION_NAME ]]; then
echo "$0: ERROR: location name" \
"for location ISO 3166 code: $LOCATION_CODE is empty: $LOCATION_NAME" 1>&2
exit 19
echo "$0: ERROR: cannot determine location name for" \
"location ISO 3166 code: $LOCATION_CODE" 1>&2
exit 1
fi
# use the location tool to obtain the location common name
#
LOCATION_COMMON_NAME=$("$LOCATION_TOOL" -c "$LOCATION_CODE" 2>/dev/null)
status="$?"
export LOCATION_COMMON_NAME
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: cannot determine location common name" \
"for location ISO 3166 code: $LOCATION_CODE" 1>&2
exit 20
fi
if [[ -z $LOCATION_COMMON_NAME ]]; then
echo "$0: ERROR: location common name" \
"for location ISO 3166 code: $LOCATION_CODE is empty: $LOCATION_COMMON_NAME" 1>&2
exit 21
exit 1
fi
AUTHOR_HANDLE=$(output_author_handle "$filename")
export AUTHOR_HANDLE
if [[ -z $AUTHOR_HANDLE ]]; then
echo "$0: ERROR: author handle is empty in: $filename" 1>&2
exit 22
fi
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: Processing author_handle $AUTHOR_HANDLE for: $AUTHORS_HTML" 1>&2
fi
# NOTE: AUTHOR_AFFILIATION will be empty if there is no affiliation
AUTHOR_AFFILIATION=$(output_affiliation "$filename")
export AUTHOR_AFFILIATION
# NOTE: AUTHOR_URL will be empty if there is no url
AUTHOR_URL=$(output_url "$filename")
export AUTHOR_URL
# NOTE: AUTHOR_ALT_URL will be empty if there is no alternate url
AUTHOR_ALT_URL=$(output_alt_url "$filename")
export AUTHOR_ALT_URL
# NOTE: AUTHOR_MASTODON_HANDLE will be empty if there is no mastodon handle
AUTHOR_MASTODON_HANDLE=$(output_mastodon "$filename")
export AUTHOR_MASTODON_HANDLE
# NOTE: AUTHOR_MASTODON_URL will be empty if there is no mastodon URL
AUTHOR_MASTODON_URL=$(output_mastodon_url "$filename")
export AUTHOR_MASTODON_URL
if [[ -n $AUTHOR_MASTODON_HANDLE && -z $AUTHOR_MASTODON_URL ]]; then
echo "$0: ERROR: mastodon url is null but mastodon handle is not null in: $filename" 1>&2
exit 23
fi
if [[ -z $AUTHOR_MASTODON_HANDLE && -n $AUTHOR_MASTODON_URL ]]; then
echo "$0: ERROR: mastodon handle is null but mastodon url is not null in: $filename" 1>&2
exit 24
fi
# NOTE: AUTHOR_GITHUB will be empty if there is no GitHub handle
AUTHOR_GITHUB=$(output_github "$filename")
export AUTHOR_GITHUB
# form github username
#
AUTHOR_GITHUB_USERNAME=${AUTHOR_GITHUB#@}
export AUTHOR_GITHUB_USERNAME
export AUTHOR_GITHUB_USERNAME
if [[ -z $AUTHOR_GITHUB_USERNAME ]]; then
echo "$0: ERROR: cannot determine github username from: $AUTHOR_GITHUB" 1>&2
exit 27
fi
# output author information
#
@@ -1228,23 +883,24 @@ EOF
echo "Location: <a class=\"normal\" href=\"location.html#$LOCATION_CODE\">$LOCATION_CODE</a> -" \
"_${LOCATION_COMMON_NAME}_"
fi
if [[ -n $AUTHOR_AFFILIATION ]]; then
if [[ -n $AUTHOR_AFFILIATION && $AUTHOR_AFFILIATION != null ]]; then
echo "<br>"
echo "Affiliation: $AUTHOR_AFFILIATION"
fi
if [[ -n $AUTHOR_URL ]]; then
if [[ -n $AUTHOR_URL && $AUTHOR_URL != null ]]; then
echo "<br>"
echo "URL: <a class=\"normal\" href=\"$AUTHOR_URL\">$AUTHOR_URL</a>"
fi
if [[ -n $AUTHOR_ALT_URL ]]; then
if [[ -n $AUTHOR_ALT_URL && $AUTHOR_ALT_URL != null ]]; then
echo "<br>"
echo "Alternate URL: <a class=\"normal\" href=\"$AUTHOR_ALT_URL\">$AUTHOR_ALT_URL</a>"
fi
if [[ -n $AUTHOR_MASTODON_HANDLE && -n $AUTHOR_MASTODON_URL ]]; then
if [[ -n $AUTHOR_MASTODON_HANDLE && $AUTHOR_MASTODON_HANDLE != null &&
-n $AUTHOR_MASTODON_URL && $AUTHOR_MASTODON_URL != null ]]; then
echo "<br>"
echo "Mastodon: <a class=\"normal\" href=\"$AUTHOR_MASTODON_URL\">$AUTHOR_MASTODON_HANDLE</a>"
fi
if [[ -n $AUTHOR_GITHUB ]]; then
if [[ -n $AUTHOR_GITHUB && $AUTHOR_GITHUB != null ]]; then
echo "<br>"
echo "GitHub: <a class=\"normal\" href=\"https://github.com/$AUTHOR_GITHUB_USERNAME\">$AUTHOR_GITHUB</a>"
fi
@@ -1254,23 +910,23 @@ EOF
# output YYYY/dir set made by this author
#
output_entry_ids "$filename" | while read -r ENTRY_ID; do
for ENTRY_ID in "${ENTRY_ID_SET[@]}"; do
# verify the entry id
#
if [[ $ENTRY_ID =~ ^[^_][^_]*_[^_][^_]*$ ]]; then
if [[ $V_FLAG -ge 7 ]]; then
echo "$0: debug[7]: entry_id: $entry_id" 1>&2
echo "$0: debug[7]: entry_id: $ENTRY_ID" 1>&2
fi
else
echo "$0: ERROR: entry_id is not in YYYY_dir formmat: $entry_id" 1>&2
exit 25
echo "$0: ERROR: entry_id is not in YYYY_dir formmat: $ENTRY_ID" 1>&2
exit 28
fi
YYYY_DIR=$(echo "$ENTRY_ID" | tr _ /)
export YYYY_DIR
if [[ ! -d $YYYY_DIR ]]; then
echo "$0: ERROR: YYYY/dir is not a directory" 1>&2
exit 26
exit 29
fi
# verify that the .entry.json file is a non-empty readable file
@@ -1279,19 +935,19 @@ EOF
export ENTRY_JSON
if [[ ! -e $ENTRY_JSON ]]; then
echo "$0: ERROR: .entry.json does not exist: $ENTRY_JSON" 1>&2
exit 27
exit 30
fi
if [[ ! -f $ENTRY_JSON ]]; then
echo "$0: ERROR: .entry.json is not a file: $ENTRY_JSON" 1>&2
exit 28
exit 31
fi
if [[ ! -r $ENTRY_JSON ]]; then
echo "$0: ERROR: .entry.json is not a readable file: $ENTRY_JSON" 1>&2
exit 29
exit 32
fi
if [[ ! -s $ENTRY_JSON ]]; then
echo "$0: ERROR: .entry.json is not a non-empty readable file: $ENTRY_JSON" 1>&2
exit 30
exit 33
fi
if [[ $V_FLAG -ge 7 ]]; then
echo "$0: debug[7]: .entry.json: $ENTRY_JSON" 1>&2
@@ -1299,11 +955,11 @@ EOF
# collect the award for this entry id
#
AWARD=$(output_award "$ENTRY_JSON")
AWARD=$(output_award "$ENTRY_JSON" | sed -f "$UNICODE_FIX_SED")
export AWARD
if [[ -z $AWARD ]]; then
echo "$0: ERROR: award not found in .entry.json: $ENTRY_JSON" 1>&2
exit 31
exit 34
fi
# output the YYYY/dir entry for this author
bin/gen-location.sh
+32 -7
View File
@@ -85,7 +85,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an
# set variables referenced in the usage message
#
export VERSION="1.6 2024-08-18"
export VERSION="1.7 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
@@ -397,6 +397,23 @@ fi
export LOCATION_HTML="location.html"
# verify that the bin/unicode-fix.sed tool is executable
#
export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed"
if [[ ! -e $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -f $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -r $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
# print running info if verbose
#
# If -v 3 or higher, print exported variables in order that they were exported.
@@ -430,6 +447,7 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: JVAL_WRAPPER=$JVAL_WRAPPER" 1>&2
echo "$0: debug[3]: LOCATION_TOOL=$LOCATION_TOOL" 1>&2
echo "$0: debug[3]: LOCATION_HTML=$LOCATION_HTML" 1>&2
echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2
fi
# -N stops early before any processing is performed
@@ -529,14 +547,18 @@ if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: about to obtain location codes from author_handle JSON files" 1>&2
fi
export AUTHOR_COUNT=0
export AUTHOR_COUNT_CYCLE=20
export AUTHOR_COUNT_CYCLE=10
export AUTHOR_COUNT_COUNT
AUTHOR_COUNT_COUNT=$(find "$AUTHOR_DIR" -mindepth 1 -maxdepth 1 -type f -name '*.json' -print | wc -l)
find "$AUTHOR_DIR" -mindepth 1 -maxdepth 1 -type f -name '*.json' -print |
while read -r AUTHOR_HANDLE_JSON; do
# collect location_code
#
LOCATION_CODE=$("$JVAL_WRAPPER" -b -w "$AUTHOR_HANDLE_JSON" '$..location_code')
if [[ -z $LOCATION_CODE ]]; then
PATTERN='$..location_code'
LOCATION_CODE=$("$JVAL_WRAPPER" -b -q "$AUTHOR_HANDLE_JSON" "$PATTERN")
status="$?"
if [[ $status -ne 0 || -z $LOCATION_CODE ]]; then
echo "$0: ERROR: cannot obtain location_code from: $AUTHOR_HANDLE_JSON" 1>&2
exit 1
fi
@@ -551,8 +573,10 @@ find "$AUTHOR_DIR" -mindepth 1 -maxdepth 1 -type f -name '*.json' -print |
# obtain the author full name
#
FULL_NAME=$("$JVAL_WRAPPER" -b -w "$AUTHOR_HANDLE_JSON" '$..full_name')
if [[ -z $FULL_NAME ]]; then
PATTERN='$..full_name'
FULL_NAME=$("$JVAL_WRAPPER" -b -q "$AUTHOR_HANDLE_JSON" "$PATTERN" | sed -f "$UNICODE_FIX_SED")
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] || -z $FULL_NAME ]]; then
echo "$0: ERROR: cannot full name from: $AUTHOR_HANDLE" 1>&2
exit 1
fi
@@ -573,7 +597,8 @@ find "$AUTHOR_DIR" -mindepth 1 -maxdepth 1 -type f -name '*.json' -print |
#
((++AUTHOR_COUNT))
if [[ $V_FLAG -ge 1 && $((AUTHOR_COUNT % AUTHOR_COUNT_CYCLE)) -eq 0 ]]; then
echo "$0: debug[1]: cycle $AUTHOR_COUNT finished processing: $AUTHOR_HANDLE_JSON" 1>&2
echo "$0: debug[1]: cycle $AUTHOR_COUNT out of $AUTHOR_COUNT_COUNT" \
"finished processing: $AUTHOR_HANDLE_JSON" 1>&2
fi
done | LC_ALL=C sort -d -u > "$TMP_LOC_WORD_HANDLE_NAME_SET"
bin/gen-status.sh
+21 -23
View File
@@ -86,7 +86,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an
# set variables referenced in the usage message
#
export VERSION="1.6 2024-08-17"
export VERSION="1.7 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
@@ -108,12 +108,6 @@ export REPO_TOP_URL="https://github.com/ioccc-src/temp-test-ioccc"
# GitHub puts individual files under the "blob/master" sub-directory.
export REPO_URL="$REPO_TOP_URL/blob/master"
export SITE_URL="https://ioccc-src.github.io/temp-test-ioccc"
JPARSE_TOOL=$(type -P jparse)
export JPARSE_TOOL
if [[ -z "$JPARSE_TOOL" ]]; then
echo "$0: FATAL: jparse tool is not installed or not in \$PATH" 1>&2
exit 5
fi
#
STAT_TOOL=$(type -P stat)
if [[ -z "$STAT_TOOL" ]]; then
@@ -294,11 +288,23 @@ function output_status_json
# determine the "contest_status" that was in status.json
#
OLD_CONTEST_STATUS=$("$JVAL_WRAPPER" -w -b "$STATUS_PATH" '$..contest_status')
PATTERN='$..contest_status'
OLD_CONTEST_STATUS=$("$JVAL_WRAPPER" -b -q "$STATUS_PATH" "$PATTERN")
status="$?"
if [[ $status -ne 0 || -z $OLD_CONTEST_STATUS ]]; then
echo "$0: ERROR: cannot determine old contest_status from status.json: $STATUS_JSON" 1>&2
return 11
fi
# determine the "status_update" that was in status.json
#
OLD_STATUS_UPDATE=$("$JVAL_WRAPPER" -w -b "$STATUS_PATH" '$..status_update')
PATTERN='$..status_update'
OLD_STATUS_UPDATE=$("$JVAL_WRAPPER" -b -q "$STATUS_PATH" "$PATTERN")
status="$?"
if [[ $status -ne 0 || -z $OLD_STATUS_UPDATE ]]; then
echo "$0: ERROR: cannot determine old status_update from status.json: $STATUS_JSON" 1>&2
return 12
fi
# update the status.json file modification date if the "contest_status" changed
#
@@ -306,11 +312,11 @@ function output_status_json
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: in output_status_json: modification date of $STATUS_PATH failed, error: $status" 1>&2
return 11
return 13
fi
if [[ -z $NEW_STATUS_UPDATE ]]; then
echo "$0: ERROR: in output_status_json: modification date of $STATUS_PATH is empty" 1>&2
return 12
return 14
fi
# only update the "status_update" If the "contest_status" changed
@@ -788,7 +794,6 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: REPO_TOP_URL=$REPO_TOP_URL" 1>&2
echo "$0: debug[3]: REPO_URL=$REPO_URL" 1>&2
echo "$0: debug[3]: SITE_URL=$SITE_URL" 1>&2
echo "$0: debug[3]: JPARSE_TOOL=$JPARSE_TOOL" 1>&2
echo "$0: debug[3]: IOCCC_STATUS_VERSION=$IOCCC_STATUS_VERSION" 1>&2
echo "$0: debug[3]: NOOP=$NOOP" 1>&2
echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2
@@ -819,23 +824,16 @@ if [[ $V_FLAG -ge 3 ]]; then
fi
# validate JSON in status.json
#
"$JPARSE_TOOL" -q "$STATUS_JSON"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: status.json is not valid JSON, jparse error code: $status" 1>&2
exit 1
fi
# obtain contest_status if CONTEST_STATUS was not set in the arg
#
if [[ -z $CONTEST_STATUS ]]; then
# obtain the contest_status from status.json
#
CONTEST_STATUS=$("$JVAL_WRAPPER" -w -b "$STATUS_JSON" '$..contest_status')
if [[ -z $CONTEST_STATUS ]]; then
PATTERN='$..contest_status'
CONTEST_STATUS=$("$JVAL_WRAPPER" -b -q "$STATUS_JSON" "$PATTERN")
status="$?"
if [[ $status -ne 0 || -z $CONTEST_STATUS ]]; then
echo "$0: ERROR: cannot determine contest_status from status.json: $STATUS_JSON" 1>&2
exit 1
fi
bin/gen-years.sh
+26 -5
View File
@@ -88,7 +88,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an
# set variables referenced in the usage message
#
export VERSION="1.4 2024-08-17"
export VERSION="1.5 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
@@ -213,8 +213,10 @@ function output_award
# obtain the award string
#
AWARD_STRING=$("$JVAL_WRAPPER" -w -b "$ENTRY_JSON_PATH" '$..award')
if [[ -z $AWARD_STRING ]]; then
PATTERN='$..award'
AWARD_STRING=$("$JVAL_WRAPPER" -b -q "$ENTRY_JSON_PATH" "$PATTERN")
status="$?"
if [[ $status -ne 0 || -z $AWARD_STRING ]]; then
echo "$0: ERROR: in output_award: no award found in .entry.json file: $ENTRY_JSON_PATH" 1>&2
return 5
fi
@@ -550,6 +552,23 @@ fi
export YEARS_HTML="years.html"
# verify that the bin/unicode-fix.sed tool is executable
#
export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed"
if [[ ! -e $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -f $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -r $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
# print running info if verbose
#
# If -v 3 or higher, print exported variables in order that they were exported.
@@ -582,6 +601,7 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: JVAL_WRAPPER=$JVAL_WRAPPER" 1>&2
echo "$0: debug[3]: TOP_FILE=$TOP_FILE" 1>&2
echo "$0: debug[3]: YEARS_HTML=$YEARS_HTML" 1>&2
echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2
fi
@@ -848,8 +868,9 @@ for YYYY in $("$TAC_TOOL" "$TOP_FILE"); do
# determine the award for this entry
#
AWARD=$(output_award "$ENTRY_JSON")
if [[ -z $AWARD ]]; then
AWARD=$(output_award "$ENTRY_JSON" | sed -f "$UNICODE_FIX_SED")
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] || -z $AWARD ]]; then
echo "$0: ERROR: cannot find award in .entry.json: $ENTRY_JSON" 1>&2
EXIT_CODE="7" # exit 7
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
bin/jfmt-wrapper.sh
+281 -67
View File
@@ -2,15 +2,30 @@
#
# jfmt-wrapper.sh - canonically format a JSON file
#
# We form a wrapper for the jfmt(1), a tool that will format a JSON
# file into a canonical style. NOTE: As of 2024 July 28 the jfmt(1)
# tool has not been written, so jfmt-wrapper.sh uses the JSONPath.sh(1)
# tool from the recently forked and modified JSONPath.sh tool:
# If the JSON file is already on canonical form, the file is not modified.
#
# We form a wrapper for the jprint(1), a tool that will format a JSON
# file into a canonical style.
#
# NOTE: As of 2024 Sep 09 the jprint(1) tool has not been written,
# so jfmt-wrapper.sh uses with the jsp tool from:
#
# https://github.com/kjozsa/jsp
#
# FYI: For macOS, use homebrew to install pipx:
#
# brew install pipx
#
# Next, use pipx to install jsp:
#
# pipx --global install jsp
#
# and then be sure that ~/.local/bin is in your $PATH.
#
# or the JSONPath.sh(1) tool from:
#
# https://github.com/lcn2/JSONPath.sh
#
# If the JSON file is already on canonical form, the file is not modified.
#
# Copyright (c) 2024 by Landon Curt Noll. All Rights Reserved.
#
# Permission to use, copy, modify, and distribute this software and
@@ -95,43 +110,50 @@ shopt -s globstar # enable ** to match all files and zero or more directories an
# set variables referenced in the usage message
#
export VERSION="1.1 2024-08-20"
export VERSION="1.2 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
#
JPARSE_TOOL=$(type -P jparse)
export JPARSE_TOOL
# Until we have a true jfmt tool, we will use the modified JSONPath.sh
#
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - until we have the jfmt command, we must FAKE PARSE IOCCC JSON files - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
#
export JSONPATH_REPO="https://github.com/lcn2/JSONPath.sh"
JSONPATH_SH=$(type -P JSONPath.sh)
export JSONPATH_SH
if [[ -z $JSONPATH_SH ]]; then
echo "$0: FATAL: JSONPath.sh tool is not installed or not in \$PATH" 1>&2
echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2
GIT_TOOL=$(type -P git)
export GIT_TOOL
if [[ -z "$GIT_TOOL" ]]; then
echo "$0: FATAL: git tool is not installed or not in \$PATH" 1>&2
exit 5
fi
# verify JSONPath.sh supports -S -A
"$GIT_TOOL" rev-parse --is-inside-work-tree >/dev/null 2>&1
status="$?"
if [[ $status -eq 0 ]]; then
TOPDIR=$("$GIT_TOOL" rev-parse --show-toplevel)
fi
export TOPDIR
export REPO_TOP_URL="https://github.com/ioccc-src/temp-test-ioccc"
# GitHub puts individual files under the "blob/master" sub-directory.
export REPO_URL="$REPO_TOP_URL/blob/master"
# Until we have a true jfmt tool, we will use wither jsp or the modified JSONPath.sh
#
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - until we have the jprint command, we must use external tools to parse JSON - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
#
export JSP_REPO="https://github.com/kjozsa/jsp"
export JSONPATH_REPO="https://github.com/lcn2/JSONPath.sh"
export FIZZBIN_JSON='"fizzbin"'
if ! "$JSONPATH_SH" -S -A -p >/dev/null 2>&1; then
echo "$0: FATAL: JSONPath.sh tool does not support -S -A: $FIZZBIN_JSON" 1>&2
echo "$0: notice: we recommend you obtain and install JSONPath.sh from: $JSONPATH_REPO" 1>&2
exit 5
fi <<< "$FIZZBIN_JSON"
export JSP_TOOL=""
JSP_TOOL=$(type -P jsp)
export JSONPATH_SH=""
JSONPATH_SH=$(type -P JSONPath.sh)
# set the defaiult XPath for JSON
export XPATHJSON_USE="jsp"
export QUICK_CHECK=""
# set usage message
#
export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] [-j JSONPath.sh] file.json
export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] [-t tool] [-Q] file.json
-h print help message and exit
-v level set verbosity level (def level: 0)
@@ -140,7 +162,8 @@ export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] [-j JSONPath.sh] file.jso
-n do not modify file.json (def: modify file.json is not canonical)
-N do not process file, just parse arguments and ignore the file (def: process the file)
-j JSONPath.sh path to the JSONPath.sh tool (not the wrapper) (def: $JSONPATH_SH)
-t tool tool to use: jsp or JSONPath.sh (def: try jsp, otherwise try JSONPath.sh)
-Q quick check tool using trivial input (def: do not)
file.json update file.json in canonical form
@@ -148,11 +171,11 @@ NOTE: file.json is not touched if it is already in canonical form
Exit codes:
0 all OK
1 JSONPath.sh exited non-zero
1 some internal tool exited non-zero
2 -h and help string printed or -V and version string printed
3 command line error
4 bash version is too old
5 internal tool not found
5 some internal tool not found
6 file.json does not exist or is not readable
7 file.json is not valid JSON
>= 10 internal error
@@ -168,7 +191,7 @@ export DO_NOT_PROCESS=
# parse command line
#
while getopts :hv:VnNj: flag; do
while getopts :hv:VnNt:Q flag; do
case "$flag" in
h) echo "$USAGE" 1>&2
exit 2
@@ -182,8 +205,19 @@ while getopts :hv:VnNj: flag; do
;;
N) DO_NOT_PROCESS="-N"
;;
j) JSONPATH_SH="$OPTARG"
;;
t) # validate -t tool
case "$OPTARG" in
jsp) XPATHJSON_USE="jsp" ;;
JSONPath.sh) XPATHJSON_USE="JSONPath.sh" ;;
*) echo "$0: ERROR: unknown -t option: $OPTARG" 1>&2
echo 1>&2
echo "$USAGE" 1>&2
exit 3
;;
esac
;;
Q) QUICK_CHECK="true"
;;
\?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2
echo 1>&2
echo "$USAGE" 1>&2
@@ -237,6 +271,155 @@ if [[ ! -r $JSON_FILE ]]; then
fi
# verify that we have a topdir directory
#
REPO_NAME=$(basename "$REPO_TOP_URL")
export REPO_NAME
if [[ -z $TOPDIR ]]; then
echo "$0: ERROR: cannot find top of git repo directory" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
if [[ ! -e $TOPDIR ]]; then
echo "$0: ERROR: TOPDIR does not exist: $TOPDIR" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
if [[ ! -d $TOPDIR ]]; then
echo "$0: ERROR: TOPDIR is not a directory: $TOPDIR" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
# cd to topdir
#
if [[ ! -e $TOPDIR ]]; then
echo "$0: ERROR: cannot cd to non-existent path: $TOPDIR" 1>&2
exit 6
fi
if [[ ! -d $TOPDIR ]]; then
echo "$0: ERROR: cannot cd to a non-directory: $TOPDIR" 1>&2
exit 6
fi
export CD_FAILED
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to: cd $TOPDIR" 1>&2
fi
cd "$TOPDIR" || CD_FAILED="true"
if [[ -n $CD_FAILED ]]; then
echo "$0: ERROR: cd $TOPDIR failed" 1>&2
exit 6
fi
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: now in directory: $(/bin/pwd)" 1>&2
fi
# verify that we have a bin subdirectory
#
export BIN_PATH="$TOPDIR/bin"
if [[ ! -d $BIN_PATH ]]; then
echo "$0: ERROR: bin is not a directory under topdir: $BIN_PATH" 1>&2
exit 6
fi
export BIN_DIR="bin"
# verify that the bin/unicode-fix.sed tool is executable
#
export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed"
if [[ ! -e $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -f $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -r $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
# validate the XPath for JSON tool
#
case "$XPATHJSON_USE" in
# case: we will use JSONPath.sh
#
JSONPath.sh)
# try JSONPath.sh
#
if [[ -z $JSONPATH_SH || ! -x $JSONPATH_SH ]]; then
echo "$0: ERROR: JSONPath.sh tool is not installed, is not executable, or not in \$PATH" 1>&2
echo "$0: notice: obtain jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
fi
# case: -Q - perform quick check
#
if [[ -n $QUICK_CHECK ]]; then
# verify JSONPath.sh supports -S -A -p
if ! "$JSONPATH_SH" -S -A -p >/dev/null 2>&1; then
echo "$0: ERROR: JSONPath.sh tool does not support -S -A: $FIZZBIN_JSON" 1>&2
echo "$0: notice: we recommend you obtain and install jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: otherwise install JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
fi <<< "$FIZZBIN_JSON"
fi
;;
# case: we will use jsp
#
jsp)
# try jsp
#
if [[ -z $JSP_TOOL || ! -x $JSP_TOOL ]]; then
echo "$0: ERROR: jsp.sh tool is not installed, is not executable, or not in \$PATH" 1>&2
echo "$0: notice: obtain jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
fi
# case: -Q - perform quick check
#
if [[ -n $QUICK_CHECK ]]; then
# verify jsp
if ! "$JSP_TOOL" --indent 4 --format --no-color >/dev/null 2>&1; then
echo "$0: ERROR: jsp tool does not support --indent 4 --format --no-color: $FIZZBIN_JSON" 1>&2
echo "$0: notice: you might need to update and reinstall jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: or you might try to obtain and install JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
fi <<< "$FIZZBIN_JSON"
fi
;;
# case: we do not have an XPath for JSON tool
#
*)
echo "$0: ERROR: cannot find an XPath JSON tool to use" 1>&2
echo "$0: notice: we recommend you obtain and install jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: otherwise install JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
;;
esac
# print running info if verbose
#
# If -v 3 or higher, print exported variables in order that they were exported.
@@ -245,27 +428,25 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: VERSION=$VERSION" 1>&2
echo "$0: debug[3]: NAME=$NAME" 1>&2
echo "$0: debug[3]: V_FLAG=$V_FLAG" 1>&2
echo "$0: debug[3]: GIT_TOOL=$GIT_TOOL" 1>&2
echo "$0: debug[3]: TOPDIR=$TOPDIR" 1>&2
echo "$0: debug[3]: REPO_TOP_URL=$REPO_TOP_URL" 1>&2
echo "$0: debug[3]: REPO_URL=$REPO_URL" 1>&2
echo "$0: debug[3]: JSP_REPO=$JSP_REPO" 1>&2
echo "$0: debug[3]: JSONPATH_REPO=$JSONPATH_REPO" 1>&2
echo "$0: debug[3]: JSONPATH_SH=$JSONPATH_SH" 1>&2
echo "$0: debug[3]: FIZZBIN_JSON=$FIZZBIN_JSON" 1>&2
echo "$0: debug[3]: JSP_TOOL=$JSP_TOOL" 1>&2
echo "$0: debug[3]: JSONPATH_SH=$JSONPATH_SH" 1>&2
echo "$0: debug[3]: XPATHJSON_USE=$XPATHJSON_USE" 1>&2
echo "$0: debug[3]: QUICK_CHECK=$QUICK_CHECK" 1>&2
echo "$0: debug[3]: NOOP=$NOOP" 1>&2
echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2
echo "$0: debug[3]: JSON_FILE=$JSON_FILE" 1>&2
fi
# verify the JSON of JSON_FILE
#
if [[ $V_FLAG -ge 7 ]]; then
echo "$0: debug[7]: about to run: $JPARSE_TOOL -q -- $JSON_FILE" 1>&2
fi
if "$JPARSE_TOOL" -q -- "$JSON_FILE"; then
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: valid JSON for: $JSON_FILE" 1>&2
fi
else
echo "$0: ERROR: not valid JSON: $JSON_FILE" 1>&2
exit 7
echo "$0: debug[3]: REPO_NAME=$REPO_NAME" 1>&2
echo "$0: debug[3]: CD_FAILED=$CD_FAILED" 1>&2
echo "$0: debug[3]: BIN_PATH=$BIN_PATH" 1>&2
echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2
echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2
fi
@@ -304,16 +485,49 @@ fi
# canonically format the JSON file
#
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: about to: $JSONPATH_SH -S -A -j -f $JSON_FILE > $TMP_JSON_FILE" 1>&2
fi
"$JSONPATH_SH" -S -A -j -f "$JSON_FILE" > "$TMP_JSON_FILE"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JSONPATH_SH -S -A -j -f $JSON_FILE > $TMP_JSON_FILE failed," \
"error code: $status" 1>&2
exit 1
fi
case "$XPATHJSON_USE" in
# case: use jsp
#
jsp)
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: about to: $JSP_TOOL --indent 4 --format --no-color < $JSON_FILE | sed ..." 1>&2
fi
"$JSP_TOOL" --indent 4 --format --no-color < "$JSON_FILE" |
sed -f "$UNICODE_FIX_SED" -e 's/\(\S\): /\1 : /' > "$TMP_JSON_FILE"
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] ]]; then
echo "$0: ERROR: $JSP_TOOL --indent 4 --format --no-color < $JSON_FILE | sed ... failed", \
"error codes: ${status_codes[*]}" 1>&2
exit 1
fi
;;
# case: use JSONPath.sh
#
JSONPath.sh)
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: about to: $JSONPATH_SH -S -A -j -f $JSON_FILE > $TMP_JSON_FILE" 1>&2
fi
"$JSONPATH_SH" -S -A -j -f "$JSON_FILE" > "$TMP_JSON_FILE"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JSONPATH_SH -S -A -j -f $JSON_FILE > $TMP_JSON_FILE failed," \
"error code: $status" 1>&2
exit 1
fi
;;
# case: we do not know which XPath JSON tool to use
#
*)
echo "$0: ERROR: XPATHJSON_USE is neither jsp nor JSONPath.sh: $XPATHJSON_USE" 1>&2
exit 12
;;
esac
# replace JSON_FILE file with canonically formatted TMP_JSON_FILE if different
@@ -344,13 +558,13 @@ if [[ -z $NOOP ]]; then
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: mv -f -- $TMP_JSON_FILE $JSON_FILE filed," \
"error code: $status" 1>&2
exit 12
exit 13
elif [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: replaced JSON_FILE file: $JSON_FILE" 1>&2
fi
if [[ ! -s $JSON_FILE ]]; then
echo "$0: ERROR: not a non-empty canonical JSON_FILE file: $JSON_FILE" 1>&2
exit 13
exit 14
fi
fi
bin/jprint-wrapper.sh
+312 -63
View File
@@ -2,9 +2,27 @@
#
# jprint-wrapper.sh - canonically format JSON to stdout
#
# We form a wrapper for the jprint(1), a tool that will format JSON in a canonical style to stdout.
# NOTE: jprint(1) is a planned tool based on jfmt(1). As of 2024 July 28 the jfmt(1) tool has not been written,
# so jprint-wrapper.sh uses the JSONPath.sh(1) tool from the recently forked and modified JSONPath.sh tool:
# Print JSON in canonical format to standard output.
#
# We form a wrapper for the jprint(1), a tool that will format a JSON
# file into a canonical style.
#
# NOTE: As of 2024 Sep 09 the jprint(1) tool has not been written,
# so jprint-wrapper.sh uses with the jsp tool from:
#
# https://github.com/kjozsa/jsp
#
# FYI: For macOS, use homebrew to install pipx:
#
# brew install pipx
#
# Next, use pipx to install jsp:
#
# pipx --global install jsp
#
# and then be sure that ~/.local/bin is in your $PATH.
#
# or the JSONPath.sh(1) tool from:
#
# https://github.com/lcn2/JSONPath.sh
#
@@ -92,43 +110,50 @@ shopt -s globstar # enable ** to match all files and zero or more directories an
# set variables referenced in the usage message
#
export VERSION="1.1 2024-08-20"
export VERSION="1.3 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
# Until we have a true jprint tool, we will use the modified JSONPath.sh
#
# BTW: A jprint tool might be a special case of a jfmt tool, with a command line
# optimized for just displaying JSON in a nicely formatted canonical way.
#
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - until we have the jprint command, we must FAKE PARSE IOCCC JSON files - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
#
export JSONPATH_REPO="https://github.com/lcn2/JSONPath.sh"
JSONPATH_SH=$(type -P JSONPath.sh)
export JSONPATH_SH
if [[ -z $JSONPATH_SH ]]; then
echo "$0: FATAL: JSONPath.sh tool is not installed or not in \$PATH" 1>&2
echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2
GIT_TOOL=$(type -P git)
export GIT_TOOL
if [[ -z "$GIT_TOOL" ]]; then
echo "$0: FATAL: git tool is not installed or not in \$PATH" 1>&2
exit 5
fi
# verify JSONPath.sh supports -S -A
"$GIT_TOOL" rev-parse --is-inside-work-tree >/dev/null 2>&1
status="$?"
if [[ $status -eq 0 ]]; then
TOPDIR=$("$GIT_TOOL" rev-parse --show-toplevel)
fi
export TOPDIR
export REPO_TOP_URL="https://github.com/ioccc-src/temp-test-ioccc"
# GitHub puts individual files under the "blob/master" sub-directory.
export REPO_URL="$REPO_TOP_URL/blob/master"
# Until we have a true jfmt tool, we will use wither jsp or the modified JSONPath.sh
#
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - until we have the jprint command, we must use external tools to parse JSON - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
#
export JSP_REPO="https://github.com/kjozsa/jsp"
export JSONPATH_REPO="https://github.com/lcn2/JSONPath.sh"
export FIZZBIN_JSON='"fizzbin"'
if ! "$JSONPATH_SH" -S -A -p >/dev/null 2>&1; then
echo "$0: FATAL: JSONPath.sh tool does not support -S -A -T: $FIZZBIN_JSON" 1>&2
echo "$0: notice: we recommend you obtain and install JSONPath.sh from: $JSONPATH_REPO" 1>&2
exit 5
fi <<< "$FIZZBIN_JSON"
export JSP_TOOL=""
JSP_TOOL=$(type -P jsp)
export JSONPATH_SH=""
JSONPATH_SH=$(type -P JSONPath.sh)
# set the defaiult XPath for JSON
export XPATHJSON_USE="jsp"
export QUICK_CHECK=""
# set usage message
#
export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] [-j JSONPath.sh] [file.json]
export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] [-t tool] [-Q] [file.json]
-h print help message and exit
-v level set verbosity level (def level: 0)
@@ -137,17 +162,18 @@ export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] [-j JSONPath.sh] [file.js
-n do nothing (same as -N) (def: do the action)
-N do not process file, just parse arguments and ignore the file (def: process the file)
-j JSONPath.sh path to the JSONPath.sh tool (not the wrapper) (def: $JSONPATH_SH)
-t tool tool to use: jsp or JSONPath.sh (def: try jsp, otherwise try JSONPath.sh)
-Q quick check tool using trivial input (def: do not)
file.json The JSON file to write, in canonical form, to stdout (def: read stdin)
Exit codes:
0 all OK
1 JSONPath.sh exited non-zero
1 some internal tool exited non-zero
2 -h and help string printed or -V and version string printed
3 command line error
4 bash version is too old
5 internal tool not found
5 some internal tool not found
6 file.json does not exist or is not readable
>= 10 internal error
@@ -162,7 +188,7 @@ export DO_NOT_PROCESS=
# parse command line
#
while getopts :hv:VnNj: flag; do
while getopts :hv:VnNt:Q flag; do
case "$flag" in
h) echo "$USAGE" 1>&2
exit 2
@@ -176,8 +202,19 @@ while getopts :hv:VnNj: flag; do
;;
N) DO_NOT_PROCESS="-N"
;;
j) JSONPATH_SH="$OPTARG"
;;
t) # validate -t tool
case "$OPTARG" in
jsp) XPATHJSON_USE="jsp" ;;
JSONPath.sh) XPATHJSON_USE="JSONPath.sh" ;;
*) echo "$0: ERROR: unknown -t option: $OPTARG" 1>&2
echo 1>&2
echo "$USAGE" 1>&2
exit 3
;;
esac
;;
Q) QUICK_CHECK="true"
;;
\?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2
echo 1>&2
echo "$USAGE" 1>&2
@@ -234,6 +271,155 @@ if [[ -n $JSON_FILE ]]; then
fi
# verify that we have a topdir directory
#
REPO_NAME=$(basename "$REPO_TOP_URL")
export REPO_NAME
if [[ -z $TOPDIR ]]; then
echo "$0: ERROR: cannot find top of git repo directory" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
if [[ ! -e $TOPDIR ]]; then
echo "$0: ERROR: TOPDIR does not exist: $TOPDIR" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
if [[ ! -d $TOPDIR ]]; then
echo "$0: ERROR: TOPDIR is not a directory: $TOPDIR" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
# cd to topdir
#
if [[ ! -e $TOPDIR ]]; then
echo "$0: ERROR: cannot cd to non-existent path: $TOPDIR" 1>&2
exit 6
fi
if [[ ! -d $TOPDIR ]]; then
echo "$0: ERROR: cannot cd to a non-directory: $TOPDIR" 1>&2
exit 6
fi
export CD_FAILED
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to: cd $TOPDIR" 1>&2
fi
cd "$TOPDIR" || CD_FAILED="true"
if [[ -n $CD_FAILED ]]; then
echo "$0: ERROR: cd $TOPDIR failed" 1>&2
exit 6
fi
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: now in directory: $(/bin/pwd)" 1>&2
fi
# verify that we have a bin subdirectory
#
export BIN_PATH="$TOPDIR/bin"
if [[ ! -d $BIN_PATH ]]; then
echo "$0: ERROR: bin is not a directory under topdir: $BIN_PATH" 1>&2
exit 6
fi
export BIN_DIR="bin"
# verify that the bin/unicode-fix.sed tool is executable
#
export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed"
if [[ ! -e $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -f $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -r $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
# validate the XPath for JSON tool
#
case "$XPATHJSON_USE" in
# case: we will use JSONPath.sh
#
JSONPath.sh)
# try JSONPath.sh
#
if [[ -z $JSONPATH_SH || ! -x $JSONPATH_SH ]]; then
echo "$0: ERROR: JSONPath.sh tool is not installed, is not executable, or not in \$PATH" 1>&2
echo "$0: notice: obtain jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
fi
# case: -Q - perform quick check
#
if [[ -n $QUICK_CHECK ]]; then
# verify JSONPath.sh supports -S -A -p
if ! "$JSONPATH_SH" -S -A -p >/dev/null 2>&1; then
echo "$0: ERROR: JSONPath.sh tool does not support -S -A: $FIZZBIN_JSON" 1>&2
echo "$0: notice: we recommend you obtain and install jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: otherwise install JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
fi <<< "$FIZZBIN_JSON"
fi
;;
# case: we will use jsp
#
jsp)
# try jsp
#
if [[ -z $JSP_TOOL || ! -x $JSP_TOOL ]]; then
echo "$0: ERROR: jsp.sh tool is not installed, is not executable, or not in \$PATH" 1>&2
echo "$0: notice: obtain jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
fi
# case: -Q - perform quick check
#
if [[ -n $QUICK_CHECK ]]; then
# verify jsp
if ! "$JSP_TOOL" --indent 4 --format --no-color >/dev/null 2>&1; then
echo "$0: ERROR: jsp tool does not support --indent 4 --format --no-color: $FIZZBIN_JSON" 1>&2
echo "$0: notice: you might need to update and reinstall jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: or you might try to obtain and install JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
fi <<< "$FIZZBIN_JSON"
fi
;;
# case: we do not have an XPath for JSON tool
#
*)
echo "$0: ERROR: cannot find an XPath JSON tool to use" 1>&2
echo "$0: notice: we recommend you obtain and install jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: otherwise install JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
;;
esac
# print running info if verbose
#
# If -v 3 or higher, print exported variables in order that they were exported.
@@ -242,12 +428,25 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: VERSION=$VERSION" 1>&2
echo "$0: debug[3]: NAME=$NAME" 1>&2
echo "$0: debug[3]: V_FLAG=$V_FLAG" 1>&2
echo "$0: debug[3]: GIT_TOOL=$GIT_TOOL" 1>&2
echo "$0: debug[3]: TOPDIR=$TOPDIR" 1>&2
echo "$0: debug[3]: REPO_TOP_URL=$REPO_TOP_URL" 1>&2
echo "$0: debug[3]: REPO_URL=$REPO_URL" 1>&2
echo "$0: debug[3]: JSP_REPO=$JSP_REPO" 1>&2
echo "$0: debug[3]: JSONPATH_REPO=$JSONPATH_REPO" 1>&2
echo "$0: debug[3]: JSONPATH_SH=$JSONPATH_SH" 1>&2
echo "$0: debug[3]: FIZZBIN_JSON=$FIZZBIN_JSON" 1>&2
echo "$0: debug[3]: JSP_TOOL=$JSP_TOOL" 1>&2
echo "$0: debug[3]: JSONPATH_SH=$JSONPATH_SH" 1>&2
echo "$0: debug[3]: XPATHJSON_USE=$XPATHJSON_USE" 1>&2
echo "$0: debug[3]: QUICK_CHECK=$QUICK_CHECK" 1>&2
echo "$0: debug[3]: NOOP=$NOOP" 1>&2
echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2
echo "$0: debug[3]: JSON_FILE=$JSON_FILE" 1>&2
echo "$0: debug[3]: REPO_NAME=$REPO_NAME" 1>&2
echo "$0: debug[3]: CD_FAILED=$CD_FAILED" 1>&2
echo "$0: debug[3]: BIN_PATH=$BIN_PATH" 1>&2
echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2
echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2
fi
@@ -271,38 +470,88 @@ if [[ -n $NOOP ]]; then
fi
# case: processing JSON from standard input
# print the JSON file in canonical format
#
if [[ -z $JSON_FILE ]]; then
case "$XPATHJSON_USE" in
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: about to: $JSONPATH_SH -S -A -j" 1>&2
fi
"$JSONPATH_SH" -S -A -j
status="$?"
# case: use jsp
#
jsp)
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JSONPATH_SH -S -A -j failed," \
"error code: $status" 1>&2
exit 1
fi
# case: processing JSON from stdin via jsp
#
if [[ -z $JSON_FILE ]]; then
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: about to: $JSP_TOOL --indent 4 --format --no-color | sed ..." 1>&2
fi
"$JSP_TOOL" --indent 4 --format --no-color |
sed -f "$UNICODE_FIX_SED" -e 's/\(\S\): /\1 : /'
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] ]]; then
echo "$0: ERROR: $JSP_TOOL --indent 4 --format --no-color | sed ... failed", \
"error codes: ${status_codes[*]}" 1>&2
exit 1
fi
# case: processing JSON from a file
#
else
# processing JSON from a file via jsp
#
else
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: about to: $JSP_TOOL --indent 4 --format --no-color < $JSON_FILE | sed ..." 1>&2
fi
"$JSP_TOOL" --indent 4 --format --no-color < "$JSON_FILE" |
sed -f "$UNICODE_FIX_SED" -e 's/\(\S\): /\1 : /'
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] ]]; then
echo "$0: ERROR: $JSP_TOOL --indent 4 --format --no-color < $JSON_FILE | sed ... failed", \
"error codes: ${status_codes[*]}" 1>&2
exit 1
fi
fi
;;
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: about to: $JSONPATH_SH -S -A -j -f $JSON_FILE" 1>&2
fi
"$JSONPATH_SH" -S -A -j -f "$JSON_FILE"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JSONPATH_SH -S -A -j -f $JSON_FILE failed," \
"error code: $status" 1>&2
exit 1
fi
# case: use JSONPath.sh
#
JSONPath.sh)
fi
# case: processing JSON from stdin via JSONPath.sh
#
if [[ -z $JSON_FILE ]]; then
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: about to: $JSONPATH_SH -S -A -j" 1>&2
fi
"$JSONPATH_SH" -S -A -j
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JSONPATH_SH -S -A -j failed," \
"error code: $status" 1>&2
exit 1
fi
# processing JSON from a file via JSONPath.sh
#
else
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: about to: $JSONPATH_SH -S -A -j -f $JSON_FILE" 1>&2
fi
"$JSONPATH_SH" -S -A -j -f "$JSON_FILE"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JSONPATH_SH -S -A -j -f $JSON_FILE failed," \
"error code: $status" 1>&2
exit 1
fi
fi
;;
# case: we do not know which XPath JSON tool to use
#
*)
echo "$0: ERROR: XPATHJSON_USE is neither jsp nor JSONPath.sh: $XPATHJSON_USE" 1>&2
exit 12
;;
esac
# All Done!!! All Done!!! -- Jessica Noll, Age 2
bin/jval-wrapper.sh
+462 -76
View File
@@ -106,47 +106,55 @@ shopt -s globstar # enable ** to match all files and zero or more directories an
# set variables referenced in the usage message
#
export VERSION="1.0.4 2024-08-20"
export VERSION="1.1 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
# Until we have a true jval tool, we will use the modified JSONPath.sh
#
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - until we have the jval command, we must FAKE PARSE IOCCC JSON files - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
#
export JSONPATH_REPO="https://github.com/lcn2/JSONPath.sh"
JSONPATH_SH=$(type -P JSONPath.sh)
export JSONPATH_SH
if [[ -z $JSONPATH_SH ]]; then
echo "$0: FATAL: JSONPath.sh tool is not installed or not in \$PATH" 1>&2
echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2
GIT_TOOL=$(type -P git)
export GIT_TOOL
if [[ -z "$GIT_TOOL" ]]; then
echo "$0: FATAL: git tool is not installed or not in \$PATH" 1>&2
exit 5
fi
# verify JSONPath.sh supports -S -A
export FIZZBIN_JSON='"fizzbin"'
if ! "$JSONPATH_SH" -S -A -p >/dev/null 2>&1; then
echo "$0: FATAL: JSONPath.sh tool does not support -S -A: $FIZZBIN_JSON" 1>&2
echo "$0: notice: we recommend you obtain and install JSONPath.sh from: $JSONPATH_REPO" 1>&2
exit 5
fi <<< "$FIZZBIN_JSON"
export JSONPATH_ARG
export PATTERN='$*'
"$GIT_TOOL" rev-parse --is-inside-work-tree >/dev/null 2>&1
status="$?"
if [[ $status -eq 0 ]]; then
TOPDIR=$("$GIT_TOOL" rev-parse --show-toplevel)
fi
export TOPDIR
export REPO_TOP_URL="https://github.com/ioccc-src/temp-test-ioccc"
# GitHub puts individual files under the "blob/master" sub-directory.
export REPO_URL="$REPO_TOP_URL/blob/master"
# Until we have a true jval tool, we will use wither jsp or the modified JSONPath.sh
#
unset OPTION
declare -ag OPTION
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - until we have the jval command, we use external tools to get data from JSON - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
#
export JSP_REPO="https://github.com/kjozsa/jsp"
export JSONPATH_REPO="https://github.com/lcn2/JSONPath.sh"
export FIZZBIN_JSON='"fizzbin"'
export JSP_TOOL=""
JSP_TOOL=$(type -P jsp)
export JSONPATH_SH=""
JSONPATH_SH=$(type -P JSONPath.sh)
# set the defaiult XPath for JSON
export XPATHJSON_USE="jsp"
export JSONPATH_ARG
export PATTERN=""
#
export PRINT_ONLY_VALUES=""
export REMOVE_QUOTES=""
export QUICK_CHECK=""
# set usage message
#
export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N]
[-w] [-b] [-i] [-u]
[-j JSONPath.sh] file.json [pattern]
export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] [-b] [-q] [-t tool] [-Q] file.json [pattern]
-h print help message and exit
-v level set verbosity level (def level: 0)
@@ -155,12 +163,11 @@ export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N]
-n do nothing (same as -N) (def: do the action)
-N do not process file, just parse arguments and ignore the file (def: process the file)
-w match whole words only for filter script expression (def: match substrings)
-b only print values (def: print both JSON member and JSON value)
-i match insensitive (def: match case)
-u strip unnecessary leading path element (def: print the entire element path)
-b print only values (def: print both JSON member and JSON value)
-q remove enclosing double quotes (def: keep any enclosing double quotes)
-j JSONPath.sh path to the JSONPath.sh tool (not the wrapper) (def: $JSONPATH_SH)
-t tool tool to use: jsp or JSONPath.sh (def: try jsp, otherwise try JSONPath.sh)
-Q quick check tool using trivial input (def: do not)
file.json JSON file to read, - ==> read stdin
pattern JSONPath query (def: $PATTERN}
@@ -186,7 +193,7 @@ export DO_NOT_PROCESS=
# parse command line
#
while getopts :hv:VnNwbiuj: flag; do
while getopts :hv:VnNbqt:Q flag; do
case "$flag" in
h) echo "$USAGE" 1>&2
exit 2
@@ -200,16 +207,23 @@ while getopts :hv:VnNwbiuj: flag; do
;;
N) DO_NOT_PROCESS="-N"
;;
j) JSONPATH_SH="$OPTARG"
;;
w) OPTION+=("-w")
;;
b) OPTION+=("-b")
;;
i) OPTION+=("-i")
;;
u) OPTION+=("-u")
b) PRINT_ONLY_VALUES="-b"
;;
q) REMOVE_QUOTES="true"
;;
t) # validate -t tool
case "$OPTARG" in
jsp) XPATHJSON_USE="jsp" ;;
JSONPath.sh) XPATHJSON_USE="JSONPath.sh" ;;
*) echo "$0: ERROR: unknown -t option: $OPTARG" 1>&2
echo 1>&2
echo "$USAGE" 1>&2
exit 3
;;
esac
;;
Q) QUICK_CHECK="true"
;;
\?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2
echo 1>&2
echo "$USAGE" 1>&2
@@ -251,6 +265,9 @@ case "$#" in
;;
esac
#
if [[ $JSON_FILE == "-" ]]; then
JSON_FILE=""
fi
if [[ -n $JSON_FILE ]]; then
if [[ ! -e $JSON_FILE ]]; then
echo "$0: ERROR: JSON_FILE does not exist: $JSON_FILE" 1>&2
@@ -267,6 +284,155 @@ if [[ -n $JSON_FILE ]]; then
fi
# verify that we have a topdir directory
#
REPO_NAME=$(basename "$REPO_TOP_URL")
export REPO_NAME
if [[ -z $TOPDIR ]]; then
echo "$0: ERROR: cannot find top of git repo directory" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
if [[ ! -e $TOPDIR ]]; then
echo "$0: ERROR: TOPDIR does not exist: $TOPDIR" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
if [[ ! -d $TOPDIR ]]; then
echo "$0: ERROR: TOPDIR is not a directory: $TOPDIR" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
# cd to topdir
#
if [[ ! -e $TOPDIR ]]; then
echo "$0: ERROR: cannot cd to non-existent path: $TOPDIR" 1>&2
exit 6
fi
if [[ ! -d $TOPDIR ]]; then
echo "$0: ERROR: cannot cd to a non-directory: $TOPDIR" 1>&2
exit 6
fi
export CD_FAILED
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to: cd $TOPDIR" 1>&2
fi
cd "$TOPDIR" || CD_FAILED="true"
if [[ -n $CD_FAILED ]]; then
echo "$0: ERROR: cd $TOPDIR failed" 1>&2
exit 6
fi
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: now in directory: $(/bin/pwd)" 1>&2
fi
# verify that we have a bin subdirectory
#
export BIN_PATH="$TOPDIR/bin"
if [[ ! -d $BIN_PATH ]]; then
echo "$0: ERROR: bin is not a directory under topdir: $BIN_PATH" 1>&2
exit 6
fi
export BIN_DIR="bin"
# verify that the bin/unicode-fix.sed tool is executable
#
export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed"
if [[ ! -e $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -f $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -r $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
# validate the XPath for JSON tool
#
case "$XPATHJSON_USE" in
# case: we will use JSONPath.sh
#
JSONPath.sh)
# try JSONPath.sh
#
if [[ -z $JSONPATH_SH || ! -x $JSONPATH_SH ]]; then
echo "$0: ERROR: JSONPath.sh tool is not installed, is not executable, or not in \$PATH" 1>&2
echo "$0: notice: obtain jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
fi
# case: -Q - perform quick check
#
if [[ -n $QUICK_CHECK ]]; then
# verify JSONPath.sh supports -S -A -p
if ! "$JSONPATH_SH" -S -A -p >/dev/null 2>&1; then
echo "$0: ERROR: JSONPath.sh tool does not support -S -A: $FIZZBIN_JSON" 1>&2
echo "$0: notice: we recommend you obtain and install jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: otherwise install JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
fi <<< "$FIZZBIN_JSON"
fi
;;
# case: we will use jsp
#
jsp)
# try jsp
#
if [[ -z $JSP_TOOL || ! -x $JSP_TOOL ]]; then
echo "$0: ERROR: jsp.sh tool is not installed, is not executable, or not in \$PATH" 1>&2
echo "$0: notice: obtain jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
fi
# case: -Q - perform quick check
#
if [[ -n $QUICK_CHECK ]]; then
# verify jsp
if ! "$JSP_TOOL" --indent 4 --format --no-color >/dev/null 2>&1; then
echo "$0: ERROR: jsp tool does not support --indent 4 --format --no-color: $FIZZBIN_JSON" 1>&2
echo "$0: notice: you might need to update and reinstall jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: or you might try to obtain and install JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
fi <<< "$FIZZBIN_JSON"
fi
;;
# case: we do not have an XPath for JSON tool
#
*)
echo "$0: ERROR: cannot find an XPath JSON tool to use" 1>&2
echo "$0: notice: we recommend you obtain and install jsp from: $JSP_REPO" 1>&2
echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2
echo "$0: notice: otherwise install JSONPath.sh from: $JSONPATH_REPO" 1>&2
echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2
exit 5
;;
esac
# print running info if verbose
#
# If -v 3 or higher, print exported variables in order that they were exported.
@@ -275,17 +441,28 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: VERSION=$VERSION" 1>&2
echo "$0: debug[3]: NAME=$NAME" 1>&2
echo "$0: debug[3]: V_FLAG=$V_FLAG" 1>&2
echo "$0: debug[3]: GIT_TOOL=$GIT_TOOL" 1>&2
echo "$0: debug[3]: TOPDIR=$TOPDIR" 1>&2
echo "$0: debug[3]: REPO_TOP_URL=$REPO_TOP_URL" 1>&2
echo "$0: debug[3]: REPO_URL=$REPO_URL" 1>&2
echo "$0: debug[3]: JSP_REPO=$JSP_REPO" 1>&2
echo "$0: debug[3]: JSONPATH_REPO=$JSONPATH_REPO" 1>&2
echo "$0: debug[3]: JSONPATH_SH=$JSONPATH_SH" 1>&2
echo "$0: debug[3]: FIZZBIN_JSON=$FIZZBIN_JSON" 1>&2
echo "$0: debug[3]: JSONPATH_ARG=$JSONPATH_ARG" 1>&2
for index in "${!OPTION[@]}"; do
echo "$0: debug[$V_FLAG]: $V_FLAG: OPTION[$index]=${OPTION[$index]}" 1>&2
done
echo "$0: debug[3]: JSP_TOOL=$JSP_TOOL" 1>&2
echo "$0: debug[3]: JSONPATH_SH=$JSONPATH_SH" 1>&2
echo "$0: debug[3]: XPATHJSON_USE=$XPATHJSON_USE" 1>&2
echo "$0: debug[3]: PATTERN=$PATTERN" 1>&2
echo "$0: debug[3]: PRINT_ONLY_VALUES=$PRINT_ONLY_VALUES" 1>&2
echo "$0: debug[3]: REMOVE_QUOTES=$REMOVE_QUOTES" 1>&2
echo "$0: debug[3]: QUICK_CHECK=$QUICK_CHECK" 1>&2
echo "$0: debug[3]: NOOP=$NOOP" 1>&2
echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2
echo "$0: debug[3]: JSON_FILE=$JSON_FILE" 1>&2
echo "$0: debug[3]: PATTERN=$PATTERN" 1>&2
echo "$0: debug[3]: REPO_NAME=$REPO_NAME" 1>&2
echo "$0: debug[3]: CD_FAILED=$CD_FAILED" 1>&2
echo "$0: debug[3]: BIN_PATH=$BIN_PATH" 1>&2
echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2
echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2
fi
@@ -309,36 +486,245 @@ if [[ -n $NOOP ]]; then
fi
# case: processing JSON from standard input
# canonically process JSON from standard input via jsp
#
if [[ -z $JSON_FILE ]]; then
case "$XPATHJSON_USE" in
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: about to: $JSONPATH_SH ${OPTION[*]} -- $PATTERN" 1>&2
fi
"$JSONPATH_SH" "${OPTION[@]}" -- "$PATTERN"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JSONPATH_SH ${OPTION[*]} -- $PATTERN failed," \
"error code: $status" 1>&2
exit 1
fi
# case: processing JSON from a file
# case: use jsp
#
else
jsp)
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: about to: $JSONPATH_SH ${OPTION[*]} -f $JSON_FILE -- $PATTERN" 1>&2
# if not set, set the default jsp pattern
#
PATTERN="${PATTERN:=$}"
# case: jsp processing from standard input
#
if [[ -z $JSON_FILE ]]; then
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: about to: $JSP_TOOL --no-color --format --indent 0 -- '$PATTERN'" 1>&2
fi
"$JSP_TOOL" --no-color --format --indent 0 -- "$PATTERN"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JSP_TOOL --no-color --format --indent 0 -- '$PATTERN' failed", \
"error code: $status" 1>&2
exit 1
fi
# case: jsp processing from a file
#
else
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: about to: $JSP_TOOL --no-color --format --indent 0 -- '$PATTERN' < $JSON_FILE" 1>&2
fi
"$JSP_TOOL" --no-color --format --indent 0 -- "$PATTERN" < "$JSON_FILE"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JSP_TOOL --no-color --format --indent 0 -- '$PATTERN' < $JSON_FILE failed", \
"error code: $status" 1>&2
exit 1
fi
# case: print only values from jsp output
#
fi | if [[ -n $PRINT_ONLY_VALUES ]]; then
# case: -q -b (print only values and remove double quotes) via jsp
#
if [[ -n $REMOVE_QUOTES ]]; then
sed -E -e '/^\s*[]{}[],?\s*$/d' \
-e 's/^\s*("[^"]*[^\\]*"):\s+/\1\t/' \
-e 's/^"[^\t]*"\t//' \
-e 's/,?$//' \
-e 's/^"//' \
-e 's/",?$//' \
-e '/^\[$/d'
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: pipe from -t jsp options: -q -b: sed -E -e ... failed", \
"error code: $status" 1>&2
exit 1
fi
# case: -b (print only values) via jsp
#
else
sed -E -e '/^\s*[]{}[],?\s*$/d' \
-e 's/^\s*("[^"]*[^\\]*"):\s+/\1\t/' \
-e 's/^"[^\t]*"\t//' \
-e 's/,?$//' \
-e '/^\[$/d'
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: pipe from -t jsp options: -b: sed -E -e ... failed", \
"error code: $status" 1>&2
exit 1
fi
fi
# case: print both JSON member and JSON value from jsp output
#
else
# case: -q without -b (print both JSON member and JSON value and remove double quotes) via jsp
#
if [[ -n $REMOVE_QUOTES ]]; then
sed -E -e '/^\s*[]{}[],?\s*$/d' \
-e 's/^\s*("[^"]*[^\\]*"):\s+/\1\t/' \
-e 's/,?$//' \
-e 's/^"//' \
-e 's/"\t"?/\t/' \
-e 's/"$//' \
-e '/\t\[$/d'
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: pipe from -t jsp options: -q: sed -E -e ... failed", \
"error code: $status" 1>&2
exit 1
fi
# case: without -b (print only values) via jsp
#
else
sed -E -e '/^\s*[]{}[],?\s*$/d' \
-e 's/^\s*("[^"]*[^\\]*"):\s+/\1\t/' \
-e 's/,?$//' \
-e '/\t\[$/d'
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: pipe from -t jsp: sed -E -e ... failed", \
"error code: $status" 1>&2
exit 1
fi
fi
fi
"$JSONPATH_SH" "${OPTION[@]}" -f "$JSON_FILE" -- "$PATTERN"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JSONPATH_SH ${OPTION[*]} -f $JSON_FILE -- $PATTERN failed," \
"error code: $status" 1>&2
exit 1
;;
# case: use JSONPath.sh
#
JSONPath.sh)
# if not set, set the default JSONPath.sh pattern
#
PATTERN="${PATTERN:=$.*}"
# case: JSONPath.sh processing from standard input
#
if [[ -z $JSON_FILE ]]; then
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: about to: $JSONPATH_SH -w -u -- '$PATTERN'" 1>&2
fi
"$JSONPATH_SH" -w -u -- "$PATTERN"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JSONPATH_SH -w -u -- '$PATTERN' failed", \
"error code: $status" 1>&2
exit 1
fi
# case: JSONPath.sh processing from a file
#
else
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: about to: $JSONPATH_SH -w -u -f $JSON_FILE -- '$PATTERN'" 1>&2
fi
"$JSONPATH_SH" -w -u -f "$JSON_FILE" -- "$PATTERN"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JSONPATH_SH -w -u -f $JSON_FILE -- '$PATTERN' failed", \
"error code: $status" 1>&2
exit 1
fi
# case: print only values from JSONPath.sh output
#
fi | if [[ -n $PRINT_ONLY_VALUES ]]; then
# case: -q -b (print only values and remove double quotes and []'s) via JSONPath.sh
#
if [[ -n $REMOVE_QUOTES ]]; then
sed -E -e 's/^\S*\s*//' \
-e 's/^"//' \
-e 's/"$//'
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: pipe from -t jJSONPath.sh options: -q -b: sed -E -e ... failed", \
"error code: $status" 1>&2
exit 1
fi
# case: -b (print only values) via JSONPath.sh
#
else
sed -E -e 's/^\S*\s*//'
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: pipe from -t jJSONPath.sh options: -b: sed -E -e ... failed", \
"error code: $status" 1>&2
exit 1
fi
fi
# case: print both JSON member and JSON value from JSONPath.sh output
#
else
# case: -q without -b (print both JSON member and JSON value and remove double quotes and []'s) via JSONPath.sh
#
if [[ -n $REMOVE_QUOTES ]]; then
sed -E -e 's/^\s+//' \
-e 's/\s+$//' \
-e 's/^\["?//' \
-e 's/^(\S*),"/\1,/' \
-e 's/\]\t/\t/' \
-e 's/"\t"?/\t/' \
-e 's/"$//'
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: pipe from -t JSONPath.sh options: -q: sed -E -e ... failed", \
"error code: $status" 1>&2
exit 1
fi
# case: without -b (print only values) via JSONPath.sh
#
else
sed -E -e 's/^\s+//' \
-e 's/\s+$//'
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: pipe from -t JSONPath.sh: sed -E -e ... failed", \
"error code: $status" 1>&2
exit 1
fi
fi
fi
fi
;;
# case: we do not know which XPath JSON tool to use
#
*)
echo "$0: ERROR: XPATHJSON_USE is neither jsp nor JSONPath.sh: $XPATHJSON_USE" 1>&2
exit 10
;;
esac
# All Done!!! All Done!!! -- Jessica Noll, Age 2
bin/manifest.entry.json.awk
+60 -22
View File
@@ -23,7 +23,7 @@ BEGIN {
# setup
#
VERSION="1.1.2 2024-07-28"
VERSION="1.2 2024-09-17"
found_manifest_array = 0; # 1 ==> we found the manifest ARRAY
within_manifest_array = 0; # 1 ==> found start of "manifest" : [ JSON array
begin_manifest_element = 0; # 1 ==> we have found the start of the manifest array element
@@ -121,7 +121,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^"
#
$1 = ""; # remove 1st field
$2 = ""; # remove 2nd field
gsub(/^[[:space:]]+/, "", $0); # any leading whitespace
gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace
file_path = $0;
# Remove any trailing ,
@@ -155,7 +155,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^"
#
$1 = ""; # remove 1st field
$2 = ""; # remove 2nd field
gsub(/^[[:space:]]+/, "", $0); # any leading whitespace
gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace
inventory_order = $0;
# Remove any trailing ,
@@ -189,7 +189,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^"
#
$1 = ""; # remove 1st field
$2 = ""; # remove 2nd field
gsub(/^[[:space:]]+/, "", $0); # any leading whitespace
gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace
OK_to_edit = $0;
# Remove any trailing ,
@@ -223,7 +223,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^"
#
$1 = ""; # remove 1st field
$2 = ""; # remove 2nd field
gsub(/^[[:space:]]+/, "", $0); # any leading whitespace
gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace
display_as = $0;
# Remove any trailing ,
@@ -257,7 +257,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^"
#
$1 = ""; # remove 1st field
$2 = ""; # remove 2nd field
gsub(/^[[:space:]]+/, "", $0); # any leading whitespace
gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace
display_via_github = $0;
# Remove any trailing ,
@@ -291,7 +291,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^"
#
$1 = ""; # remove 1st field
$2 = ""; # remove 2nd field
gsub(/^[[:space:]]+/, "", $0); # any leading whitespace
gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace
entry_text = $0;
# Remove any trailing ,
@@ -300,12 +300,50 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^"
entry_text = substr(entry_text, 1, length(entry_text)-1);
}
# so we can single quote the entry_text as a single string, we protect any single quotes within it
#
jstrdecode_arg = entry_text;
gsub(/'/, "'\"'\"'", jstrdecode_arg);
# form the jstrdecode command
#
if (jstrdecode_arg ~ /^".*"$/) {
jstrdecode_cmd = "jstrdecode -Q -- " "'" substr(jstrdecode_arg, 2, length(jstrdecode_arg)-2) "'";
} else {
jstrdecode_cmd = "jstrdecode -Q -- " "'" jstrdecode_arg "'";
}
jstrdecode_cmd = jstrdecode_cmd " 2>/dev/null; echo $?";
# JSON decode entry_text
#
numLines = 0;
result = "";
exit_code = 0;
# process each pipe line
while ((jstrdecode_cmd | getline pipe_output) > 0) {
if (numLines++ == 0) {
# case: 1st line is from jstrdecode
result = pipe_output;
} else {
# case: 2nd/line line is the jstrdecode exit code
exit_code = pipe_output;
}
}
close(jstrdecode_cmd);
# validate exit code from jstrdecode
#
if (length(exit_code) == 0 || exit_code != 0) {
print "ERROR: unable to jstrdecode entry_text:", entry_text > "/dev/stderr";
exit 213; # END section will output ERROR message about cannot JSON decode entry_text
}
# remove any enclosing double quotes
#
if (entry_text ~ /^".*"$/) {
unquote_entry_text = substr(entry_text, 2, length(entry_text)-2);
if (result ~ /^".*"$/) {
unquote_entry_text = substr(result, 2, length(result)-2);
} else {
unquote_entry_text = entry_text;
unquote_entry_text = result;
}
}
@@ -317,42 +355,42 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF == 1 && $1 ~ /^}
#
if (length(file_path) == 0) {
print "ERROR: manifest array JSON member did not have a file_path JSON member in:", ARGV[1] > "/dev/stderr";
exit 213;
exit 214;
}
# verify we have seen a inventory_order JSON member
#
if (length(inventory_order) == 0) {
print "ERROR: manifest array JSON member did not have a inventory_order JSON member in:", ARGV[1] > "/dev/stderr";
exit 214;
exit 215;
}
# verify we have seen a OK_to_edit JSON member
#
if (length(OK_to_edit) == 0) {
print "ERROR: manifest array JSON member did not have a OK_to_edit JSON member in:", ARGV[1] > "/dev/stderr";
exit 215;
exit 216;
}
# verify we have seen a display_as JSON member
#
if (length(display_as) == 0) {
print "ERROR: manifest array JSON member did not have a display_as JSON member in:", ARGV[1] > "/dev/stderr";
exit 216;
exit 217;
}
# verify we have seen a display_via_github JSON member
#
if (length(display_via_github) == 0) {
print "ERROR: manifest array JSON member did not have a display_via_github JSON member in:", ARGV[1] > "/dev/stderr";
exit 217;
exit 218;
}
# verify we have seen a entry_text JSON member
#
if (length(entry_text) == 0) {
print "ERROR: manifest array JSON member did not have a entry_text JSON member in:", ARGV[1] > "/dev/stderr";
exit 218;
exit 219;
}
# print manifest information
@@ -395,39 +433,39 @@ END {
#
if (length(github) == 0) {
print "ERROR: github variable not set, call with -v github=REPO_URL" > "/dev/stderr";
exit 219; # use 200 to match exit code from BEGIN
exit 220; # use 200 to match exit code from BEGIN
}
# case: we quit because we could not determine dir from ARGV[1]
#
if (length(dir) == 0) {
print "ERROR: cannot determine dir from:", ARGV[1] > "/dev/stderr";
exit 220; # END section will output ERROR message about cannot find dir
exit 221; # END section will output ERROR message about cannot find dir
}
if (length(YYYY) == 0) {
print "ERROR: cannot determine YYYY from:", ARGV[1] > "/dev/stderr";
exit 221; # END section will output ERROR message about cannot find YYYY
exit 222; # END section will output ERROR message about cannot find YYYY
}
# case: no manifest array was found
#
if (!found_manifest_array) {
print "ERROR: we did not find a manifest JSON array in:", ARGV[1] > "/dev/stderr";
exit 222;
exit 223;
}
# case: manifest array was found, but manifest array element did not finish
#
if (begin_manifest_element) {
print "ERROR: manifest JSON array element did not end in:", ARGV[1] > "/dev/stderr";
exit 223;
exit 224;
}
# case: manifest array did not end
#
if (within_manifest_array) {
print "ERROR: manifest JSON array did not end in:", ARGV[1] > "/dev/stderr";
exit 224;
exit 225;
}
}
bin/output-index-author.sh
+38 -12
View File
@@ -104,7 +104,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an
# set variables referenced in the usage message
#
export VERSION="1.8 2024-08-17"
export VERSION="1.9 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
@@ -188,6 +188,7 @@ export DO_NOT_PROCESS=
function output_author_handles
{
local ENTRY_JSON_PATH; # the .entry.json path
local PATTERN; # XPath for JSON pattern
# parse args
#
@@ -211,10 +212,11 @@ function output_author_handles
# extract author handles from .entry.json
#
"$JVAL_WRAPPER" -w -b "$ENTRY_JSON_PATH" '$..author_handle'
PATTERN='$..author_handle'
"$JVAL_WRAPPER" -b -q "$ENTRY_JSON_PATH" "$PATTERN"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JVAL_WRAPPER -w -b $ENTRY_JSON_PATH \$..author_handle failed," \
echo "$0: ERROR: in output_author_handles: $JVAL_WRAPPER -b -q $ENTRY_JSON_PATH '$PATTERN' failed," \
"error: $status" 1>&2
return 5
fi
@@ -236,6 +238,7 @@ function output_author_handles
function output_entry_ids
{
local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path
local PATTERN; # XPath for JSON pattern
# parse args
#
@@ -259,10 +262,11 @@ function output_entry_ids
# extract entry ids from the author/author_handle.json file
#
"$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..entry_id'
PATTERN='$..entry_id'
"$JVAL_WRAPPER" -b -q "$AUTHOR_HANDLE_JSON_PATH" "$PATTERN"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JVAL_WRAPPER -w -b $AUTHOR_HANDLE_JSON_PATH \$..entry_id failed," \
echo "$0: ERROR: in output_entry_ids: $JVAL_WRAPPER -b -q $AUTHOR_HANDLE_JSON_PATH '$PATTERN' failed," \
"error: $status" 1>&2
return 5
fi
@@ -283,6 +287,7 @@ function output_entry_ids
function output_full_name
{
local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path
local PATTERN; # XPath for JSON pattern
# parse args
#
@@ -306,11 +311,12 @@ function output_full_name
# extract Full Name from the author/author_handle.json file
#
"$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..full_name'
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JVAL_WRAPPER -w -b $AUTHOR_HANDLE_JSON_PATH \$..full_name failed," \
"error: $status" 1>&2
PATTERN='$..full_name'
"$JVAL_WRAPPER" -b -q "$AUTHOR_HANDLE_JSON_PATH" "$PATTERN" | sed -f "$UNICODE_FIX_SED"
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] ]]; then
echo "$0: ERROR: in output_full_name: $JVAL_WRAPPER -b -q $AUTHOR_HANDLE_JSON_PATH '$PATTERN' failed," \
"error codes: ${status_codes[*]}" 1>&2
return 5
fi
return 0
@@ -331,6 +337,7 @@ function output_full_name
function output_location_code
{
local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path
local PATTERN; # XPath for JSON pattern
# parse args
#
@@ -354,10 +361,11 @@ function output_location_code
# extract Location Code from the author/author_handle.json file
#
"$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..location_code'
PATTERN='$..location_code'
"$JVAL_WRAPPER" -b -q "$AUTHOR_HANDLE_JSON_PATH" "$PATTERN"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: $JVAL_WRAPPER -w -b $ENTRY_JSON_PATH \$..location_code failed," \
echo "$0: ERROR: in output_location_code: $JVAL_WRAPPER -b -q $ENTRY_JSON_PATH '$PATTERN' failed," \
"error: $status" 1>&2
return 5
fi
@@ -638,6 +646,23 @@ if [[ ! -x $LOCATION_TOOL ]]; then
fi
# verify that the bin/unicode-fix.sed tool is executable
#
export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed"
if [[ ! -e $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -f $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -r $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
# parameter debugging
#
if [[ $V_FLAG -ge 3 ]]; then
@@ -673,6 +698,7 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: DOT_PATH_CONTENT=$DOT_PATH_CONTENT" 1>&2
echo "$0: debug[3]: ENTRY_JSON=$ENTRY_JSON" 1>&2
echo "$0: debug[3]: LOCATION_TOOL=$LOCATION_TOOL" 1>&2
echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2
fi
bin/output-year-index.sh
+34 -5
View File
@@ -113,7 +113,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an
# set variables referenced in the usage message
#
export VERSION="1.4.4 2024-08-05"
export VERSION="1.5 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
@@ -190,6 +190,7 @@ $NAME version: $VERSION"
function output_award
{
local ENTRY_JSON_PATH; # the .entry.json path
local PATTERN; # XPath for JSON pattern
# parse args
#
@@ -213,7 +214,17 @@ function output_award
# obtain the award string
#
"$JVAL_WRAPPER" -w -b "$ENTRY_JSON_PATH" '$..award'
PATTERN='$..award'
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN'" 1>&2
fi
"$JVAL_WRAPPER" -b -q "$ENTRY_JSON_PATH" "$PATTERN"
status="$?"
if [[ $status -ne 0 ]]; then
echo "$0: ERROR: in output_award: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN' failed," \
"error code: $status" 1>&2
return 5
fi
return 0
}
@@ -523,6 +534,23 @@ if [[ -z $ORDINAL ]]; then
fi
# verify that the bin/unicode-fix.sed tool is executable
#
export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed"
if [[ ! -e $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -f $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -r $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
# parameter debugging
#
if [[ $V_FLAG -ge 3 ]]; then
@@ -544,6 +572,7 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: YEAR_FILE=$YEAR_FILE" 1>&2
echo "$0: debug[3]: LINE_NUM=$LINE_NUM" 1>&2
echo "$0: debug[3]: ORDINAL=$ORDINAL" 1>&2
echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2
fi
@@ -668,9 +697,9 @@ for YYYY_DIR in $(< "$YEAR_FILE"); do
# determine the award for this entry
#
AWARD=$(output_award "$ENTRY_JSON")
export AWARD
if [[ -z $AWARD ]]; then
AWARD=$(output_award "$ENTRY_JSON" | sed -f "$UNICODE_FIX_SED")
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] || -z $AWARD ]]; then
echo "$0: ERROR: cannot find award in .entry.json: $ENTRY_JSON" 1>&2
exit 7
fi
bin/subst.entry-index.sh
+32 -2
View File
@@ -106,7 +106,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an
# set variables referenced in the usage message
#
export VERSION="1.5 2024-08-13"
export VERSION="1.6 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
@@ -180,6 +180,7 @@ $NAME version: $VERSION"
function output_award
{
local ENTRY_JSON_PATH; # the .entry.json path
local PATTERN; # XPath for JSON pattern
# parse args
#
@@ -203,7 +204,18 @@ function output_award
# obtain the award string
#
"$JVAL_WRAPPER" -w -b "$ENTRY_JSON_PATH" '$..award'
PATTERN='$..award'
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN' | sed ..." 1>&2
fi
"$JVAL_WRAPPER" -b -q "$ENTRY_JSON_PATH" "$PATTERN" | sed -f "$UNICODE_FIX_SED"
status="$?"
status_codes=("${PIPESTATUS[@]}")
if [[ ${status_codes[*]} =~ [1-9] ]]; then
echo "$0: ERROR: in output_award: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN | sed ... failed," \
"error codes: ${status_codes[*]}" 1>&2
return 5
fi
return 0
}
@@ -381,6 +393,23 @@ if [[ ! -x $JVAL_WRAPPER ]]; then
fi
# verify that the bin/unicode-fix.sed tool is executable
#
export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed"
if [[ ! -e $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -f $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -r $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
# verify we have our awk script
#
export ENTRY_NAVBAR_AWK="$BIN_DIR/subst.entry-navbar.awk"
@@ -506,6 +535,7 @@ if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: BIN_PATH=$BIN_PATH" 1>&2
echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2
echo "$0: debug[3]: JVAL_WRAPPER=$JVAL_WRAPPER" 1>&2
echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2
echo "$0: debug[3]: ENTRY_JSON=$ENTRY_JSON" 1>&2
echo "$0: debug[3]: YEAR_DIR=$YEAR_DIR" 1>&2
echo "$0: debug[3]: ENTRY_DIR=$ENTRY_DIR" 1>&2
bin/unicode-chk.sh
Executable
+690
View File
@@ -0,0 +1,690 @@
#!/usr/bin/env bash
#
# unicode-chk.sh - verify all \\u[[:xdigit:]]{4} in entry and author JSON files
#
# Pass entry and author JSON files thru the following sed script and
# flag any \\u[[:xdigit:]]{4} that we might find as problems.
#
# bin/unicode-fix.sed
#
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - until we have fixed jstrdecode, we must sed out \u[0-0A-fa-f]{4} from JSON - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
#
# Until this bug is the jparse repo is fixed, we need to avoid
# using \\u[[:xdigit:]]{4} within JSON encoded strings.
#
# https://github.com/xexyl/jparse/issues/13
#
# The above mentioned bug results in certain tools that are based on the
# jparse repo:
#
# https://github.com/xexyl/jparse
#
# including code from the mkiocccentry repo's jparse sub-directory:
#
# https://github.com/ioccc-src/mkiocccentry/tree/master/jparse
#
# We need to avoid using \\u[[:xdigit:]]{4} within JSON encoded strings.
# Sorry (tm Canada :-)).
#
# The following sed script converts known problematic \\u[[:xdigit:]]{4}
# combinations into their RAW character form:
#
# bin/unicode-fix.sed
#
# Use of that sed script serves as a work-a-round to avoid to avoid any
# problematic \\u[[:xdigit:]]{4} combinations in JSON encoded strings.
#
# Copyright (c) 2024 by Landon Curt Noll. All Rights Reserved.
#
# Permission to use, copy, modify, and distribute this software and
# its documentation for any purpose and without fee is hereby granted,
# provided that the above copyright, this permission notice and text
# this comment, and the disclaimer below appear in all of the following:
#
# supporting documentation
# source copies
# source works derived from this source
# binaries derived from this source or from derived source
#
# LANDON CURT NOLL DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE,
# INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO
# EVENT SHALL LANDON CURT NOLL BE LIABLE FOR ANY SPECIAL, INDIRECT OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF
# USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
# OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
# PERFORMANCE OF THIS SOFTWARE.
#
# chongo (Landon Curt Noll, http://www.isthe.com/chongo/index.html) /\oo/\
#
# Share and enjoy! :-)
# firewall - run only with a bash that is version 5.1.8 or later
#
# The "/usr/bin/env bash" command must result in using a bash that
# is version 5.1.8 or later.
#
# We could relax this version and insist on version 4.2 or later. Versions
# of bash between 4.2 and 5.1.7 might work. However, to be safe, we will require
# bash version 5.1.8 or later.
#
# WHY 5.1.8 and not 4.2? This safely is done because macOS Homebrew bash we
# often use is "version 5.2.26(1)-release" or later, and the RHEL Linux bash we
# use often use is "version 5.1.8(1)-release" or later. These versions are what
# we initially tested. We recommend you either upgrade bash or install a newer
# version of bash and adjust your $PATH so that "/usr/bin/env bash" finds a bash
# that is version 5.1.8 or later.
#
# NOTE: The macOS shipped, as of 2024 March 15, a version of bash is something like
# bash "version 3.2.57(1)-release". That macOS shipped version of bash
# will NOT work. For users of macOS we recommend you install Homebrew,
# (see https://brew.sh), and then run "brew install bash" which will
# typically install it into /opt/homebrew/bin/bash, and then arrange your $PATH
# so that "/usr/bin/env bash" finds "/opt/homebrew/bin" (or whatever the
# Homebrew bash is).
#
# NOTE: And while MacPorts might work, we noticed a number of subtle differences
# with some of their ported tools to suggest you might be better off
# with installing Homebrew (see https://brew.sh). No disrespect is intended
# to the MacPorts team as they do a commendable job. Nevertheless we ran
# into enough differences with MacPorts environments to suggest you
# might find a better experience with this tool under Homebrew instead.
#
if [[ -z ${BASH_VERSINFO[0]} ||
${BASH_VERSINFO[0]} -lt 5 ||
${BASH_VERSINFO[0]} -eq 5 && ${BASH_VERSINFO[1]} -lt 1 ||
${BASH_VERSINFO[0]} -eq 5 && ${BASH_VERSINFO[1]} -eq 1 && ${BASH_VERSINFO[2]} -lt 8 ]]; then
echo "$0: ERROR: bash version needs to be >= 5.1.8: $BASH_VERSION" 1>&2
echo "$0: Warning: bash version >= 4.2 might work but 5.1.8 was the minimum we tested" 1>&2
echo "$0: Notice: For macOS users: install Homebrew (see https://brew.sh), then run" \
""brew install bash" and then modify your \$PATH so that \"#!/usr/bin/env bash\"" \
"finds the Homebrew installed (usually /opt/homebrew/bin/bash) version of bash" 1>&2
exit 4
fi
# setup bash file matching
#
# We must declare arrays with -ag or -Ag, and we need loops to "export" modified variables.
# This requires a bash with a version 4.2 or later. See the larger comment above about bash versions.
#
shopt -s nullglob # enable expanded to nothing rather than remaining unexpanded
shopt -u failglob # disable error message if no matches are found
shopt -u dotglob # disable matching files starting with .
shopt -u nocaseglob # disable strict case matching
shopt -u extglob # enable extended globbing patterns
shopt -s globstar # enable ** to match all files and zero or more directories and subdirectories
# set variables referenced in the usage message
#
export VERSION="1.0 2024-09-23"
NAME=$(basename "$0")
export NAME
export V_FLAG=0
GIT_TOOL=$(type -P git)
export GIT_TOOL
if [[ -z "$GIT_TOOL" ]]; then
echo "$0: FATAL: git tool is not installed or not in \$PATH" 1>&2
exit 5
fi
"$GIT_TOOL" rev-parse --is-inside-work-tree >/dev/null 2>&1
status="$?"
if [[ $status -eq 0 ]]; then
TOPDIR=$("$GIT_TOOL" rev-parse --show-toplevel)
fi
export TOPDIR
export DOCROOT_SLASH="../../"
export TAGLINE="unspecified"
export REPO_TOP_URL="https://github.com/ioccc-src/temp-test-ioccc"
# GitHub puts individual files under the "blob/master" sub-directory.
export REPO_URL="$REPO_TOP_URL/blob/master"
export SITE_URL="https://ioccc-src.github.io/temp-test-ioccc"
#
export NOOP=
export DO_NOT_PROCESS=
export EXIT_CODE="0"
# usage
#
export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-D docroot/] [-n] [-N] [file.json ..]
-h print help message and exit
-v level set verbosity level (def level: 0)
-V print version string and exit
-d topdir set topdir (def: $TOPDIR)
-D docroot/ set the document root path followed by slash (def: $DOCROOT_SLASH)
NOTE: The '-D docroot/' is passed as leading options on tool command lines.
NOTE: 'docroot' must end in a slash
-n go thru the actions, but do not update any files (def: do the action)
-N do not process anything, just parse arguments (def: process something)
[file.json ...] check file.json files (def: check all entry and author JSON files)
Exit codes:
0 all OK
1 found unexpected \\uHexHexHexHex strings in JSON files
2 -h and help string printed or -V and version string printed
3 command line error
4 bash version is too old
5 some internal tool is not found or not an executable file
6 problems found with or in the topdir or topdir/YYYY directory
7 problems found with or in the entry topdir/YYYY/dir directory
8 file.json not a readable file
>= 10 internal error
$NAME version: $VERSION"
# parse command line
#
while getopts :hv:Vd:D:j:nN flag; do
case "$flag" in
h) echo "$USAGE" 1>&2
exit 2
;;
v) V_FLAG="$OPTARG"
;;
V) echo "$VERSION"
exit 2
;;
d) TOPDIR="$OPTARG"
;;
D) # parse -D docroot/
case "$OPTARG" in
*/) ;;
*) echo "$0: ERROR: in -D docroot/, the docroot must end in /" 1>&2
echo 1>&2
print_usage 1>&2
exit 3
;;
esac
DOCROOT_SLASH="$OPTARG"
;;
n) NOOP="-n"
;;
N) DO_NOT_PROCESS="-N"
;;
\?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2
echo 1>&2
print_usage 1>&2
exit 3
;;
:) echo "$0: ERROR: option -$OPTARG requires an argument" 1>&2
echo 1>&2
echo "$USAGE" 1>&2
exit 3
;;
*) echo "$0: ERROR: unexpected value from getopts: $flag" 1>&2
echo 1>&2
echo "$USAGE" 1>&2
exit 3
;;
esac
done
#
# remove the options
#
shift $(( OPTIND - 1 ));
#
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: file argument count: $#" 1>&2
fi
# verify that we have a topdir directory
#
REPO_NAME=$(basename "$REPO_TOP_URL")
export REPO_NAME
if [[ -z $TOPDIR ]]; then
echo "$0: ERROR: cannot find top of git repo directory" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
if [[ ! -e $TOPDIR ]]; then
echo "$0: ERROR: TOPDIR does not exist: $TOPDIR" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
if [[ ! -d $TOPDIR ]]; then
echo "$0: ERROR: TOPDIR is not a directory: $TOPDIR" 1>&2
echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2
exit 6
fi
# cd to topdir
#
if [[ ! -e $TOPDIR ]]; then
echo "$0: ERROR: cannot cd to non-existent path: $TOPDIR" 1>&2
exit 6
fi
if [[ ! -d $TOPDIR ]]; then
echo "$0: ERROR: cannot cd to a non-directory: $TOPDIR" 1>&2
exit 6
fi
export CD_FAILED
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to: cd $TOPDIR" 1>&2
fi
cd "$TOPDIR" || CD_FAILED="true"
if [[ -n $CD_FAILED ]]; then
echo "$0: ERROR: cd $TOPDIR failed" 1>&2
exit 6
fi
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: now in directory: $(/bin/pwd)" 1>&2
fi
# verify that we have an author subdirectory
#
export AUTHOR_PATH="$TOPDIR/author"
if [[ ! -d $AUTHOR_PATH ]]; then
echo "$0: ERROR: author is not a directory under topdir: $AUTHOR_PATH" 1>&2
exit 6
fi
export AUTHOR_DIR="author"
# verify that we have a bin subdirectory
#
export BIN_PATH="$TOPDIR/bin"
if [[ ! -d $BIN_PATH ]]; then
echo "$0: ERROR: bin is not a directory under topdir: $BIN_PATH" 1>&2
exit 6
fi
export BIN_DIR="bin"
# verify that the bin/unicode-fix.sed tool is executable
#
export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed"
if [[ ! -e $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -f $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
if [[ ! -r $UNICODE_FIX_SED ]]; then
echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2
exit 5
fi
# verify we have a non-empty readable .top file
#
export TOP_FILE=".top"
if [[ ! -e $TOP_FILE ]]; then
echo "$0: ERROR: .top does not exist: $TOP_FILE" 1>&2
exit 6
fi
if [[ ! -f $TOP_FILE ]]; then
echo "$0: ERROR: .top is not a regular file: $TOP_FILE" 1>&2
exit 6
fi
if [[ ! -r $TOP_FILE ]]; then
echo "$0: ERROR: .top is not an readable file: $TOP_FILE" 1>&2
exit 6
fi
if [[ ! -s $TOP_FILE ]]; then
echo "$0: ERROR: .top is not a non-empty readable file: $TOP_FILE" 1>&2
exit 6
fi
# print running info if verbose
#
# If -v 3 or higher, print exported variables in order that they were exported.
#
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: VERSION=$VERSION" 1>&2
echo "$0: debug[3]: NAME=$NAME" 1>&2
echo "$0: debug[3]: V_FLAG=$V_FLAG" 1>&2
echo "$0: debug[3]: GIT_TOOL=$GIT_TOOL" 1>&2
echo "$0: debug[3]: TOPDIR=$TOPDIR" 1>&2
echo "$0: debug[3]: DOCROOT_SLASH=$DOCROOT_SLASH" 1>&2
echo "$0: debug[3]: TAGLINE=$TAGLINE" 1>&2
echo "$0: debug[3]: REPO_TOP_URL=$REPO_TOP_URL" 1>&2
echo "$0: debug[3]: REPO_URL=$REPO_URL" 1>&2
echo "$0: debug[3]: SITE_URL=$SITE_URL" 1>&2
echo "$0: debug[3]: NOOP=$NOOP" 1>&2
echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2
echo "$0: debug[3]: EXIT_CODE=$EXIT_CODE" 1>&2
echo "$0: debug[3]: REPO_NAME=$REPO_NAME" 1>&2
echo "$0: debug[3]: CD_FAILED=$CD_FAILED" 1>&2
echo "$0: debug[3]: AUTHOR_PATH=$AUTHOR_PATH" 1>&2
echo "$0: debug[3]: AUTHOR_DIR=$AUTHOR_DIR" 1>&2
echo "$0: debug[3]: BIN_PATH=$BIN_PATH" 1>&2
echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2
echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2
echo "$0: debug[3]: TOP_FILE=$TOP_FILE" 1>&2
fi
# -N stops early before any processing is performed
#
if [[ -n $DO_NOT_PROCESS ]]; then
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: arguments parsed, -N given, exiting 0" 1>&2
fi
exit 0
fi
# create a JSON problem file
#
export TMP_JSON_PROBLEM=".tmp.$NAME.JSON_PROBLEM.$$.tmp"
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: JSON problem inventory file: $TMP_JSON_PROBLEM" 1>&2
fi
trap 'rm -f $TMP_AUTHOR_HANDLE_INVENTORY; exit' 0 1 2 3 15
rm -f "$TMP_JSON_PROBLEM"
if [[ -e $TMP_JSON_PROBLEM ]]; then
echo "$0: ERROR: cannot remove JSON problem inventory file: $TMP_JSON_PROBLEM" 1>&2
exit 10
fi
: > "$TMP_JSON_PROBLEM"
if [[ ! -e $TMP_JSON_PROBLEM ]]; then
echo "$0: ERROR: cannot create JSON problem inventory file: $TMP_JSON_PROBLEM" 1>&2
exit 11
fi
# case: check only file given as args
#
if [[ $# -gt 0 ]]; then
# check specific JSON files
#
for JSON_FILE in "$@"; do
# just be a readable file
#
if [[ ! -e $JSON_FILE ]]; then
echo "$0: ERROR: does not exist: $JSON_FILE" 1>&2
exit 8
fi
if [[ ! -f $JSON_FILE ]]; then
echo "$0: ERROR: is not a regular file: $JSON_FILE" 1>&2
exit 8
fi
if [[ ! -r $JSON_FILE ]]; then
echo "$0: ERROR: is not an readable file: $JSON_FILE" 1>&2
exit 8
fi
# scan the author/author_handle.json file for unexpected \\u[[:xdigit:]]{4} strings
#
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: about to run: sed -f $UNICODE_FIX_SED -- $JSON_FILE 2>&1 | grep -E ..." 1>&2
fi
FOUND_UHEX=$(sed -f "$UNICODE_FIX_SED" -- "$JSON_FILE" 2>&1 | grep -E '\\u[[:xdigit:]]{4}')
if [[ -n $FOUND_UHEX ]]; then
echo "$JSON_FILE"
fi
done >> "$TMP_JSON_PROBLEM"
# case: check all .entry.json files and author JSON files
#
else
# check all .entry.json files
#
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: starting to check .entry.json files under: $TOPDIR" 1>&2
fi
export YYYY
for YYYY in $(< "$TOP_FILE"); do
# debug YYYY
#
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: starting to check year: $YYYY" 1>&2
fi
# verify that YYYY is a readable directory
#
if [[ ! -e $YYYY ]]; then
echo "$0: ERROR: YYYY does not exist: $YYYY" 1>&2
EXIT_CODE="6" # exit 6
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
if [[ ! -d $YYYY ]]; then
echo "$0: ERROR: YYYY is not a directory: $YYYY" 1>&2
EXIT_CODE="6" # exit 6
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
if [[ ! -r $YYYY ]]; then
echo "$0: ERROR: YYYY is not an readable directory: $YYYY" 1>&2
EXIT_CODE="6" # exit 6
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
# verify that YYYY has a non-empty readable .year file
#
export YEAR_FILE="$YYYY/.year"
if [[ ! -e $YEAR_FILE ]]; then
echo "$0: ERROR: YYYY/.year does not exist: $YEAR_FILE" 1>&2
EXIT_CODE="6" # exit 6
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
if [[ ! -f $YEAR_FILE ]]; then
echo "$0: ERROR: YYYY/.year is not a regular file: $YEAR_FILE" 1>&2
EXIT_CODE="6" # exit 6
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
if [[ ! -r $YEAR_FILE ]]; then
echo "$0: ERROR: YYYY/.year is not an readable file: $YEAR_FILE" 1>&2
EXIT_CODE="6" # exit 6
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
if [[ ! -s $YEAR_FILE ]]; then
echo "$0: ERROR: YYYY/.year is not a non-empty readable file: $YEAR_FILE" 1>&2
EXIT_CODE="6" # exit 6
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
# process each entry directory under YYYY
#
export YYYY_DIR
for YYYY_DIR in $(< "$YEAR_FILE"); do
# debug YYYY
#
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: starting to check year/dir: $YYYY_DIR" 1>&2
fi
# parse YYYY_DIR
#
if [[ ! -d $YYYY_DIR ]]; then
echo "$0: ERROR: YYYY_DIR is not a directory: $YYYY_DIR" 1>&2
EXIT_CODE="6" # exit 6
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
if [[ ! -w $YYYY_DIR ]]; then
echo "$0: ERROR: YYYY_DIR is not a writable directory: $YYYY_DIR" 1>&2
EXIT_CODE="6" # exit 6
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
export YEAR_DIR=${YYYY_DIR%%/*}
if [[ -z $YEAR_DIR ]]; then
echo "$0: ERROR: YYYY_DIR not in YYYY/dir form: $YYYY_DIR" 1>&2
EXIT_CODE="6" # exit 6
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
export ENTRY_DIR=${YYYY_DIR#*/}
if [[ -z $ENTRY_DIR ]]; then
echo "$0: ERROR: YYYY_DIR not in $YEAR_DIR/dir form: $YYYY_DIR" 1>&2
EXIT_CODE="6" # exit 6
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
if [[ $ENTRY_DIR = */* ]]; then
echo "$0: ERROR: YYYY_DIR: $YYYY_DIR dir contains a /: $ENTRY_DIR" 1>&2
EXIT_CODE="6" # exit 6
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
# verify that YYYY_DIR is a writable directory
#
if [[ ! -e $YYYY_DIR ]]; then
echo "$0: ERROR: YYYY_DIR does not exist: $YYYY_DIR" 1>&2
EXIT_CODE="7" # exit 7
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
if [[ ! -d $YYYY_DIR ]]; then
echo "$0: ERROR: YYYY_DIR is not a directory: $YYYY_DIR" 1>&2
EXIT_CODE="7" # exit 7
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
if [[ ! -w $YYYY_DIR ]]; then
echo "$0: ERROR: YYYY_DIR is not an writable directory: $YYYY_DIR" 1>&2
EXIT_CODE="7" # exit 7
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
# verify YYYY/dir/.path
#
export DOT_PATH="$YYYY_DIR/.path"
if [[ ! -s $DOT_PATH ]]; then
echo "$0: ERROR: not a non-empty file: $DOT_PATH" 1>&2
EXIT_CODE="7" # exit 7
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
DOT_PATH_CONTENT=$(< "$DOT_PATH")
if [[ $YYYY_DIR != "$DOT_PATH_CONTENT" ]]; then
echo "$0: ERROR: arg: $YYYY_DIR does not match $DOT_PATH contents: $DOT_PATH_CONTENT" 1>&2
EXIT_CODE="7" # exit 7
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
# verify the YYYY/dir/.entry.json
#
export ENTRY_JSON="$YYYY_DIR/.entry.json"
if [[ ! -e $ENTRY_JSON ]]; then
echo "$0: ERROR: .entry.json does not exist: $ENTRY_JSON" 1>&2
EXIT_CODE="7" # exit 7
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
if [[ ! -f $ENTRY_JSON ]]; then
echo "$0: ERROR: .entry.json is not a file: $ENTRY_JSON" 1>&2
EXIT_CODE="7" # exit 7
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
if [[ ! -r $ENTRY_JSON ]]; then
echo "$0: ERROR: .entry.json is not a readable file: $ENTRY_JSON" 1>&2
EXIT_CODE="7" # exit 7
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
continue
fi
# scan the .entry.json file for unexpected \\u[[:xdigit:]]{4} strings
#
if [[ -z $NOOP ]]; then
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: sed -f $UNICODE_FIX_SED -- $YYYY_DIR/.entry.json 2>&1 | grep -E ..." 1>&2
fi
FOUND_UHEX=$(sed -f "$UNICODE_FIX_SED" -- "$YYYY_DIR/.entry.json" 2>&1 | grep -E '\\u[[:xdigit:]]{4}' 2>&1)
export FOUND_UHEX
if [[ -n $FOUND_UHEX ]]; then
echo "$YYYY_DIR/.entry.json"
fi
fi
done
done >> "$TMP_JSON_PROBLEM"
# create a temporary author_handle inventory file
#
export TMP_AUTHOR_HANDLE_INVENTORY=".tmp.$NAME.AUTHOR_HANDLE_INVENTORY.$$.tmp"
if [[ $V_FLAG -ge 3 ]]; then
echo "$0: debug[3]: temporary author_handle inventory file: $TMP_AUTHOR_HANDLE_INVENTORY" 1>&2
fi
trap 'rm -f $TMP_AUTHOR_HANDLE_INVENTORY $TMP_AUTHOR_HANDLE_INVENTORY; exit' 0 1 2 3 15
rm -f "$TMP_AUTHOR_HANDLE_INVENTORY"
if [[ -e $TMP_AUTHOR_HANDLE_INVENTORY ]]; then
echo "$0: ERROR: cannot remove temporary author_handle inventory file: $TMP_AUTHOR_HANDLE_INVENTORY" 1>&2
exit 12
fi
: > "$TMP_AUTHOR_HANDLE_INVENTORY"
if [[ ! -e $TMP_AUTHOR_HANDLE_INVENTORY ]]; then
echo "$0: ERROR: cannot create temporary author_handle inventory file: $TMP_AUTHOR_HANDLE_INVENTORY" 1>&2
exit 13
fi
# check author/author_handle.json files
#
if [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: starting to check author_handle.json files under: $AUTHOR_DIR" 1>&2
fi
find "$AUTHOR_DIR" -mindepth 1 -maxdepth 1 -type f -name '*.json' 2>/dev/null |
LC_ALL=C sort -d -u > "$TMP_AUTHOR_HANDLE_INVENTORY"
while read -r AUTHOR_HANDLE_JSON; do
# scan the author/author_handle.json file for unexpected \\u[[:xdigit:]]{4} strings
#
if [[ $V_FLAG -ge 5 ]]; then
echo "$0: debug[5]: about to run: sed -f $UNICODE_FIX_SED -- $AUTHOR_HANDLE_JSON 2>&1 | grep -E ..." 1>&2
fi
FOUND_UHEX=$(sed -f "$UNICODE_FIX_SED" -- "$AUTHOR_HANDLE_JSON" 2>&1 | grep -E '\\u[[:xdigit:]]{4}')
if [[ -n $FOUND_UHEX ]]; then
echo "$AUTHOR_HANDLE_JSON"
fi
done < "$TMP_AUTHOR_HANDLE_INVENTORY" >> "$TMP_JSON_PROBLEM"
fi
# report any problems
#
if [[ -s $TMP_JSON_PROBLEM ]]; then
echo "$0: ERROR: found unexpected \\uHexHexHexHex strings in JSON files" 1>&2
echo "$0: debug[5]: list of JSON files with unexpected \\uHexHexHexHex strings, starts below" 1>&2
cat "$TMP_JSON_PROBLEM" 1>&2
echo "$0: debug[5]: list of JSON files with unexpected \\uHexHexHexHex strings, ends above" 1>&2
if [[ $EXIT_CODE -eq 0 ]]; then
EXIT_CODE="1" # exit 1
echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2
fi
fi
# All Done!!! All Done!!! -- Jessica Noll, Age 2
#
if [[ $EXIT_CODE -ne 0 ]]; then
echo "$0: Warning: about to exit non-zero: $EXIT_CODE" 1>&2
elif [[ $V_FLAG -ge 1 ]]; then
echo "$0: debug[1]: about to exit $EXIT_CODE" 1>&2
fi
exit "$EXIT_CODE"
bin/unicode-fix.sed
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env sed
#
# bin/unicode-fix.sed - work-a-round to pre-translate certain JSON /u codes
#
# Until the jsrdecode(1) is fixed so that it properly converts 1 or 2 consecutive
# JSON string \uHexHexHexHex values into the proper binary symbol, we pre-translate
# JSON string \uHexHexHexHex values used in this web site as a work-a-round to
# avoid the jsrdecode(1) bug.
#
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - until we have fixed jstrdecode, we must FAKE PARSE IOCCC JSON files - XXX
# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX
# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX
#
# Until this bug is the jparse repo is fixed, we need to avoid
# using \\u[[:xdigit:]]{4} within JSON encoded strings.
#
# https://github.com/xexyl/jparse/issues/13
#
# The above mentioned bug results in certain tools that are based on the
# jparse repo:
#
# https://github.com/xexyl/jparse
#
# including code from the mkiocccentry repo's jparse sub-directory:
#
# https://github.com/ioccc-src/mkiocccentry/tree/master/jparse
#
# We need to avoid using \\u[[:xdigit:]]{4} within JSON encoded strings.
# Sorry (tm Canada :-)).
#
# The following sed script converts known problematic \\u[[:xdigit:]]{4}
# combinations into their RAW character form:
#
# bin/unicode-fix.sed
#
# Use of that sed script serves as a work-a-round to avoid to avoid any
# problematic \\u[[:xdigit:]]{4} combinations in JSON encoded strings.
#
# For information on the jsrdecode(1) bug, see:
#
# https://github.com/xexyl/jparse/issues/13
#
# usage:
#
# cmd | sed -f bin/unicode-fix.sed
#
# sed -f bin/unicode-fix.sed file > output
#
# sed -i -f bin/unicode-fix.sed file
#
# VERSION="1.0 2024-09-23"
#
s/\\u00c4/Ä/g
s/\\u00e1/á/g
s/\\u00eb/ë/g
s/\\u00fd/ý/g
s/\\u0159/ř/g
s/\\u2019/’/g