From 499be5983d0c8a6c4b090b57f218be9a110bd1d5 Mon Sep 17 00:00:00 2001 From: Landon Curt Noll Date: Tue, 24 Sep 2024 04:04:27 -0700 Subject: [PATCH] use jsp tool, workaround JSON Unicode bugs, misc fixes If the [jsp tool](https://github.com/kjozsa/jsp) is installed, it will be used in favor of the [JSONPath.sh tool](https://github.com/lcn2/JSONPath.sh). Updated `bin/jfmt-wrapper.sh`, `bin/jprint-wrapper.sh`, and `bin/jval-wrapper.sh` to use jsp if found. These 3 tools may be given `-t tool` where tool is either "_jsp_" or "_JSONPath.sh_" for force the use of the tool. They also use `-Q` to perform a quick check tool using trivial input. The `bin/jval-wrapper.sh` tool had been simplified. The `-w`, `-i` and `-u` options are gone. Instead it has: ``` -b print only values (def: print both JSON member and JSON value) -q remove enclosing double quotes (def: keep any enclosing double quotes) ``` Add `make all_jfmt` to Makefile to canonically format all entry and author handle JSON files. A sed script `bin/unicode-fix.sed` has been added that converts JSON string "_\u[0-9A-Fa-f]{4}_" encoding that may arise from known JSON data. This is done as a work-a-round for [Unicode / UTF-8 issues](https://github.com/xexyl/jparse/issues/13) with `jstrdecode(1)`. The `bin/unicode-chk.sh` tool, if given on file arguments, scans all `.entry.json` and `author/author_handle.json` JSON files to be sure that, after `bin/unicode-fix.sed` processing, no JSON string "_\u[0-9A-Fa-f]{4}_" encodings remain. Removed all `-j jparse` args from commands. Instead the `jparse(1)` tool is looked for along $PATH. Added `combine_author_handle.sh` tool to transform all of the `author/author_handle.json` files into a temporary as a single JSON file so that `bin/jval-wrapper.sh` tool may be used to extract all "_sort_word_" or all "_entry_id_" values. Changed `bin/all-jfmt.sh` VERSION from "1.0 2024-08-31" to "1.1 2024-09-24". Changed `bin/csv2entry.sh` VERSION from "1.2.3 2024-08-30" to "1.3 2024-09-23". Changed `bin/cvt-submission.sh` VERSION from "1.1 2024-08-31" to "1.2 2024-09-23". Changed `bin/entry2csv.sh` VERSION from "1.2.1 2024-08-17" to "1.3 2024-09-23". Changed `bin/gen-authors.sh` VERSION from "1.12 2024-08-18" to "1.13 2024-09-23". Changed `bin/gen-location.sh` VERSION from "1.6 2024-08-18" to "1.7 2024-09-23". Changed `bin/gen-status.sh` VERSION from "1.6 2024-08-17" to "1.7 2024-09-23". Changed `bin/gen-years.sh` VERSION from "1.4 2024-08-17" to "1.5 2024-09-23". Changed `bin/jfmt-wrapper.sh` VERSION from "1.1 2024-08-20" to "1.2 2024-09-23". Changed `bin/jprint-wrapper.sh` VERSION from "1.1 2024-08-20" to "1.3 2024-09-23". Changed `bin/jval-wrapper.sh` VERSION from "1.0.4 2024-08-20" to "1.1 2024-09-23". Changed `bin/manifest.entry.json.awk` VERSION from "1.1.2 2024-07-28" to "1.2 2024-09-17". Changed `bin/output-index-author.sh` VERSION from "1.8 2024-08-17" to "1.9 2024-09-23". Changed `bin/output-year-index.sh` VERSION from "1.4.4 2024-08-05" to "1.5 2024-09-23". Changed `bin/subst.entry-index.sh` VERSION from "1.5 2024-08-13" to "1.6 2024-09-23". Added `bin/combine_author_handle.sh` VERSION "1.0 2024-09-23". Added `bin/unicode-chk.sh` VERSION "1.0 2024-09-23". Added `bin/unicode-fix.sed` VERSION "1.0 2024-09-23". Perform `make www` to perform the above and test under macOS. FYI: On our MacBook Pro, `make www` took: ``` make www 631.79s user 264.95s system 94% cpu 15:53.71 total ``` FYI: On our MacBook Pro, `make quick_www` took: ``` make quick_www 81.43s user 58.26s system 102% cpu 2:16.02 total ``` --- Makefile | 5 +- bin/all-jfmt.sh | 41 +- bin/combine_author_handle.sh | 447 +++++++++++++++++++++ bin/csv2entry.sh | 269 ++++++------- bin/cvt-submission.sh | 259 +++++++----- bin/entry2csv.sh | 154 ++++--- bin/gen-authors.sh | 752 ++++++++++------------------------- bin/gen-location.sh | 39 +- bin/gen-status.sh | 44 +- bin/gen-years.sh | 31 +- bin/jfmt-wrapper.sh | 348 ++++++++++++---- bin/jprint-wrapper.sh | 375 ++++++++++++++--- bin/jval-wrapper.sh | 538 +++++++++++++++++++++---- bin/manifest.entry.json.awk | 82 +++- bin/output-index-author.sh | 50 ++- bin/output-year-index.sh | 39 +- bin/subst.entry-index.sh | 34 +- bin/unicode-chk.sh | 690 ++++++++++++++++++++++++++++++++ bin/unicode-fix.sed | 60 +++ 19 files changed, 3044 insertions(+), 1213 deletions(-) create mode 100755 bin/combine_author_handle.sh create mode 100755 bin/unicode-chk.sh create mode 100644 bin/unicode-fix.sed diff --git a/Makefile b/Makefile index aff324b77..830f79cd4 100644 --- a/Makefile +++ b/Makefile @@ -311,7 +311,6 @@ help: @echo 'make tab_check ;: check for ASCII tabs in markdown files' @echo 'make genpath ;: form top level .top and .allyear, YYYY level .year and winner .path files' @echo 'make genfilelist ;: generate YYYY level .filelist' - @echo 'make all_jfmt ;: be sure all JSON files are canonical' @echo 'make verify_entry_files ;: check to be sure all files in all entries exist' @echo 'make sort_gitignore ;: sort .gitignore files according to rules in bin/sgi.sh' @echo 'make gen_authors ;: generate the top level authors.html page' @@ -354,6 +353,8 @@ help: @echo 'make untar_entry_tarball;: untar all entry tarballs' @echo 'make untar_year_tarball ;: untar all year level tarballs' @echo + @echo 'make all_jfmt ;: convert all entry and author JSON files into canonical form' + @echo @echo 'make form_entry_tarball ;: form all entry compressed tarballs' @echo 'make form_year_tarball ;: form all IOCCC year level compressed tarballs' @echo 'make tar ;: build all tarballs' @@ -715,7 +716,6 @@ www: @echo '=-=-=-=-= IOCCC complete make genpath =-=-=-=-=' ${MAKE} genfilelist ${MAKE} sort_gitignore - ${MAKE} all_jfmt ${MAKE} verify_entry_files ${MAKE} gen_authors ${MAKE} gen_location @@ -769,6 +769,7 @@ tar: ${MAKE} clobber ${MAKE} genpath ${MAKE} genfilelist + ${MAKE} all_jfmt ${MAKE} verify_entry_files ${MAKE} sort_gitignore ${MAKE} form_entry_tarball diff --git a/bin/all-jfmt.sh b/bin/all-jfmt.sh index fe7e112f0..cd9ea4dd4 100755 --- a/bin/all-jfmt.sh +++ b/bin/all-jfmt.sh @@ -86,7 +86,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an # set variables referenced in the usage message # -export VERSION="1.0 2024-08-31" +export VERSION="1.1 2024-09-24" NAME=$(basename "$0") export NAME export V_FLAG=0 @@ -112,9 +112,6 @@ export SITE_URL="https://ioccc-src.github.io/temp-test-ioccc" export NOOP= export DO_NOT_PROCESS= export EXIT_CODE="0" -# -JPARSE_TOOL=$(type -P jparse) -export JPARSE_TOOL # clear options we will add to tools @@ -125,7 +122,7 @@ declare -ag TOOL_OPTION # usage # -export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-D docroot/] [-j jparse] [-n] [-N] +export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-D docroot/] [-n] [-N] -h print help message and exit -v level set verbosity level (def level: 0) @@ -136,8 +133,6 @@ export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-D docroot/] [-j jpars NOTE: The '-D docroot/' is passed as leading options on tool command lines. NOTE: 'docroot' must end in a slash - -j jparse path to the jparse tool (def: $JPARSE_TOOL) - -n go thru the actions, but do not update any files (def: do the action) -N do not process anything, just parse arguments (def: process something) @@ -182,8 +177,6 @@ while getopts :hv:Vd:D:j:nN flag; do TOOL_OPTION+=("-D") TOOL_OPTION+=("$DOCROOT_SLASH") ;; - j) JPARSE_TOOL="$OPTARG" - ;; n) NOOP="-n" ;; N) DO_NOT_PROCESS="-N" @@ -326,22 +319,6 @@ if [[ ! -s $TOP_FILE ]]; then fi -# verify jparse tool -# -if [[ -z "$JPARSE_TOOL" ]]; then - echo "$0: ERROR: cannot find an executable jparse tool" 1>&2 - exit 5 -fi -if [[ ! -f "$JPARSE_TOOL" ]]; then - echo "$0: ERROR: jparse is not a file: $JPARSE_TOOL" 1>&2 - exit 5 -fi -if [[ ! -x "$JPARSE_TOOL" ]]; then - echo "$0: ERROR: jparse is not an executable file: $JPARSE_TOOL" 1>&2 - exit 5 -fi - - # print running info if verbose # # If -v 3 or higher, print exported variables in order that they were exported. @@ -371,7 +348,6 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2 echo "$0: debug[3]: JFMT_WRAPPER=$JFMT_WRAPPER" 1>&2 echo "$0: debug[3]: TOP_FILE=$TOP_FILE" 1>&2 - echo "$0: debug[3]: JPARSE_TOOL=$JPARSE_TOOL" 1>&2 fi @@ -569,19 +545,6 @@ for YYYY in $(< "$TOP_FILE"); do echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 continue fi - if [[ $V_FLAG -ge 3 ]]; then - echo "$0: debug[3]: about to run: $JPARSE_TOOL -q -- $ENTRY_JSON" 1>&2 - fi - if "$JPARSE_TOOL" -q -- "$ENTRY_JSON"; then - if [[ $V_FLAG -ge 5 ]]; then - echo "$0: debug[5]: valid JSON: $ENTRY_JSON" 1>&2 - fi - else - echo "$0: ERROR: .entry.json is not valid JSON: $ENTRY_JSON" 1>&2 - EXIT_CODE="7" # exit 7 - echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 - continue - fi # canonically format YYYY/dir/.entry.json unless -n # diff --git a/bin/combine_author_handle.sh b/bin/combine_author_handle.sh new file mode 100755 index 000000000..da9a526b7 --- /dev/null +++ b/bin/combine_author_handle.sh @@ -0,0 +1,447 @@ +#!/usr/bin/env bash +# +# combine_author_handle.sh - output all author/author_handle.JSON files as single JSON file +# +# We form a single JSON file out of all author/author_handle.JSON files. +# We cannot simply cat the author/author_handle.JSON files because +# each author/author_handle.JSON file opens and closes the JSON. +# So this tool removes the intermediate JSON document opening "{" +# and closing "}". +# +# Because jsp open remembers the last copy of a given JSON member name. +# As such, we change the JSON member name "winning_entry_set" into +# a unique "winning_entry_set.FILENO" where FILENO is the file number. +# This the JSON member value will be preserved across all files. +# +# We also convert where FILENO is the file number and FILENAME is the filename: +# +# "sort_word" : "data", +# +# into: +# +# "sort_word.FILENO" : [ +# { +# "sort_word" : "data FILENAME" +# } +# ], +# +# This will allow jsp to print sort_word values. +# +# We also make sure that the last item from a given file ends in a comma (",)", +# due to the bogosity of the so-called JSON spec. +# +# NOTE: We assume that all JSON files have been formatted with the +# bin/jprint-wrapper.sh tool. In particular the first line is just "{:, +# and the last line is just "}" and each JSON element is on its own line. +# +# Copyright (c) 2024 by Landon Curt Noll. All Rights Reserved. +# +# Permission to use, copy, modify, and distribute this software and +# its documentation for any purpose and without fee is hereby granted, +# provided that the above copyright, this permission notice and text +# this comment, and the disclaimer below appear in all of the following: +# +# supporting documentation +# source copies +# source works derived from this source +# binaries derived from this source or from derived source +# +# LANDON CURT NOLL DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, +# INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO +# EVENT SHALL LANDON CURT NOLL BE LIABLE FOR ANY SPECIAL, INDIRECT OR +# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF +# USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR +# OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +# PERFORMANCE OF THIS SOFTWARE. +# +# chongo (Landon Curt Noll, http://www.isthe.com/chongo/index.html) /\oo/\ +# +# Share and enjoy! :-) + + +# firewall - run only with a bash that is version 5.1.8 or later +# +# The "/usr/bin/env bash" command must result in using a bash that +# is version 5.1.8 or later. +# +# We could relax this version and insist on version 4.2 or later. Versions +# of bash between 4.2 and 5.1.7 might work. However, to be safe, we will require +# bash version 5.1.8 or later. +# +# WHY 5.1.8 and not 4.2? This safely is done because macOS Homebrew bash we +# often use is "version 5.2.26(1)-release" or later, and the RHEL Linux bash we +# use often use is "version 5.1.8(1)-release" or later. These versions are what +# we initially tested. We recommend you either upgrade bash or install a newer +# version of bash and adjust your $PATH so that "/usr/bin/env bash" finds a bash +# that is version 5.1.8 or later. +# +# NOTE: The macOS shipped, as of 2024 March 15, a version of bash is something like +# bash "version 3.2.57(1)-release". That macOS shipped version of bash +# will NOT work. For users of macOS we recommend you install Homebrew, +# (see https://brew.sh), and then run "brew install bash" which will +# typically install it into /opt/homebrew/bin/bash, and then arrange your $PATH +# so that "/usr/bin/env bash" finds "/opt/homebrew/bin" (or whatever the +# Homebrew bash is). +# +# NOTE: And while MacPorts might work, we noticed a number of subtle differences +# with some of their ported tools to suggest you might be better off +# with installing Homebrew (see https://brew.sh). No disrespect is intended +# to the MacPorts team as they do a commendable job. Nevertheless we ran +# into enough differences with MacPorts environments to suggest you +# might find a better experience with this tool under Homebrew instead. +# +if [[ -z ${BASH_VERSINFO[0]} || + ${BASH_VERSINFO[0]} -lt 5 || + ${BASH_VERSINFO[0]} -eq 5 && ${BASH_VERSINFO[1]} -lt 1 || + ${BASH_VERSINFO[0]} -eq 5 && ${BASH_VERSINFO[1]} -eq 1 && ${BASH_VERSINFO[2]} -lt 8 ]]; then + echo "$0: ERROR: bash version needs to be >= 5.1.8: $BASH_VERSION" 1>&2 + echo "$0: Warning: bash version >= 4.2 might work but 5.1.8 was the minimum we tested" 1>&2 + echo "$0: Notice: For macOS users: install Homebrew (see https://brew.sh), then run" \ + ""brew install bash" and then modify your \$PATH so that \"#!/usr/bin/env bash\"" \ + "finds the Homebrew installed (usually /opt/homebrew/bin/bash) version of bash" 1>&2 + exit 4 +fi + + +# setup bash file matching +# +# We must declare arrays with -ag or -Ag, and we need loops to "export" modified variables. +# This requires a bash with a version 4.2 or later. See the larger comment above about bash versions. +# +shopt -s nullglob # enable expanded to nothing rather than remaining unexpanded +shopt -u failglob # disable error message if no matches are found +shopt -u dotglob # disable matching files starting with . +shopt -u nocaseglob # disable strict case matching +shopt -u extglob # enable extended globbing patterns +shopt -s globstar # enable ** to match all files and zero or more directories and subdirectories + + +# set variables referenced in the usage message +# +export VERSION="1.0 2024-09-23" +NAME=$(basename "$0") +export NAME +export V_FLAG=0 +GIT_TOOL=$(type -P git) +export GIT_TOOL +if [[ -z "$GIT_TOOL" ]]; then + echo "$0: FATAL: git tool is not installed or not in \$PATH" 1>&2 + exit 5 +fi +"$GIT_TOOL" rev-parse --is-inside-work-tree >/dev/null 2>&1 +status="$?" +if [[ $status -eq 0 ]]; then + TOPDIR=$("$GIT_TOOL" rev-parse --show-toplevel) +fi +export TOPDIR +export DOCROOT_SLASH="../../" +export TAGLINE="unspecified" +export REPO_TOP_URL="https://github.com/ioccc-src/temp-test-ioccc" +# GitHub puts individual files under the "blob/master" sub-directory. +export REPO_URL="$REPO_TOP_URL/blob/master" +export SITE_URL="https://ioccc-src.github.io/temp-test-ioccc" +# +export NOOP= +export DO_NOT_PROCESS= +export EXIT_CODE="0" + + +# usage +# +export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-D docroot/] [-n] [-N] + + -h print help message and exit + -v level set verbosity level (def level: 0) + -V print version string and exit + + -d topdir set topdir (def: $TOPDIR) + -D docroot/ set the document root path followed by slash (def: $DOCROOT_SLASH) + NOTE: The '-D docroot/' is passed as leading options on tool command lines. + NOTE: 'docroot' must end in a slash + + -n go thru the actions, but do not update any files (def: do the action) + -N do not process anything, just parse arguments (def: process something) + +Exit codes: + 0 all OK + 1 found unexpected \\uHexHexHexHex strings in JSON files + 2 -h and help string printed or -V and version string printed + 3 command line error + 4 bash version is too old + 5 some internal tool is not found or not an executable file + 6 problems found with or in the topdir or topdir/YYYY directory + 7 problems found with or in the entry topdir/YYYY/dir directory + 8 file.json not a readable file + >= 10 internal error + +$NAME version: $VERSION" + + +# parse command line +# +while getopts :hv:Vd:D:j:nN flag; do + case "$flag" in + h) echo "$USAGE" 1>&2 + exit 2 + ;; + v) V_FLAG="$OPTARG" + ;; + V) echo "$VERSION" + exit 2 + ;; + d) TOPDIR="$OPTARG" + ;; + D) # parse -D docroot/ + case "$OPTARG" in + */) ;; + *) echo "$0: ERROR: in -D docroot/, the docroot must end in /" 1>&2 + echo 1>&2 + print_usage 1>&2 + exit 3 + ;; + esac + DOCROOT_SLASH="$OPTARG" + ;; + n) NOOP="-n" + ;; + N) DO_NOT_PROCESS="-N" + ;; + \?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2 + echo 1>&2 + print_usage 1>&2 + exit 3 + ;; + :) echo "$0: ERROR: option -$OPTARG requires an argument" 1>&2 + echo 1>&2 + echo "$USAGE" 1>&2 + exit 3 + ;; + *) echo "$0: ERROR: unexpected value from getopts: $flag" 1>&2 + echo 1>&2 + echo "$USAGE" 1>&2 + exit 3 + ;; + esac +done +# +# remove the options +# +shift $(( OPTIND - 1 )); +# +if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: file argument count: $#" 1>&2 +fi +if [[ $# -ne 0 ]]; then + echo "$0: ERROR: expected 0 args, found: $#" 1>&2 + echo 1>&2 + echo "$USAGE" 1>&2 + exit 3 +fi + + +# verify that we have a topdir directory +# +REPO_NAME=$(basename "$REPO_TOP_URL") +export REPO_NAME +if [[ -z $TOPDIR ]]; then + echo "$0: ERROR: cannot find top of git repo directory" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi +if [[ ! -e $TOPDIR ]]; then + echo "$0: ERROR: TOPDIR does not exist: $TOPDIR" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi +if [[ ! -d $TOPDIR ]]; then + echo "$0: ERROR: TOPDIR is not a directory: $TOPDIR" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi + + +# cd to topdir +# +if [[ ! -e $TOPDIR ]]; then + echo "$0: ERROR: cannot cd to non-existent path: $TOPDIR" 1>&2 + exit 6 +fi +if [[ ! -d $TOPDIR ]]; then + echo "$0: ERROR: cannot cd to a non-directory: $TOPDIR" 1>&2 + exit 6 +fi +export CD_FAILED +if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to: cd $TOPDIR" 1>&2 +fi +cd "$TOPDIR" || CD_FAILED="true" +if [[ -n $CD_FAILED ]]; then + echo "$0: ERROR: cd $TOPDIR failed" 1>&2 + exit 6 +fi +if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: now in directory: $(/bin/pwd)" 1>&2 +fi + + +# verify that we have an author subdirectory +# +export AUTHOR_PATH="$TOPDIR/author" +if [[ ! -d $AUTHOR_PATH ]]; then + echo "$0: ERROR: author is not a directory under topdir: $AUTHOR_PATH" 1>&2 + exit 6 +fi +export AUTHOR_DIR="author" + + +# verify we have a non-empty readable .top file +# +export TOP_FILE=".top" +if [[ ! -e $TOP_FILE ]]; then + echo "$0: ERROR: .top does not exist: $TOP_FILE" 1>&2 + exit 6 +fi +if [[ ! -f $TOP_FILE ]]; then + echo "$0: ERROR: .top is not a regular file: $TOP_FILE" 1>&2 + exit 6 +fi +if [[ ! -r $TOP_FILE ]]; then + echo "$0: ERROR: .top is not an readable file: $TOP_FILE" 1>&2 + exit 6 +fi +if [[ ! -s $TOP_FILE ]]; then + echo "$0: ERROR: .top is not a non-empty readable file: $TOP_FILE" 1>&2 + exit 6 +fi + + +# print running info if verbose +# +# If -v 3 or higher, print exported variables in order that they were exported. +# +if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: VERSION=$VERSION" 1>&2 + echo "$0: debug[3]: NAME=$NAME" 1>&2 + echo "$0: debug[3]: V_FLAG=$V_FLAG" 1>&2 + echo "$0: debug[3]: GIT_TOOL=$GIT_TOOL" 1>&2 + echo "$0: debug[3]: TOPDIR=$TOPDIR" 1>&2 + echo "$0: debug[3]: DOCROOT_SLASH=$DOCROOT_SLASH" 1>&2 + echo "$0: debug[3]: TAGLINE=$TAGLINE" 1>&2 + echo "$0: debug[3]: REPO_TOP_URL=$REPO_TOP_URL" 1>&2 + echo "$0: debug[3]: REPO_URL=$REPO_URL" 1>&2 + echo "$0: debug[3]: SITE_URL=$SITE_URL" 1>&2 + echo "$0: debug[3]: NOOP=$NOOP" 1>&2 + echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2 + echo "$0: debug[3]: EXIT_CODE=$EXIT_CODE" 1>&2 + echo "$0: debug[3]: REPO_NAME=$REPO_NAME" 1>&2 + echo "$0: debug[3]: CD_FAILED=$CD_FAILED" 1>&2 + echo "$0: debug[3]: AUTHOR_PATH=$AUTHOR_PATH" 1>&2 + echo "$0: debug[3]: AUTHOR_DIR=$AUTHOR_DIR" 1>&2 +fi + + +# determine the number of author_handle.json files +# +AUTHOR_COUNT=$(find "$AUTHOR_DIR" -type f -name '*.json' -print | wc -l) +export AUTHOR_COUNT +if [[ -z $AUTHOR_COUNT || $AUTHOR_COUNT -le 0 ]]; then + echo "$0: ERROR: found no author_handle.json files under: $AUTHOR_DIR" 1>&2 + exit 10 +fi +if [[ $AUTHOR_COUNT -le 1 ]]; then + echo "$0: ERROR: expected more than 1 author_handle.json files under: $AUTHOR_DIR" 1>&2 + exit 11 +fi + + +# -N stops early before any processing is performed +# +if [[ -n $DO_NOT_PROCESS ]]; then + if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: arguments parsed, -N given, exiting 0" 1>&2 + fi + exit 0 +fi + + +# create a list of all author/author_handle files +# +export TMP_AUTHOR_HANDLE_LIST=".tmp.$NAME.AUTHOR_HANDLE_LIST.$$.tmp" +if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: list pf author_handle JSON files: $TMP_AUTHOR_HANDLE_LIST" 1>&2 +fi +trap 'rm -f $TMP_AUTHOR_HANDLE_LIST; exit' 0 1 2 3 15 +rm -f "$TMP_AUTHOR_HANDLE_LIST" +if [[ -e $TMP_AUTHOR_HANDLE_LIST ]]; then + echo "$0: ERROR: cannot remove list pf author_handle JSON files: $TMP_AUTHOR_HANDLE_LIST" 1>&2 + exit 12 +fi +find "$AUTHOR_DIR" -type f -name '*.json' -print | LC_ALL=C sort -t / -d > "$TMP_AUTHOR_HANDLE_LIST" +if [[ ! -e $TMP_AUTHOR_HANDLE_LIST ]]; then + echo "$0: ERROR: cannot create list pf author_handle JSON files: $TMP_AUTHOR_HANDLE_LIST" 1>&2 + exit 13 +fi + + +# combine all author_handle.json files +# +export SED_PART1='s;^(\s*)("sort_word" : )"(.*)",$;\1"sort_word.' +export SED_PART2='" : [\n\1 {\n\1 \2"\3 ' +export SED_PART3='"\n\1 }\n\1],;' +export count=0 +echo "{" +for AUTHOR_HANDLE_JSON in $(< "$TMP_AUTHOR_HANDLE_LIST"); do + + # count file + # + ((++count)) + + # case: not the last file + # + # Remove 1st and last lines, and add , (comma) to the end of the last line. + # + # We also replace "winning_entry_set" with "winning_entry_set.FILENO" where + # FILENO is the file number. + # + # We also convert where FILENO is the file number and FILENAME is the filename: + # + # "sort_word" : "data", + # + # into: + # + # "sort_word.FILENO" : [ + # { + # "sort_word" : "data FILENAME" + # } + # ], + # + if [[ $count -lt $AUTHOR_COUNT ]]; then + + tail -n +2 "$AUTHOR_HANDLE_JSON" | + head -n -1 | + sed -E -e 's/"winning_entry_set"/"winning_entry_set.'"$count"'"/' \ + -e "$SED_PART1$count$SED_PART2$AUTHOR_HANDLE_JSON$SED_PART3" \ + -e '$s/$/,/' + + # case: last file + # + else + + tail -n +2 "$AUTHOR_HANDLE_JSON" | + head -n -1 | + sed -E -e 's/"winning_entry_set"/"winning_entry_set.'"$count"'"/' \ + -e "$SED_PART1$count$SED_PART2$AUTHOR_HANDLE_JSON$SED_PART3" + echo '}' + + fi +done + + +# All Done!!! All Done!!! -- Jessica Noll, Age 2 +# +if [[ $EXIT_CODE -ne 0 ]]; then + echo "$0: Warning: about to exit non-zero: $EXIT_CODE" 1>&2 +elif [[ $V_FLAG -ge 1 ]]; then + echo "$0: debug[1]: about to exit $EXIT_CODE" 1>&2 +fi +exit "$EXIT_CODE" diff --git a/bin/csv2entry.sh b/bin/csv2entry.sh index 279b38990..0dcf647c6 100755 --- a/bin/csv2entry.sh +++ b/bin/csv2entry.sh @@ -142,7 +142,7 @@ shopt -s lastpipe # explicitly run the last command of the pipe line in the curr # set variables referenced in the usage message # -export VERSION="1.2.3 2024-08-30" +export VERSION="1.3 2024-09-23" NAME=$(basename "$0") export NAME export V_FLAG=0 @@ -169,8 +169,6 @@ export NOOP= export DO_NOT_PROCESS= export EXIT_CODE="0" # -JPARSE_TOOL=$(type -P jparse) -export JPARSE_TOOL export AUTHOR_WINS_CSV="author_wins.csv" export MANIFEST_CSV="manifest.csv" export YEAR_PRIZE_CSV="year_prize.csv" @@ -289,7 +287,7 @@ function canonicalize_csv # usage # -export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-n] [-N] [-j jparse] +export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-n] [-N] [author_wins.csv [manifest.csv [year_prize.csv]]] -h print help message and exit @@ -301,8 +299,6 @@ export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-n] [-N] [-j jparse] -n go thru the actions, but do not update any files (def: do the action) -N do not process anything, just parse arguments (def: process something) - -j jparse path to the jparse tool (def: $JPARSE_TOOL) - author_wins.csv where to form author_wins.csv (def: $AUTHOR_WINS_CSV) manifest.csv where to form manifest.csv (def: $MANIFEST_CSV) year_prize.csv where to form year_prize.csv (def: $YEAR_PRIZE_CSV) @@ -325,7 +321,7 @@ $NAME version: $VERSION" # parse command line # -while getopts :hv:Vd:D:nNj: flag; do +while getopts :hv:Vd:D:nN flag; do case "$flag" in h) echo "$USAGE" 1>&2 exit 2 @@ -341,8 +337,6 @@ while getopts :hv:Vd:D:nNj: flag; do ;; N) DO_NOT_PROCESS="-N" ;; - j) JPARSE_TOOL="$OPTARG" - ;; \?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2 echo 1>&2 print_usage 1>&2 @@ -490,7 +484,7 @@ export BIN_DIR="bin" # verify that the bin/jval-wrapper.sh tool is executable # -JVAL_WRAPPER="$BIN_DIR/jval-wrapper.sh" +export JVAL_WRAPPER="$BIN_DIR/jval-wrapper.sh" if [[ ! -e $JVAL_WRAPPER ]]; then echo "$0: ERROR: bin/jval-wrapper.sh does not exist: $JVAL_WRAPPER" 1>&2 exit 5 @@ -505,18 +499,36 @@ if [[ ! -x $JVAL_WRAPPER ]]; then fi -# verify jparse tool +# verify that the bin/jval-wrapper.sh tool is executable # -if [[ -z "$JPARSE_TOOL" ]]; then - echo "$0: ERROR: cannot find an executable jparse tool" 1>&2 +export UNICODE_CHK="$BIN_DIR/unicode-chk.sh" +if [[ ! -e $UNICODE_CHK ]]; then + echo "$0: ERROR: bin/unicode-chk.sh does not exist: $UNICODE_CHK" 1>&2 exit 5 fi -if [[ ! -f "$JPARSE_TOOL" ]]; then - echo "$0: ERROR: jparse is not a file: $JPARSE_TOOL" 1>&2 +if [[ ! -f $UNICODE_CHK ]]; then + echo "$0: ERROR: bin/unicode-chk.sh is not a regular file: $UNICODE_CHK" 1>&2 exit 5 fi -if [[ ! -x "$JPARSE_TOOL" ]]; then - echo "$0: ERROR: jparse is not an executable file: $JPARSE_TOOL" 1>&2 +if [[ ! -x $UNICODE_CHK ]]; then + echo "$0: ERROR: bin/unicode-chk.sh is not an executable file: $UNICODE_CHK" 1>&2 + exit 5 +fi + + +# verify that the bin/combine_author_handle.sh tool is executable +# +export COMBINE_AUTHOR="$BIN_DIR/combine_author_handle.sh" +if [[ ! -e $COMBINE_AUTHOR ]]; then + echo "$0: ERROR: bin/unicode-chk.sh does not exist: $COMBINE_AUTHOR" 1>&2 + exit 5 +fi +if [[ ! -f $COMBINE_AUTHOR ]]; then + echo "$0: ERROR: bin/unicode-chk.sh is not a regular file: $COMBINE_AUTHOR" 1>&2 + exit 5 +fi +if [[ ! -x $COMBINE_AUTHOR ]]; then + echo "$0: ERROR: bin/unicode-chk.sh is not an executable file: $COMBINE_AUTHOR" 1>&2 exit 5 fi @@ -534,7 +546,6 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: NOOP=$NOOP" 1>&2 echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2 echo "$0: debug[3]: EXIT_CODE=$EXIT_CODE" 1>&2 - echo "$0: debug[3]: JPARSE_TOOL=$JPARSE_TOOL" 1>&2 echo "$0: debug[3]: AUTHOR_WINS_CSV=$AUTHOR_WINS_CSV" 1>&2 echo "$0: debug[3]: MANIFEST_CSV=$MANIFEST_CSV" 1>&2 echo "$0: debug[3]: YEAR_PRIZE_CSV=$YEAR_PRIZE_CSV" 1>&2 @@ -546,6 +557,8 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: BIN_PATH=$BIN_DIR" 1>&2 echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2 echo "$0: debug[3]: JVAL_WRAPPER=$JVAL_WRAPPER" 1>&2 + echo "$0: debug[3]: UNICODE_CHK=$UNICODE_CHK" 1>&2 + echo "$0: debug[3]: COMBINE_AUTHOR=$COMBINE_AUTHOR" 1>&2 echo "$0: debug[3]: DOT_ENTRY_JSON_BASENAME=$DOT_ENTRY_JSON_BASENAME" 1>&2 echo "$0: debug[3]: ENTRY_JSON_FORMAT_VERSION=$ENTRY_JSON_FORMAT_VERSION" 1>&2 echo "$0: debug[3]: NO_COMMENT=$NO_COMMENT" 1>&2 @@ -705,26 +718,46 @@ if [[ ! -e $TMP_EXIT_CODE ]]; then fi -# create a temporary author_handle inventory file +# create a temporary sorted list of author_handles from author/author_handle.json files # -export TMP_AUTHOR_HANDLE_INVENTORY=".tmp.$NAME.AUTHOR_HANDLE_INVENTORY.$$.tmp" +export TMP_AUTHOR_HANDLE_FROM_FILES=".tmp.$NAME.AUTHOR_HANDLE_FROM_FILES.$$.tmp" if [[ $V_FLAG -ge 3 ]]; then - echo "$0: debug[3]: temporary author_handle inventory file: $TMP_AUTHOR_HANDLE_INVENTORY" 1>&2 + echo "$0: debug[3]: temporary sorted list of author_handles: $TMP_AUTHOR_HANDLE_FROM_FILES" 1>&2 fi trap 'rm -f $TMP_AUTHOR_WINS_CSV $TMP_MANIFEST_CSV $TMP_YEAR_PRIZE_CSV $TMP_YYYY_DIR_INV $TMP_YYYY_DIR_INV \ - $TMP_2ND_YYYY_DIR_INV $TMP_ENTRY_JSON $TMP_EXIT_CODE $TMP_AUTHOR_HANDLE_INVENTORY; exit' 0 1 2 3 15 -rm -f "$TMP_AUTHOR_HANDLE_INVENTORY" -if [[ -e $TMP_AUTHOR_HANDLE_INVENTORY ]]; then - echo "$0: ERROR: cannot remove temporary author_handle inventory file: $TMP_AUTHOR_HANDLE_INVENTORY" 1>&2 + $TMP_2ND_YYYY_DIR_INV $TMP_ENTRY_JSON $TMP_EXIT_CODE $TMP_AUTHOR_HANDLE_FROM_FILES; exit' 0 1 2 3 15 +rm -f "$TMP_AUTHOR_HANDLE_FROM_FILES" +if [[ -e $TMP_AUTHOR_HANDLE_FROM_FILES ]]; then + echo "$0: ERROR: cannot remove temporary sorted list of author_handles: $TMP_AUTHOR_HANDLE_FROM_FILES" 1>&2 exit 24 fi -: > "$TMP_AUTHOR_HANDLE_INVENTORY" -if [[ ! -e $TMP_AUTHOR_HANDLE_INVENTORY ]]; then - echo "$0: ERROR: cannot create temporary author_handle inventory file: $TMP_AUTHOR_HANDLE_INVENTORY" 1>&2 +: > "$TMP_AUTHOR_HANDLE_FROM_FILES" +if [[ ! -e $TMP_AUTHOR_HANDLE_FROM_FILES ]]; then + echo "$0: ERROR: cannot create temporary sorted list of author_handles: $TMP_AUTHOR_HANDLE_FROM_FILES" 1>&2 exit 25 fi +# create a temporary of author_handles from CSV file +# +export TMP_AUTHOR_HANDLE_FROM_CSV=".tmp.$NAME.AUTHOR_HANDLE_FROM_CSV.$$.tmp" +if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: temporary sorted list of author_handles: $TMP_AUTHOR_HANDLE_FROM_CSV" 1>&2 +fi +trap 'rm -f $TMP_AUTHOR_WINS_CSV $TMP_MANIFEST_CSV $TMP_YEAR_PRIZE_CSV $TMP_YYYY_DIR_INV $TMP_YYYY_DIR_INV \ + $TMP_2ND_YYYY_DIR_INV $TMP_ENTRY_JSON $TMP_EXIT_CODE $TMP_AUTHOR_HANDLE_FROM_CSV; exit' 0 1 2 3 15 +rm -f "$TMP_AUTHOR_HANDLE_FROM_CSV" +if [[ -e $TMP_AUTHOR_HANDLE_FROM_CSV ]]; then + echo "$0: ERROR: cannot remove temporary sorted list of author_handles: $TMP_AUTHOR_HANDLE_FROM_CSV" 1>&2 + exit 26 +fi +: > "$TMP_AUTHOR_HANDLE_FROM_CSV" +if [[ ! -e $TMP_AUTHOR_HANDLE_FROM_CSV ]]; then + echo "$0: ERROR: cannot create temporary sorted list of author_handles: $TMP_AUTHOR_HANDLE_FROM_CSV" 1>&2 + exit 26 +fi + + # canonicalize author_wins.csv into temporary author_wins.csv file # if [[ -z $NOOP ]]; then @@ -812,142 +845,70 @@ elif [[ $V_FLAG -ge 3 ]]; then fi -# scan author_wins.csv file for unknown author handles and -# author handles w/o author/author_handle.json file and -# author/author_handle.json file that do not reference the entry id +# form sorted list of author handles from author/author_handle.json files # if [[ $V_FLAG -ge 1 ]]; then - echo "$0: debug[1]: about to: scan for unknown author handles" 1>&2 + echo "$0: debug[1]: forming list of author handles from author/author_handle.json files" 1>&2 +fi +PATTERN='$..entry_id' +"$COMBINE_AUTHOR" | "$JVAL_WRAPPER" -b -q - "$PATTERN" | LC_ALL=C sort -t _ -d -u > "$TMP_AUTHOR_HANDLE_FROM_FILES" +status_codes=("${PIPESTATUS[@]}") +if [[ ${status_codes[*]} =~ [1-9] ]]; then + echo "$0: ERROR: $COMBINE_AUTHOR | $JVAL_WRAPPER -b -q '$PATTERN' | sort ... failed", \ + "error codes: ${status_codes[*]}" 1>&2 + EXIT_CODE="9" # exit 9 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + echo "$EXIT_CODE" > "$TMP_EXIT_CODE" + # NOTE: We do NOT want to stop processing via continue - proceeding instead +fi +if [[ ! -s $TMP_AUTHOR_HANDLE_FROM_FILES ]]; then + echo "$0: ERROR: unable to form sorted list of author handles from author/author_handle.json files" 1>&2 + EXIT_CODE="9" # exit 9 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + echo "$EXIT_CODE" > "$TMP_EXIT_CODE" + # NOTE: We do NOT want to stop processing via continue - proceeding instead fi -export line=0 -sed -e 's/,/ /g' "$AUTHOR_WINS_CSV" | while read -r AUTHOR_HANDLE ENTRY_ID_SET; do - # skip comment lines - # - ((++line)) - if [[ $AUTHOR_HANDLE =~ '#' ]]; then - continue; - fi - # look for unknown author handles - # - export AUTHOR_HANDLE_JSON="$AUTHOR_DIR/$AUTHOR_HANDLE.json" - if [[ ! -f $AUTHOR_DIR/$AUTHOR_HANDLE.json ]]; then - echo "$0: ERROR: entry_id $ENTRY_ID refers to an unknown author_handle: $AUTHOR_HANDLE" 1>&2 - echo "$0: Warning: no such author_handle file: $AUTHOR_HANDLE_JSON" 1>&2 - echo "$0: Warning: line: $line for: $AUTHOR_WINS_CSV flagging unknown author_handle as an error: continuing" 1>&2 - EXIT_CODE="9" # exit 9 - echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 - echo "$EXIT_CODE" > "$TMP_EXIT_CODE" - # NOTE: We do NOT want to stop processing via continue - proceeding instead - - # look for lines with no entry ids - # - elif [[ -z $ENTRY_ID_SET ]]; then - echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV as no entry ids on the line" 1>&2 - EXIT_CODE="8" # exit 8 - echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 - echo "$EXIT_CODE" > "$TMP_EXIT_CODE" - # NOTE: We do NOT want to stop processing via continue - proceeding instead - - # look for unknown entry_id's - # - else - - # verify the JSON of author/author_handle.json - # - if [[ $V_FLAG -ge 7 ]]; then - echo "$0: debug[7]: about to run: $JPARSE_TOOL -q -- $AUTHOR_HANDLE_JSON" 1>&2 - fi - echo "$AUTHOR_HANDLE_JSON" >> "$TMP_AUTHOR_HANDLE_INVENTORY" - if "$JPARSE_TOOL" -q -- "$AUTHOR_HANDLE_JSON"; then - - # case: author/author_handle.json file is valid JSON - # - if [[ $V_FLAG -ge 9 ]]; then - echo "$0: debug[9]: valid JSON for: $AUTHOR_HANDLE_JSON" 1>&2 - fi - - # check each entry id on the line - # - echo "$ENTRY_ID_SET" | sed -e 's/ /\n/g' | while read -r ENTRY_ID; do - - # convert ENTRY_ID into YYYY_DIR - # - export YEAR_DIR=${ENTRY_ID%%_*} - export ENTRY_DIR=${ENTRY_ID#*_} - export YYYY_DIR="${YEAR_DIR}/${ENTRY_DIR}" - - # verify that the entry exists - # - if [[ ! -d $YYYY_DIR ]]; then - - # case: entry does not exist - # - echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV" \ - "refers to unknown entry id: $ENTRY_ID" 1>&2 - EXIT_CODE="7" # exit 7 - echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 - echo "$EXIT_CODE" > "$TMP_EXIT_CODE" - # NOTE: We do NOT want to stop processing via continue - proceeding instead - - # verify that the author/author_handle.json file refers to the entry id - # - else - - # check if the author/author_handle.json file refers to the entry id - # - FOUND=$("$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON" '$..entry_id') - if [[ -z $FOUND ]]; then - echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV" \ - "entry id: $ENTRY_ID not in winning_entry_set of: $AUTHOR_HANDLE_JSON" 1>&2 - EXIT_CODE="9" # exit 9 - echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 - echo "$EXIT_CODE" > "$TMP_EXIT_CODE" - # NOTE: We do NOT want to stop processing via continue - proceeding instead - fi - fi - done - - # case: author/author_handle.json is not valid JSON - # - else - echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV" \ - "is not valid JSON for: $AUTHOR_HANDLE_JSON" 1>&2 - EXIT_CODE="1" # exit 1 - echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 - echo "$EXIT_CODE" > "$TMP_EXIT_CODE" - # NOTE: We do NOT want to stop processing via continue - proceeding instead - fi - fi -done +# form sorted list of author handles from the author_wins.csv file +# +if [[ $V_FLAG -ge 1 ]]; then + echo "$0: debug[1]: forming list of author handles from the author_wins.csv file" 1>&2 +fi +sed -e '/^#/d' -e 's/^[^,]*,//' -e 's/,/\n/g' "$AUTHOR_WINS_CSV" | LC_ALL=C sort -t _ -d -u > "$TMP_AUTHOR_HANDLE_FROM_CSV" +status_codes=("${PIPESTATUS[@]}") +if [[ ${status_codes[*]} =~ [1-9] ]]; then + echo "$0: ERROR: sed -e ... | sort ... failed", \ + "error codes: ${status_codes[*]}" 1>&2 + EXIT_CODE="9" # exit 9 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + echo "$EXIT_CODE" > "$TMP_EXIT_CODE" + # NOTE: We do NOT want to stop processing via continue - proceeding instead +fi +if [[ ! -s $TMP_AUTHOR_HANDLE_FROM_CSV ]]; then + echo "$0: ERROR: unable to form sorted list of author handles from the author_wins.csv file" 1>&2 + EXIT_CODE="9" # exit 9 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + echo "$EXIT_CODE" > "$TMP_EXIT_CODE" + # NOTE: We do NOT want to stop processing via continue - proceeding instead +fi # verify that the author directory contains only author_handle.json files referenced by author_wins.csv # -if [[ $V_FLAG -ge 5 ]]; then - echo "$0: debug[5]: about to: sort $TMP_AUTHOR_HANDLE_INVENTORY -o $TMP_AUTHOR_HANDLE_INVENTORY" 1>&2 -fi -LC_ALL=C sort "$TMP_AUTHOR_HANDLE_INVENTORY" -o "$TMP_AUTHOR_HANDLE_INVENTORY" -status="$?" -if [[ $status -ne 0 ]]; then - echo "$0: ERROR: LC_ALL=C sort $TMP_AUTHOR_HANDLE_INVENTORY -o $TMP_AUTHOR_HANDLE_INVENTORY failed," \ - "error code: $status" 1>&2 - exit 9 -fi -AUTHOR_DIFF=$(find "$AUTHOR_DIR" -mindepth 1 -maxdepth 1 -type f -name '*.json' 2>&1 | - LC_ALL=C sort | - diff -u - "$TMP_AUTHOR_HANDLE_INVENTORY") +AUTHOR_DIFF=$(diff "$TMP_AUTHOR_HANDLE_FROM_FILES" "$TMP_AUTHOR_HANDLE_FROM_CSV" 2>/dev/null) export AUTHOR_DIFF if [[ -n $AUTHOR_DIFF ]]; then echo "$0: ERROR: author handles in author_wins.csv: $AUTHOR_WINS_CSV differs from author directory: $AUTHOR_DIR" 1>&2 echo "$0: Warning: author directory differences start below" 1>&2 echo "$AUTHOR_DIFF" 1>&2 echo "$0: Warning: author directory differences end above" 1>&2 - EXIT_CODE="9" # exit 1 + EXIT_CODE="9" # exit 9 echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 echo "$EXIT_CODE" > "$TMP_EXIT_CODE" # NOTE: We do NOT want to stop processing via continue - proceeding instead +elif [[ $V_FLAG -ge 1 ]]; then + echo "$0: debug[1]: all author handles are accounted for" 1>&2 fi @@ -1478,17 +1439,15 @@ sed -e '/^#/d' -e 's/,/ /g' "$MANIFEST_CSV" | printf "}\n" } >> "$TMP_ENTRY_JSON" - # verify JSON of temporary .entry.json file + # check the JSON file # - if [[ $V_FLAG -ge 7 ]]; then - echo "$0: debug[7]: about to run: $JPARSE_TOOL -q -- $TMP_ENTRY_JSON" 1>&2 + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to check: $PREV_ENTRY_JSON" 1>&2 fi - if "$JPARSE_TOOL" -q -- "$TMP_ENTRY_JSON"; then - if [[ $V_FLAG -ge 9 ]]; then - echo "$0: debug[9]: valid JSON for $PREV_ENTRY_DIR: $TMP_ENTRY_JSON" 1>&2 - fi - else - echo "$0: ERROR: .entry.json is not valid JSON for $PREV_ENTRY_DIR: $TMP_ENTRY_JSON" 1>&2 + "$UNICODE_CHK" "$TMP_ENTRY_JSON" + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: JSON file failed check: $TMP_ENTRY_JSON" 1>&2 exit 1 fi diff --git a/bin/cvt-submission.sh b/bin/cvt-submission.sh index 23566fdd7..dad1b088d 100755 --- a/bin/cvt-submission.sh +++ b/bin/cvt-submission.sh @@ -94,7 +94,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an # set variables referenced in the usage message # -export VERSION="1.1 2024-08-31" +export VERSION="1.2 2024-09-23" NAME=$(basename "$0") export NAME export V_FLAG=0 @@ -155,8 +155,6 @@ export AUTHOR_JSON_FORMAT_VERSION="1.1 2024-02-11" export NOOP= export DO_NOT_PROCESS= # -unset TAR_FILE_SET -declare -ag TAR_FILE_SET export TARBALL_DIR="/var/tmp" # export X_3="X""X""X" @@ -1003,29 +1001,6 @@ export TEMPLATE_README_MD_HEAD="$TEMPLATE_ENTRY_DIR/README.md.head" export TEMPLATE_README_MD_TAIL="$TEMPLATE_ENTRY_DIR/README.md.tail" -# initialize the list of files that we might tar -# -for FILE in "$INFO_JSON" "$AUTH_JSON" "$REMARKS_MD" "$README_MD" "$INDEX_HTML" "$TAR_BZ2"; do - if [[ -f $FILE ]]; then - TAR_FILE_SET+=("$FILE") - fi -done -# -# add each author/author_handle.json file referenced by .auth.json file that exists -# -export PATTERN -PATTERN="\$authors..author_handle" -if [[ $V_FLAG -ge 3 ]]; then - echo "$0: debug[3]: about to run: $JVAL_WRAPPER -b -- $AUTH_JSON $PATTERN" 1>&2 -fi -for AUTHOR_HANDLE in $("$JVAL_WRAPPER" -b -- "$AUTH_JSON" "$PATTERN"); do - AUTHOR_HANDLE_JSON="author/$AUTHOR_HANDLE.json" - if [[ -f $AUTHOR_HANDLE_JSON ]]; then - TAR_FILE_SET+=("$AUTHOR_HANDLE_JSON") - fi -done - - # determine TARBALL filename # NOW=$(date '+%Y%m%d.%H%M%S') @@ -1084,9 +1059,6 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: TEMPLATE_GITIGNORE=$TEMPLATE_GITIGNORE" 1>&2 echo "$0: debug[3]: TEMPLATE_README_MD_HEAD=$TEMPLATE_README_MD_HEAD" 1>&2 echo "$0: debug[3]: TEMPLATE_README_MD_TAIL=$TEMPLATE_README_MD_TAIL" 1>&2 - for index in "${!TAR_FILE_SET[@]}"; do - echo "$0: debug[3]: TAR_FILE_SET[$index]=${TAR_FILE_SET[$index]}" 1>&2 - done echo "$0: debug[3]: TARBALL_DIR=$TARBALL_DIR" 1>&2 echo "$0: debug[3]: NOW=$NOW" 1>&2 echo "$0: debug[3]: TARBALL=$TARBALL" 1>&2 @@ -1206,6 +1178,56 @@ if [[ ! -s $AUTH_JSON ]]; then fi +# initialize the list of files that we might tar +# +unset TAR_FILE_SET +declare -ag TAR_FILE_SET +# +for FILE in "$INFO_JSON" "$AUTH_JSON" "$REMARKS_MD" "$README_MD" "$INDEX_HTML" "$TAR_BZ2"; do + if [[ -f $FILE ]]; then + TAR_FILE_SET+=("$FILE") + fi +done +# +# add each author/author_handle.json file referenced by .auth.json file that exists +# +unset AUTHOR_HANDLE_SET +declare -ag AUTHOR_HANDLE_SET +# +export PATTERN +PATTERN='$..author_handle' +if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $AUTH_JSON '$PATTERN'" 1>&2 +fi +export FOUND_AUTHOR_HANDLES +FOUND_AUTHOR_HANDLES=$("$JVAL_WRAPPER" -b -q -- "$AUTH_JSON" "$PATTERN") +status="$?" +if [[ $status -ne 0 ]]; then + echo "$0: ERROR: $JVAL_WRAPPER -b -q -- $AUTH_JSON '$PATTERN' failed," \ + "error code: $status" 1>&2 + exit 10 +fi +if [[ -z $FOUND_AUTHOR_HANDLES ]]; then + echo "$0: ERROR: found no author_handle's in: $AUTH_JSON" 1>&2 + exit 11 +fi +for AUTHOR_HANDLE in $FOUND_AUTHOR_HANDLES; do + # add referenced by .auth.json if it exists + AUTHOR_HANDLE_JSON="author/$AUTHOR_HANDLE.json" + if [[ -f $AUTHOR_HANDLE_JSON ]]; then + TAR_FILE_SET+=("$AUTHOR_HANDLE_JSON") + fi + # remember the author_handle for later even if it doesn't exist + AUTHOR_HANDLE_SET+=("$AUTHOR_HANDLE") +done +# +if [[ $V_FLAG -ge 3 ]]; then + for index in "${!TAR_FILE_SET[@]}"; do + echo "$0: debug[3]: TAR_FILE_SET[$index]=${TAR_FILE_SET[$index]}" 1>&2 + done +fi + + # perform the semantic check on .info.json and .auth.json # if [[ $V_FLAG -ge 1 ]]; then @@ -1213,7 +1235,7 @@ if [[ $V_FLAG -ge 1 ]]; then "$CHKENTRY_TOOL" -v 1 -- "$INFO_JSON" "$AUTH_JSON" status="$?" if [[ $status -ne 0 ]]; then - echo "$0: $CHKENTRY_TOOL -v 1 -- $INFO_JSON $AUTH_JSON failed," \ + echo "$0: ERROR: $CHKENTRY_TOOL -v 1 -- $INFO_JSON $AUTH_JSON failed," \ "error code: $status" 1>&2 exit 7 fi @@ -1252,7 +1274,7 @@ if [[ -z $NOOP ]]; then echo "$0: ERROR:" \ "TZ=UTC LC_ALL=C $GTAR_TOOL --sort=name" \ "--totals -jcvf $TARBALL ${TAR_FILE_SET[*]} failed, error: $status" 1>&2 - exit 10 + exit 12 fi # case: form the tarball silently @@ -1265,7 +1287,7 @@ if [[ -z $NOOP ]]; then echo "$0: ERROR:" \ "TZ=UTC LC_ALL=C $GTAR_TOOL --sort=name" \ "-jcf $TARBALL ${TAR_FILE_SET[*]} failed, error: $status" 1>&2 - exit 11 + exit 13 fi fi elif [[ $V_FLAG -ge 3 ]]; then @@ -1284,12 +1306,12 @@ if [[ -z $NOOP ]]; then rm -f "$TMP_ENTRY_JSON" if [[ -e $TMP_ENTRY_JSON ]]; then echo "$0: ERROR: cannot remove temporary .entry.json file: $TMP_ENTRY_JSON" 1>&2 - exit 12 + exit 14 fi :> "$TMP_ENTRY_JSON" if [[ ! -e $TMP_ENTRY_JSON ]]; then echo "$0: ERROR: cannot create temporary .entry.json file: $TMP_ENTRY_JSON" 1>&2 - exit 13 + exit 15 fi elif [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: because of -n, temporary .entry.json file is not used: $TMP_ENTRY_JSON" 1>&2 @@ -1307,12 +1329,12 @@ if [[ -z $NOOP ]]; then rm -f "$TMP_AUTHOR_HANDLE_JSON" if [[ -e $TMP_AUTHOR_HANDLE_JSON ]]; then echo "$0: ERROR: cannot remove temporary author_handle file: $TMP_AUTHOR_HANDLE_JSON" 1>&2 - exit 14 + exit 16 fi :> "$TMP_AUTHOR_HANDLE_JSON" if [[ ! -e $TMP_AUTHOR_HANDLE_JSON ]]; then echo "$0: ERROR: cannot create temporary author_handle file: $TMP_AUTHOR_HANDLE_JSON" 1>&2 - exit 15 + exit 17 fi elif [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: because of -n, temporary author_handle file is not used: $TMP_AUTHOR_HANDLE_JSON" 1>&2 @@ -1330,12 +1352,12 @@ if [[ -z $NOOP ]]; then rm -f "$TMP_MANIFEST_CSV" if [[ -e $TMP_MANIFEST_CSV ]]; then echo "$0: ERROR: cannot remove temporary manifest csv file: $TMP_MANIFEST_CSV" 1>&2 - exit 16 + exit 18 fi :> "$TMP_MANIFEST_CSV" if [[ ! -e $TMP_MANIFEST_CSV ]]; then echo "$0: ERROR: cannot create temporary manifest csv file: $TMP_MANIFEST_CSV" 1>&2 - exit 17 + exit 19 fi elif [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: because of -n, temporary manifest csv file is not used: $TMP_MANIFEST_CSV" 1>&2 @@ -1395,7 +1417,7 @@ if [[ -z $NOOP ]]; then # SC2129 (style): Consider using { cmd1; cmd2; } >> file instead of individual redirects # https://www.shellcheck.net/wiki/SC2129 # shellcheck disable=SC2129 - for AUTHOR_HANDLE in $("$JVAL_WRAPPER" -w -b -- "$AUTH_JSON" "\$authors..author_handle"); do + for AUTHOR_HANDLE in "${AUTHOR_HANDLE_SET[@]}"; do # case: not the first author_handle # @@ -1414,12 +1436,22 @@ if [[ -z $NOOP ]]; then # parse an element value of the this particular author in the authors JSON array # + PATTERN='$.authors['"$i"']' if [[ $V_FLAG -ge 5 ]]; then - echo "$0: debug[5]: about to run: $JVAL_WRAPPER -u $AUTH_JSON \$.authors[$i]" 1>&2 + echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $AUTH_JSON '$PATTERN'" 1>&2 fi - AUTHOR_DATA=$("$JVAL_WRAPPER" -u -- "$AUTH_JSON" "\$.authors[$i]" | - sed -e 's/^\["//' -e 's/^\([^"]*\)"]/\1/') export AUTHOR_DATA + AUTHOR_DATA=$("$JVAL_WRAPPER" -b -q -- "$AUTH_JSON" "$PATTERN") + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: $JVAL_WRAPPER -b -q -- $AUTH_JSON '$PATTERN' failed," \ + "error code: $status" 1>&2 + exit 20 + fi + if [[ -z $AUTHOR_DATA ]]; then + echo "$0: ERROR: found nothing for author[$i] in: $AUTH_JSON" 1>&2 + exit 21 + fi export AUTHOR_PAST_WINNING_AUTHOR= export AUTHOR_DEFAULT_HANDLE= export AUTHOR_AUTHOR_NUMBER= @@ -1461,20 +1493,20 @@ if [[ -z $NOOP ]]; then author_handle) if [[ \"$AUTHOR_HANDLE\" != "$VALUE" ]]; then echo "$0: ERROR: AUTHOR_HANDLE: \"$AUTHOR_HANDLE\" != author_handle value: $VALUE" 1>&2 - exit 18 + exit 22 fi AUTHOR_INFO[author_handle]="$VALUE" ;; author_number) if [[ $i != "$VALUE" ]]; then echo "$0: ERROR: AUTHOR_HANDLE: $AUTHOR_HANDLE i: $i != author_number: $VALUE" 1>&2 - exit 19 + exit 23 fi AUTHOR_AUTHOR_NUMBER="$VALUE" ;; *) echo "$0: ERROR: AUTHOR_HANDLE: $AUTHOR_HANDLE unknown name: $NAME" 1>&2 - exit 20 + exit 24 ;; esac done <<< "$AUTHOR_DATA" @@ -1484,20 +1516,20 @@ if [[ -z $NOOP ]]; then for NAME in name location_code email url alt_url mastodon github affiliation author_handle; do if [[ -z ${AUTHOR_INFO[${NAME}]} ]]; then echo "$0: ERROR: AUTHOR_HANDLE[$i]: $AUTHOR_HANDLE missing $NAME value from: $AUTH_JSON" 1>&2 - exit 21 + exit 25 fi done if [[ -z $AUTHOR_PAST_WINNING_AUTHOR ]]; then echo "$0: ERROR: AUTHOR_HANDLE[$i]: $AUTHOR_HANDLE missing past_winning_author value from: $AUTH_JSON" 1>&2 - exit 22 + exit 26 fi if [[ -z $AUTHOR_DEFAULT_HANDLE ]]; then echo "$0: ERROR: AUTHOR_HANDLE[$i]: $AUTHOR_HANDLE missing default_handle value from: $AUTH_JSON" 1>&2 - exit 23 + exit 27 fi if [[ -z $AUTHOR_AUTHOR_NUMBER ]]; then echo "$0: ERROR: AUTHOR_HANDLE[$i]: $AUTHOR_HANDLE missing author_number value from: $AUTH_JSON" 1>&2 - exit 24 + exit 28 fi # determine sort_word from the Author's last name @@ -1506,7 +1538,7 @@ if [[ -z $NOOP ]]; then export SORT_WORD if [[ -z $SORT_WORD ]]; then echo "$0: ERROR: AUTHOR_HANDLE: $AUTHOR_HANDLE unable to determine the sort_word for: ${AUTHOR_INFO[name]}" 1>&2 - exit 25 + exit 29 fi case "$SORT_WORD" in [a-z]*) ;; @@ -1545,22 +1577,35 @@ if [[ -z $NOOP ]]; then # if [[ ! -f $AUTHOR_HANDLE_JSON ]]; then echo "$0: ERROR: author_handle JSON is not a file: $AUTHOR_HANDLE_JSON" 1>&2 - exit 26 + exit 30 fi if [[ ! -r $AUTHOR_HANDLE_JSON ]]; then echo "$0: ERROR: author_handle JSON is not a readable file: $AUTHOR_HANDLE_JSON" 1>&2 - exit 27 + exit 31 fi if [[ ! -s $AUTHOR_HANDLE_JSON ]]; then echo "$0: ERROR: author_handle JSON is not a non-empty readable file: $AUTHOR_HANDLE_JSON" 1>&2 - exit 28 + exit 32 fi # parse author/author_handle.json file other than the winning_entry_set # - AUTHOR_DATA=$("$JVAL_WRAPPER" -u "$AUTHOR_HANDLE_JSON" | - grep -v winning_entry_set | - sed -e 's/^\["//' -e 's/^\([^"]*\)"]/\1/') + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to run: $JVAL_WRAPPER -q -- $AUTHOR_HANDLE_JSON | sed -E -e ..." 1>&2 + fi + export AUTHOR_DATA + AUTHOR_DATA=$("$JVAL_WRAPPER" -q -- "$AUTHOR_HANDLE_JSON" | + sed -E -e '/^entry_id\t/d') + status_codes=("${PIPESTATUS[@]}") + if [[ ${status_codes[*]} =~ [1-9] ]]; then + echo "$0: ERROR: $JVAL_WRAPPER -q -- $AUTHOR_HANDLE_JSON | sed -E -e ... failed," \ + "error codes: ${status_codes[*]}" 1>&2 + exit 33 + fi + if [[ -z $AUTHOR_DATA ]]; then + echo "$0: ERROR: found entry_id's in: $AUTH_JSON" 1>&2 + exit 34 + fi while read -r NAME VALUE; do # parse name value got non @@ -1570,21 +1615,21 @@ if [[ -z $NOOP ]]; then if [[ $VALUE != \"$NO_COMMENT\" ]]; then echo "$0: ERROR: author_handle file: $AUTHOR_HANDLE_JSON" \ "invalid no_comment: $VALUE != \"$NO_COMMENT\"" 1>&2 - exit 29 + exit 35 fi ;; author_JSON_format_version) if [[ $VALUE != \"$AUTHOR_JSON_FORMAT_VERSION\" ]]; then echo "$0: ERROR: author_handle file: $AUTHOR_HANDLE_JSON" \ "invalid author_JSON_format_version: $VALUE != \"$AUTHOR_JSON_FORMAT_VERSION\"" 1>&2 - exit 30 + exit 36 fi ;; author_handle) if [[ $VALUE != "${AUTHOR_INFO[author_handle]}" ]]; then echo "$0: ERROR: author_handle file: $AUTHOR_HANDLE_JSON" \ "invalid author_handle: $VALUE != ${AUTHOR_INFO[author_handle]}" 1>&2 - exit 31 + exit 37 fi ;; full_name) @@ -1663,17 +1708,31 @@ if [[ -z $NOOP ]]; then *) echo "$0: ERROR: author_handle file: $AUTHOR_HANDLE_JSON" \ "unknown $NAME : $VALUE" 1>&2 - exit 32 + exit 38 ;; esac done <<< "$AUTHOR_DATA" # collect the entry_ids from the winning_entry_set and add the new entry_id # - ENTRY_IDS=$( { "$JVAL_WRAPPER" -w -b -- "$AUTHOR_HANDLE_JSON" "\$..entry_id" | - sed -e 's/^\["//' -e 's/^\([^"]*\)"]/\1/' + PATTERN='$..entry_id' + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $AUTHOR_HANDLE_JSON '$PATTERN'" 1>&2 + fi + export ENTRY_IDS + ENTRY_IDS=$( { "$JVAL_WRAPPER" -b -q -- "$AUTHOR_HANDLE_JSON" "$PATTERN"; + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: $JVAL_WRAPPER -b -q -- $AUTHOR_HANDLE_JSON '$PATTERN' failed," \ + "error code: $status" 1>&2 + exit 39 + fi echo "$ENTRY_ID" } | LC_ALL=C sort -d -u ) + if [[ -z $AUTHOR_DATA ]]; then + echo "$0: ERROR: found no entry_id's in: $AUTHOR_HANDLE_JSON" 1>&2 + exit 40 + fi while read -r ENTRY_ID_SET_VALUE; do ENTRY_ID_SET+=("$ENTRY_ID_SET_VALUE") done <<< "$ENTRY_IDS" @@ -1692,7 +1751,7 @@ if [[ -z $NOOP ]]; then echo "$0: ERROR: write_author_handle_file AUTHOR_INFO ENTRY_ID_SET" \ "$TMP_AUTHOR_HANDLE_JSON $AUTHOR_HANDLE_JSON failed," \ "error code: $status" 1>&2 - exit 33 + exit 41 fi # output the author_handle information for .entry.json @@ -1718,17 +1777,17 @@ if [[ -z $NOOP ]]; then # leave out the trailing comma. # ((++count)) - PATTERN="\$manifest..c_src" - if [[ $V_FLAG -ge 3 ]]; then - echo "$0: debug[3]: about to run: $JVAL_WRAPPER -b -- $INFO_JSON $PATTERN" 1>&2 + PATTERN='$..c_src' + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $INFO_JSON '$PATTERN'" 1>&2 fi export FILE_PATH - FILE_PATH=$("$JVAL_WRAPPER" -b -- "$INFO_JSON" "$PATTERN") + FILE_PATH=$("$JVAL_WRAPPER" -b -q -- "$INFO_JSON" "$PATTERN") status="$?" if [[ $status -ne 0 ]]; then - echo "$0: ERROR: $JVAL_WRAPPER -b -- $INFO_JSON $PATTERN failed," \ + echo "$0: ERROR: $JVAL_WRAPPER -b -q -- $INFO_JSON '$PATTERN' failed," \ "error code: $status" 1>&2 - exit 34 + exit 42 fi if [[ -z $FILE_PATH ]]; then echo "$0: ERROR: cannot determine c_src in manifest from: $INFO_JSON" 1>&2 @@ -1736,11 +1795,11 @@ if [[ -z $NOOP ]]; then fi if [[ ! -e $YYYY_DIR/$FILE_PATH ]]; then echo "$0: ERROR: manifest file for c_src does not exist: $YYYY_DIR/$FILE_PATH" 1>&2 - exit 35 + exit 43 fi if [[ ! -f $YYYY_DIR/$FILE_PATH ]]; then echo "$0: ERROR: manifest file for c_src is not a file: $YYYY_DIR/$FILE_PATH" 1>&2 - exit 36 + exit 44 fi manifest_csv "$FILE_PATH" 20 true c true "entry source code" >> "$TMP_MANIFEST_CSV" export PROG_C="$FILE_PATH" @@ -1748,29 +1807,29 @@ if [[ -z $NOOP ]]; then # write manifest csv line for entry Makefile # ((++count)) - PATTERN="\$manifest..Makefile" - if [[ $V_FLAG -ge 3 ]]; then - echo "$0: debug[3]: about to run: $JVAL_WRAPPER -b -- $INFO_JSON $PATTERN" 1>&2 + PATTERN='$..Makefile' + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $INFO_JSON '$PATTERN'" 1>&2 fi export FILE_PATH - FILE_PATH=$("$JVAL_WRAPPER" -b -- "$INFO_JSON" "$PATTERN") + FILE_PATH=$("$JVAL_WRAPPER" -b -q -- "$INFO_JSON" "$PATTERN") status="$?" if [[ $status -ne 0 ]]; then - echo "$0: ERROR: $JVAL_WRAPPER -b -- $INFO_JSON $PATTERN failed," \ + echo "$0: ERROR: $JVAL_WRAPPER -b -q -- $INFO_JSON '$PATTERN' failed," \ "error code: $status" 1>&2 - exit 37 + exit 45 fi if [[ -z $FILE_PATH ]]; then echo "$0: ERROR: cannot determine Makefile in manifest from: $INFO_JSON" 1>&2 - exit 38 + exit 46 fi if [[ ! -e $YYYY_DIR/$FILE_PATH ]]; then echo "$0: ERROR: manifest file for Makefile does not exist: $YYYY_DIR/$FILE_PATH" 1>&2 - exit 39 + exit 47 fi if [[ ! -f $YYYY_DIR/$FILE_PATH ]]; then echo "$0: ERROR: manifest file for Makefile is not a file: $YYYY_DIR/$FILE_PATH" 1>&2 - exit 40 + exit 48 fi manifest_csv "$FILE_PATH" 30 true makefile true "entry Makefile" >> "$TMP_MANIFEST_CSV" @@ -1798,7 +1857,7 @@ if [[ -z $NOOP ]]; then if [[ $status -ne 0 ]]; then echo "$0: ERROR: cp -f -p -v $YYYY_DIR/$PROG_C $YYYY_DIR/$ORIG_C failed," \ "error code: $status" 1>&2 - exit 41 + exit 49 fi else cp -f -p "$YYYY_DIR/$PROG_C" "$YYYY_DIR/$ORIG_C" @@ -1806,7 +1865,7 @@ if [[ -z $NOOP ]]; then if [[ $status -ne 0 ]]; then echo "$0: ERROR: cp -f -p $YYYY_DIR/$PROG_C $YYYY_DIR/$ORIG_C failed," \ "error code: $status" 1>&2 - exit 42 + exit 50 fi fi fi @@ -1819,17 +1878,17 @@ if [[ -z $NOOP ]]; then # collect manifest for extra_file(s) # export PATTERN - PATTERN="\$manifest..extra_file" - if [[ $V_FLAG -ge 3 ]]; then - echo "$0: debug[3]: about to run: $JVAL_WRAPPER -b -- $INFO_JSON $PATTERN" 1>&2 + PATTERN='$..extra_file' + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $INFO_JSON '$PATTERN' | ,,, sort ..." 1>&2 fi export EXTRA_FILE_SET - EXTRA_FILE_SET=$("$JVAL_WRAPPER" -b -- "$INFO_JSON" "$PATTERN" | LC_ALL=C sort -d -u) + EXTRA_FILE_SET=$("$JVAL_WRAPPER" -b -q -- "$INFO_JSON" "$PATTERN" | LC_ALL=C sort -d -u) status_codes=("${PIPESTATUS[@]}") if [[ ${status_codes[*]} =~ [1-9] ]]; then - echo "$0: ERROR: $JVAL_WRAPPER -b -- $INFO_JSON $PATTERN failed," \ + echo "$0: ERROR: $JVAL_WRAPPER -b -q -- $INFO_JSON '$PATTERN' | ... sort ... failed," \ "error code: ${status_codes[*]}" 1>&2 - exit 43 + exit 51 fi # perform some validation checks on the extra_file set @@ -1841,7 +1900,7 @@ if [[ -z $NOOP ]]; then case "$EXTRA_FILE" in prog.c|Makefile|prog.orig.c|index.html|README.md|remarks.md|\.*) echo "$0: ERROR: invalid extra_file filename: $EXTRA_FILE" 1>&2 - exit 44 + exit 52 ;; *) ;; esac @@ -1850,11 +1909,11 @@ if [[ -z $NOOP ]]; then # if [[ ! -e $YYYY_DIR/$EXTRA_FILE ]]; then echo "$0: ERROR: manifest file for extra_file does not exist: $YYYY_DIR/$EXTRA_FILE" 1>&2 - exit 45 + exit 53 fi if [[ ! -f $YYYY_DIR/$EXTRA_FILE ]]; then echo "$0: ERROR: manifest file for extra_file is not a file: $YYYY_DIR/$EXTRA_FILE" 1>&2 - exit 46 + exit 54 fi done @@ -2007,7 +2066,7 @@ if [[ -z $NOOP ]]; then if [[ $status -ne 0 ]]; then echo "$0: ERROR: cp -f -p -v $TEMPLATE_GITIGNORE $DOT_GITIGNORE failed," \ "error code: $status" 1>&2 - exit 47 + exit 55 fi else cp -f -p "$TEMPLATE_GITIGNORE" "$DOT_GITIGNORE" @@ -2015,7 +2074,7 @@ if [[ -z $NOOP ]]; then if [[ $status -ne 0 ]]; then echo "$0: ERROR: cp -f -p $TEMPLATE_GITIGNORE $DOT_GITIGNORE failed," \ "error code: $status" 1>&2 - exit 48 + exit 56 fi fi fi @@ -2075,7 +2134,7 @@ if [[ -z $NOOP ]]; then if [[ $status -ne 0 ]]; then echo "$0: ERROR: LC_ALL=C sort -t, -k2n -k1,1 -k3,6 $TMP_MANIFEST_CSV -o $TMP_MANIFEST_CSV failed," \ "error code: $status" 1>&2 - exit 49 + exit 57 elif [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: sorted temporary manifest csv file: $TMP_MANIFEST_CSV" 1>&2 fi @@ -2118,7 +2177,7 @@ if [[ -z $NOOP ]]; then if [[ $status -ne 0 ]]; then echo "$0: ERROR: $JPARSE_TOOL -q -- $TMP_ENTRY_JSON filed," \ "error code: $status" 1>&2 - exit 50 + exit 58 elif [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: valid JSON: $TMP_ENTRY_JSON" 1>&2 fi @@ -2150,13 +2209,13 @@ if [[ -z $NOOP ]]; then if [[ $status -ne 0 ]]; then echo "$0: ERROR: mv -f -- $TMP_ENTRY_JSON $ENTRY_JSON filed," \ "error code: $status" 1>&2 - exit 51 + exit 59 elif [[ $V_FLAG -ge 1 ]]; then echo "$0: debug[1]: updated .entry.json: $ENTRY_JSON" 1>&2 fi if [[ ! -s $ENTRY_JSON ]]; then echo "$0: ERROR: not a non-empty .entry.json: $ENTRY_JSON" 1>&2 - exit 52 + exit 60 fi fi @@ -2191,7 +2250,7 @@ for EXTRA_FILE in $EXTRA_FILE_SET; do if [[ $status -ne 0 ]]; then echo "$0: ERROR: $OTHERMD2HTML_SH -v $V_FLAG -- $YYYY_DIR/$EXTRA_FILE filed," \ "error code: $status" 1>&2 - exit 53 + exit 61 fi elif [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: -n disabled forming HTML: $YYYY_DIR/$HTML_FILE" 1>&2 @@ -2219,7 +2278,7 @@ if [[ -z $NOOP ]]; then if [[ $status -ne 0 ]]; then echo "$0: ERROR: $MD2HTML_SH -v $V_FLAG -- $README_MD $INDEX_HTML filed," \ "error code: $status" 1>&2 - exit 54 + exit 62 fi fi @@ -2240,7 +2299,7 @@ status="$?" if [[ $status -ne 0 ]]; then echo "$0: ERROR: rm -f -v -- $INFO_JSON $AUTH_JSON $REMARKS_MD filed," \ "error code: $status" 1>&2 - exit 55 + exit 63 fi diff --git a/bin/entry2csv.sh b/bin/entry2csv.sh index 4a9ee08dc..a4900151c 100755 --- a/bin/entry2csv.sh +++ b/bin/entry2csv.sh @@ -118,7 +118,7 @@ shopt -s lastpipe # explicitly run the last command of the pipe line in the curr # set variables referenced in the usage message # -export VERSION="1.2.1 2024-08-17" +export VERSION="1.3 2024-09-23" NAME=$(basename "$0") export NAME export V_FLAG=0 @@ -139,8 +139,6 @@ export NOOP= export DO_NOT_PROCESS= export EXIT_CODE="0" # -JPARSE_TOOL=$(type -P jparse) -export JPARSE_TOOL export AUTHOR_WINS_CSV="author_wins.csv" export MANIFEST_CSV="manifest.csv" export YEAR_PRIZE_CSV="year_prize.csv" @@ -161,6 +159,7 @@ export AUTHOR_DIR="author" function output_award { local ENTRY_JSON_PATH; # the .entry.json path + local PATTERN; # XPath for JSON pattern # parse args # @@ -184,7 +183,17 @@ function output_award # obtain the award string # - "$JVAL_WRAPPER" -w -b "$ENTRY_JSON_PATH" '$..award' + PATTERN='$..award' + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN'" 1>&2 + fi + "$JVAL_WRAPPER" -b -q "$ENTRY_JSON_PATH" "$PATTERN" | sed -f "$UNICODE_FIX_SED" + status_codes=("${PIPESTATUS[@]}") + if [[ ${status_codes[*]} =~ [1-9] ]]; then + echo "$0: ERROR: in output_award: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN' failed," \ + "error codes: ${status_codes[*]}" 1>&2 + return 5 + fi return 0 } @@ -203,6 +212,7 @@ function output_award function output_author_handles { local ENTRY_JSON_PATH; # the .entry.json path + local PATTERN; # XPath for JSON pattern # parse args # @@ -226,14 +236,24 @@ function output_author_handles # extract author handles from .entry.json # - "$JVAL_WRAPPER" -w -b "$ENTRY_JSON_PATH" '$..author_handle' + PATTERN='$..author_handle' + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN'" 1>&2 + fi + "$JVAL_WRAPPER" -b -q "$ENTRY_JSON_PATH" "$PATTERN" + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: in output_award: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN' failed," \ + "error code: $status" 1>&2 + return 5 + fi return 0 } # usage # -export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-n] [-N] [-j jparse] +export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-n] [-N] [author_wins.csv [manifest.csv [year_prize.csv]]] -h print help message and exit @@ -245,8 +265,6 @@ export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-n] [-N] [-j jparse] -n go thru the actions, but do not update any files (def: do the action) -N do not process anything, just parse arguments (def: process something) - -j jparse path to the jparse tool (def: $JPARSE_TOOL) - author_wins.csv where to form author_wins.csv (def: $AUTHOR_WINS_CSV) manifest.csv where to form manifest.csv (def: $MANIFEST_CSV) year_prize.csv where to form year_prize.csv (def: $YEAR_PRIZE_CSV) @@ -269,7 +287,7 @@ $NAME version: $VERSION" # parse command line # -while getopts :hv:Vd:D:nNj: flag; do +while getopts :hv:Vd:D:nN flag; do case "$flag" in h) echo "$USAGE" 1>&2 exit 2 @@ -285,8 +303,6 @@ while getopts :hv:Vd:D:nNj: flag; do ;; N) DO_NOT_PROCESS="-N" ;; - j) JPARSE_TOOL="$OPTARG" - ;; \?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2 echo 1>&2 print_usage 1>&2 @@ -471,30 +487,31 @@ fi export MANIFEST_ENTRY_CSV_ENTRY_AWK="$BIN_DIR/manifest.csv.entry.awk" if [[ ! -e $MANIFEST_ENTRY_CSV_ENTRY_AWK ]]; then echo "$0: ERROR: bin/manifest.csv.entry.awk does not exist: $MANIFEST_ENTRY_CSV_ENTRY_AWK" 1>&2 - exit 6 + exit 5 fi if [[ ! -f $MANIFEST_ENTRY_CSV_ENTRY_AWK ]]; then echo "$0: ERROR: bin/manifest.csv.entry.awk is not a regular file: $MANIFEST_ENTRY_CSV_ENTRY_AWK" 1>&2 - exit 6 + exit 5 fi if [[ ! -r $MANIFEST_ENTRY_CSV_ENTRY_AWK ]]; then echo "$0: ERROR: bin/manifest.csv.entry.awk is not a readable file: $MANIFEST_ENTRY_CSV_ENTRY_AWK" 1>&2 - exit 6 + exit 5 fi -# verify jparse tool +# verify that the bin/unicode-fix.sed tool is executable # -if [[ -z "$JPARSE_TOOL" ]]; then - echo "$0: ERROR: cannot find an executable jparse tool" 1>&2 +export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed" +if [[ ! -e $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2 exit 5 fi -if [[ ! -f "$JPARSE_TOOL" ]]; then - echo "$0: ERROR: jparse is not a file: $JPARSE_TOOL" 1>&2 +if [[ ! -f $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2 exit 5 fi -if [[ ! -x "$JPARSE_TOOL" ]]; then - echo "$0: ERROR: jparse is not an executable file: $JPARSE_TOOL" 1>&2 +if [[ ! -r $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2 exit 5 fi @@ -512,7 +529,6 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: NOOP=$NOOP" 1>&2 echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2 echo "$0: debug[3]: EXIT_CODE=$EXIT_CODE" 1>&2 - echo "$0: debug[3]: JPARSE_TOOL=$JPARSE_TOOL" 1>&2 echo "$0: debug[3]: AUTHOR_WINS_CSV=$AUTHOR_WINS_CSV" 1>&2 echo "$0: debug[3]: MANIFEST_CSV=$MANIFEST_CSV" 1>&2 echo "$0: debug[3]: YEAR_PRIZE_CSV=$YEAR_PRIZE_CSV" 1>&2 @@ -524,6 +540,7 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: TOP_FILE=$TOP_FILE" 1>&2 echo "$0: debug[3]: BIN_PATH=$BIN_DIR" 1>&2 echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2 + echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2 echo "$0: debug[3]: JVAL_WRAPPER=$JVAL_WRAPPER" 1>&2 echo "$0: debug[3]: MANIFEST_ENTRY_CSV_ENTRY_AWK=$MANIFEST_ENTRY_CSV_ENTRY_AWK" 1>&2 fi @@ -859,21 +876,6 @@ for YYYY in $(< "$TOP_FILE"); do echo "$EXIT_CODE" > "$TMP_EXIT_CODE" continue fi - # - if [[ $V_FLAG -ge 7 ]]; then - echo "$0: debug[7]: about to run: $JPARSE_TOOL -q -- $ENTRY_JSON" 1>&2 - fi - if "$JPARSE_TOOL" -q -- "$ENTRY_JSON"; then - if [[ $V_FLAG -ge 9 ]]; then - echo "$0: debug[9]: valid JSON: $ENTRY_JSON" 1>&2 - fi - else - echo "$0: ERROR: .entry.json is not valid JSON: $ENTRY_JSON" 1>&2 - EXIT_CODE="1" # exit 1 - echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 - echo "$EXIT_CODE" > "$TMP_EXIT_CODE" - continue - fi # process the year_prize.csv update # @@ -1136,71 +1138,49 @@ sed -e 's/,/ /g' "$AUTHOR_WINS_CSV" | while read -r AUTHOR_HANDLE ENTRY_ID_SET; # else - # verify the JSON of author/author_handle.json + # check each entry id on the line # - if [[ $V_FLAG -ge 7 ]]; then - echo "$0: debug[7]: about to run: $JPARSE_TOOL -q -- $AUTHOR_HANDLE_JSON" 1>&2 - fi echo "$AUTHOR_HANDLE_JSON" >> "$TMP_AUTHOR_HANDLE_INVENTORY" - if "$JPARSE_TOOL" -q -- "$AUTHOR_HANDLE_JSON"; then + echo "$ENTRY_ID_SET" | sed -e 's/ /\n/g' | while read -r ENTRY_ID; do - # case: author/author_handle.json file is valid JSON + # convert ENTRY_ID into YYYY_DIR # - if [[ $V_FLAG -ge 9 ]]; then - echo "$0: debug[9]: valid JSON for: $AUTHOR_HANDLE_JSON" 1>&2 - fi + export YEAR_DIR=${ENTRY_ID%%_*} + export ENTRY_DIR=${ENTRY_ID#*_} + export YYYY_DIR="${YEAR_DIR}/${ENTRY_DIR}" - # check each entry id on the line + # verify that the entry exists # - echo "$ENTRY_ID_SET" | sed -e 's/ /\n/g' | while read -r ENTRY_ID; do + if [[ ! -d $YYYY_DIR ]]; then - # convert ENTRY_ID into YYYY_DIR + # case: entry does not exist # - export YEAR_DIR=${ENTRY_ID%%_*} - export ENTRY_DIR=${ENTRY_ID#*_} - export YYYY_DIR="${YEAR_DIR}/${ENTRY_DIR}" + echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV" \ + "refers to unknown entry id: $ENTRY_ID" 1>&2 + EXIT_CODE="7" # exit 7 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + echo "$EXIT_CODE" > "$TMP_EXIT_CODE" + # NOTE: We do NOT want to stop processing via continue - proceeding instead - # verify that the entry exists + # verify that the author/author_handle.json file refers to the entry id + # + else + + # check if the author/author_handle.json file refers to the entry id # - if [[ ! -d $YYYY_DIR ]]; then - - # case: entry does not exist - # + PATTERN='$..entry_id' + FOUND=$("$JVAL_WRAPPER" -b -q "$AUTHOR_HANDLE_JSON" "$PATTERN") + status="$?" + if [[ $status -ne 0 || -z $FOUND ]]; then echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV" \ - "refers to unknown entry id: $ENTRY_ID" 1>&2 - EXIT_CODE="7" # exit 7 + "entry id: $ENTRY_ID not in winning_entry_set of: $AUTHOR_HANDLE_JSON" 1>&2 + EXIT_CODE="9" # exit 9 echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 echo "$EXIT_CODE" > "$TMP_EXIT_CODE" # NOTE: We do NOT want to stop processing via continue - proceeding instead - - # verify that the author/author_handle.json file refers to the entry id - # - else - - # check if the author/author_handle.json file refers to the entry id - # - FOUND=$("$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON" '$..entry_id') - if [[ -z $FOUND ]]; then - echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV" \ - "entry id: $ENTRY_ID not in winning_entry_set of: $AUTHOR_HANDLE_JSON" 1>&2 - EXIT_CODE="9" # exit 9 - echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 - echo "$EXIT_CODE" > "$TMP_EXIT_CODE" - # NOTE: We do NOT want to stop processing via continue - proceeding instead - fi fi - done - - # case: author/author_handle.json is not valid JSON - # - else - echo "$0: ERROR: line: $line of author_wins.csv: $AUTHOR_WINS_CSV" \ - "is not valid JSON for: $AUTHOR_HANDLE_JSON" 1>&2 - EXIT_CODE="1" # exit 1 - echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 - echo "$EXIT_CODE" > "$TMP_EXIT_CODE" - # NOTE: We do NOT want to stop processing via continue - proceeding instead - fi + fi + done fi done diff --git a/bin/gen-authors.sh b/bin/gen-authors.sh index c7da3df3f..3a94fb237 100755 --- a/bin/gen-authors.sh +++ b/bin/gen-authors.sh @@ -88,7 +88,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an # set variables referenced in the usage message # -export VERSION="1.12 2024-08-18" +export VERSION="1.13 2024-09-23" NAME=$(basename "$0") export NAME export V_FLAG=0 @@ -200,175 +200,6 @@ Exit codes: $NAME version: $VERSION" -# output_entry_ids -# -# Write the entry ids from an author/author_handle.json file to standard output (stdout) -# -# usage: -# output_entry_ids author/author_handle.json -# -# returns: -# 0 ==> no errors detected, but output may be empty -# > 0 ==> function error number -# -function output_entry_ids -{ - local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path - local entry_id; # a found entry id - - # parse args - # - if [[ $# -ne 1 ]]; then - echo "$0: ERROR: in output_entry_ids: expected 1 arg, found $#" 1>&2 - return 1 - fi - AUTHOR_HANDLE_JSON_PATH="$1" - if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_entry_ids: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 2 - fi - if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_entry_ids: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 3 - fi - if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_entry_ids: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 4 - fi - - # extract entry ids from the author/author_handle.json file - # - "$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..entry_id' - return 0 -} - - -# output_author_handle -# -# Write the author handles from an author/author_handle.json file to standard output (stdout) -# -# usage: -# output_author_handle author/author_handle.json -# -# returns: -# 0 ==> no errors detected, but output may be empty -# > 0 ==> function error number -# -function output_author_handle -{ - local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path - - # parse args - # - if [[ $# -ne 1 ]]; then - echo "$0: ERROR: in output_author_handle: expected 1 arg, found $#" 1>&2 - return 1 - fi - AUTHOR_HANDLE_JSON_PATH="$1" - if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_author_handle: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 2 - fi - if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_author_handle: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 3 - fi - if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_author_handle: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 4 - fi - - # extract entry ids from the author/author_handle.json file - # - "$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..author_handle' - return 0 -} - - -# output_full_name -# -# Write the Full Name from an author/author_handle.json file to standard output (stdout) -# -# usage: -# output_full_name author/author_handle.json -# -# returns: -# 0 ==> no errors detected, but output may be empty -# > 0 ==> function error number -# -function output_full_name -{ - local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path - - # parse args - # - if [[ $# -ne 1 ]]; then - echo "$0: ERROR: in output_full_name: expected 1 arg, found $#" 1>&2 - return 1 - fi - AUTHOR_HANDLE_JSON_PATH="$1" - if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_full_name: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 2 - fi - if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_full_name: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 3 - fi - if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_full_name: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 4 - fi - - # extract Full Name from the author/author_handle.json file - # - "$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..full_name' - return 0 -} - - -# output_location_code -# -# Write the Location Code from an author/author_handle.json file to standard output (stdout) -# -# usage: -# output_location_code author/author_handle.json -# -# returns: -# 0 ==> no errors detected, but output may be empty -# > 0 ==> function error number -# -function output_location_code -{ - local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path - - # parse args - # - if [[ $# -ne 1 ]]; then - echo "$0: ERROR: in output_location_code: expected 1 arg, found $#" 1>&2 - return 1 - fi - AUTHOR_HANDLE_JSON_PATH="$1" - if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_location_code: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 2 - fi - if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_location_code: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 3 - fi - if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_location_code: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 4 - fi - - # extract Location Code from the author/author_handle.json file - # - "$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..location_code' - return 0 -} - - # output_award # # Write the award name to standard output (stdout) @@ -383,6 +214,7 @@ function output_location_code function output_award { local ENTRY_JSON_PATH; # the .entry.json path + local PATTERN; # XPath for JSON pattern # parse args # @@ -406,283 +238,17 @@ function output_award # obtain the award string # - "$JVAL_WRAPPER" -w -b "$ENTRY_JSON_PATH" '$..award' - return 0 -} - - -# output_url -# -# Write the url from an author/author_handle.json file to standard output (stdout) -# -# NOTE: If the url for the author is null, nothing is written to standard output. -# -# usage: -# output_url author/author_handle.json -# -# returns: -# 0 ==> no errors detected, but output may be empty -# > 0 ==> function error number -# -function output_url -{ - local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path - - # parse args - # - if [[ $# -ne 1 ]]; then - echo "$0: ERROR: in output_url: expected 1 arg, found $#" 1>&2 - return 1 + PATTERN='$..award' + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN'" 1>&2 fi - AUTHOR_HANDLE_JSON_PATH="$1" - if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_url: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 2 + "$JVAL_WRAPPER" -b -q "$ENTRY_JSON_PATH" "$PATTERN" + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: in output_award: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN' failed," \ + "error code: $status" 1>&2 + return 5 fi - if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_url: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 3 - fi - if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_url: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 4 - fi - - # extract url from the author/author_handle.json file - # - # NOTE: We output nothing if the JSON member value is null. - # - "$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..url' | grep -F -v null - return 0 -} - - -# output_alt_url -# -# Write the alternate url from an author/author_handle.json file to standard output (stdout) -# -# NOTE: If the alternate url for the author is null, nothing is written to standard output. -# -# usage: -# output_alt_url author/author_handle.json -# -# returns: -# 0 ==> no errors detected, but output may be empty -# > 0 ==> function error number -# -function output_alt_url -{ - local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path - - # parse args - # - if [[ $# -ne 1 ]]; then - echo "$0: ERROR: in output_alt_url: expected 1 arg, found $#" 1>&2 - return 1 - fi - AUTHOR_HANDLE_JSON_PATH="$1" - if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_alt_url: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 2 - fi - if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_alt_url: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 3 - fi - if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_alt_url: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 4 - fi - - # extract alternate url from the author/author_handle.json file - # - # NOTE: We output nothing if the JSON member value is null. - # - "$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..alt_url' | grep -F -v null - return 0 -} - - -# output_mastodon -# -# Write the mastodon handle from an author/author_handle.json file to standard output (stdout) -# -# NOTE: If the mastodon handle for the author is null, nothing is written to standard output. -# -# usage: -# output_mastodon author/author_handle.json -# -# returns: -# 0 ==> no errors detected, but output may be empty -# > 0 ==> function error number -# -function output_mastodon -{ - local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path - - # parse args - # - if [[ $# -ne 1 ]]; then - echo "$0: ERROR: in output_mastodon: expected 1 arg, found $#" 1>&2 - return 1 - fi - AUTHOR_HANDLE_JSON_PATH="$1" - if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_mastodon: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 2 - fi - if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_mastodon: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 3 - fi - if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_mastodon: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 4 - fi - - # extract mastodon handle from the author/author_handle.json file - # - # NOTE: We output nothing if the JSON member value is null. - # - "$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..mastodon' | grep -F -v null - return 0 -} - - -# output_mastodon_url -# -# Write the mastodon handle from an author/author_handle.json file to standard output (stdout) -# -# NOTE: If the mastodon URL for the author is null, nothing is written to standard output. -# -# usage: -# output_mastodon_url author/author_handle.json -# -# returns: -# 0 ==> no errors detected, but output may be empty -# > 0 ==> function error number -# -function output_mastodon_url -{ - local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path - - # parse args - # - if [[ $# -ne 1 ]]; then - echo "$0: ERROR: in output_mastodon_url: expected 1 arg, found $#" 1>&2 - return 1 - fi - AUTHOR_HANDLE_JSON_PATH="$1" - if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_mastodon_url: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 2 - fi - if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_mastodon_url: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 3 - fi - if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_mastodon_url: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 4 - fi - - # extract mastodon URL from the author/author_handle.json file - # - # NOTE: We output nothing if the JSON member value is null. - # - "$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..mastodon_url' | grep -F -v null - return 0 -} - - -# output_github -# -# Write the GitHub handle from an author/author_handle.json file to standard output (stdout) -# -# NOTE: If the GitHub handle for the author is null, nothing is written to standard output. -# -# usage: -# output_github author/author_handle.json -# -# returns: -# 0 ==> no errors detected, but output may be empty -# > 0 ==> function error number -# -function output_github -{ - local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path - - # parse args - # - if [[ $# -ne 1 ]]; then - echo "$0: ERROR: in output_github: expected 1 arg, found $#" 1>&2 - return 1 - fi - AUTHOR_HANDLE_JSON_PATH="$1" - if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_github: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 2 - fi - if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_github: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 3 - fi - if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_github: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 4 - fi - - # extract GitHub handle from the author/author_handle.json file - # - # NOTE: We output nothing if the JSON member value is null. - # - "$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..github' | grep -F -v null - return 0 -} - - -# output_affiliation -# -# Write the affiliation from an author/author_handle.json file to standard output (stdout) -# -# NOTE: If the affiliation for the author is null, nothing is written to standard output. -# -# usage: -# output_affiliation author/author_handle.json -# -# returns: -# 0 ==> no errors detected, but output may be empty -# > 0 ==> function error number -# -function output_affiliation -{ - local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path - - # parse args - # - if [[ $# -ne 1 ]]; then - echo "$0: ERROR: in output_affiliation: expected 1 arg, found $#" 1>&2 - return 1 - fi - AUTHOR_HANDLE_JSON_PATH="$1" - if [[ ! -e $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_affiliation: author_handle.json does not exist: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 2 - fi - if [[ ! -f $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_affiliation: author_handle.json is not a file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 3 - fi - if [[ ! -r $AUTHOR_HANDLE_JSON_PATH ]]; then - echo "$0: ERROR: in output_affiliation: author_handle.json is not a readable file: $AUTHOR_HANDLE_JSON_PATH" 1>&2 - return 4 - fi - - # extract affiliation from the author/author_handle.json file - # - # NOTE: We output nothing if the JSON member value is null. - # - "$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..affiliation' | grep -F -v null return 0 } @@ -919,6 +485,40 @@ fi export AUTHORS_HTML="authors.html" +# verify that the bin/combine_author_handle.sh tool is executable +# +export COMBINE_AUTHOR="$BIN_DIR/combine_author_handle.sh" +if [[ ! -e $COMBINE_AUTHOR ]]; then + echo "$0: ERROR: bin/unicode-chk.sh does not exist: $COMBINE_AUTHOR" 1>&2 + exit 5 +fi +if [[ ! -f $COMBINE_AUTHOR ]]; then + echo "$0: ERROR: bin/unicode-chk.sh is not a regular file: $COMBINE_AUTHOR" 1>&2 + exit 5 +fi +if [[ ! -x $COMBINE_AUTHOR ]]; then + echo "$0: ERROR: bin/unicode-chk.sh is not an executable file: $COMBINE_AUTHOR" 1>&2 + exit 5 +fi + + +# verify that the bin/unicode-fix.sed tool is executable +# +export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed" +if [[ ! -e $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -f $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -r $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi + + # print running info if verbose # # If -v 3 or higher, print exported variables in order that they were exported. @@ -938,8 +538,8 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: SITE_URL=$SITE_URL" 1>&2 echo "$0: debug[3]: NOOP=$NOOP" 1>&2 echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2 - echo "$0: debug[3]: EXIT_CODE=$EXIT_CODE" 1>&2 echo "$0: debug[3]: QUICK_MODE=$QUICK_MODE" 1>&2 + echo "$0: debug[3]: EXIT_CODE=$EXIT_CODE" 1>&2 for index in "${!IS_FOR[@]}"; do echo "$0: debug[3]: IS_FOR[$index]=${IS_FOR[$index]}" 1>&2 done @@ -955,6 +555,8 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: JVAL_WRAPPER=$JVAL_WRAPPER" 1>&2 echo "$0: debug[3]: LOCATION_TOOL=$LOCATION_TOOL" 1>&2 echo "$0: debug[3]: AUTHORS_HTML=$AUTHORS_HTML" 1>&2 + echo "$0: debug[3]: COMBINE_AUTHOR=$COMBINE_AUTHOR" 1>&2 + echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2 fi @@ -1026,31 +628,32 @@ elif [[ $V_FLAG -ge 3 ]]; then fi -# load initial letter . lines into the temporary sort word list file -# -for letter in "${!IS_FOR[@]}"; do - echo "$letter ." -done > "$TMP_SORT_WORD" # add author sort_word filename to the temporary sort word list file # -if [[ $V_FLAG -ge 1 ]]; then - echo "$0: debug[1]: Forming temporary sort word list file: $TMP_SORT_WORD" +if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: Forming temporary sort word list file: $TMP_SORT_WORD" fi -find "$AUTHOR_DIR" -mindepth 1 -maxdepth 1 -type f -name '*.json' 2>/dev/null | while read -r json_file; do - echo "$(bin/jval-wrapper.sh -w -b "$json_file" '$..sort_word') $json_file" -done >> "$TMP_SORT_WORD" +export PATTERN='$..sort_word' +{ + # load initial letter . lines + # + for letter in "${!IS_FOR[@]}"; do + echo "$letter ." + done - -# sort the temporary sort word list file -# -LC_ALL=C sort -f -d "$TMP_SORT_WORD" -o "$TMP_SORT_WORD" -status="$?" -if [[ $status -ne 0 ]]; then - echo "$0: ERROR: LC_ALL=C sort -f -d $TMP_SORT_WORD -o $TMP_SORT_WORD failed, error: $status" 1>&2 - exit 14 -fi + # add sort_word filenames lines + # + "$COMBINE_AUTHOR" | "$JVAL_WRAPPER" -b -q -- - "$PATTERN" + status_codes=("${PIPESTATUS[@]}") + if [[ ${status_codes[*]} =~ [1-9] ]]; then + echo "$0: ERROR: $COMBINE_AUTHOR | $JVAL_WRAPPER -b -q -- - '$PATTERN' |" \ + "LC_ALL=C sort -f -d > $TMP_SORT_WORD failed," \ + "error codes: ${status_codes[*]}" 1>&2 + exit 14 + fi +} | LC_ALL=C sort -f -d > "$TMP_SORT_WORD" # generate the temporary entry markdown file @@ -1114,9 +717,7 @@ EOF echo '
' echo if [[ $V_FLAG -ge 1 ]]; then - echo 1>&2 echo "$0: debug[1]: Processing author_handles that begin with $l for: $AUTHORS_HTML" 1>&2 - echo 1>&2 fi # case: we have a "sort_word file" line @@ -1130,91 +731,145 @@ EOF exit 15 fi - FULL_NAME=$(output_full_name "$filename") - export FULL_NAME - if [[ -z $FULL_NAME ]]; then - echo "$0: ERROR: full name is empty in: $filename" 1>&2 + # parse an element value of the this particular author in the authors JSON array + # + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to run: $JVAL_WRAPPER -q -- - | sed ... | LC_ALL=C sort -d)" 1>&2 + fi + export AUTHOR_DATA + AUTHOR_DATA=$("$JVAL_WRAPPER" -q -- "$filename" | + sed -f "$UNICODE_FIX_SED" | + LC_ALL=C sort -d) + status_codes=("${PIPESTATUS[@]}") + if [[ ${status_codes[*]} =~ [1-9] ]]; then + echo "$0: ERROR: $JVAL_WRAPPER -q -- $filename | sed ... | LC_ALL=C sort -d failed," \ + "error codes: ${status_codes[*]}" 1>&2 exit 16 fi - - LOCATION_CODE=$(output_location_code "$filename") - export LOCATION_CODE - if [[ -z $LOCATION_CODE ]]; then - echo "$0: ERROR: location code is empty in: $filename" 1>&2 + if [[ -z $AUTHOR_DATA ]]; then + echo "$0: ERROR: found nothing for author in: $filename" 1>&2 exit 17 fi + export FULL_NAME= + export LOCATION_CODE= + export AUTHOR_URL= + export AUTHOR_ALT_URL= + export AUTHOR_MASTODON_HANDLE= + export AUTHOR_MASTODON_URL= + export AUTHOR_GITHUB= + export AUTHOR_AFFILIATION= + export AUTHOR_HANDLE= + unset ENTRY_ID_SET + declare -ag ENTRY_ID_SET + while read -r NAME VALUE; do + # parse name value + # + case "$NAME" in + full_name) + FULL_NAME="$VALUE" + ;; + location_code) + LOCATION_CODE="$VALUE" + ;; + url) + AUTHOR_URL="$VALUE" + ;; + alt_url) + AUTHOR_ALT_URL="$VALUE" + ;; + mastodon) + AUTHOR_MASTODON_HANDLE="$VALUE" + ;; + mastodon_url) + AUTHOR_MASTODON_URL="$VALUE" + ;; + github) + AUTHOR_GITHUB="$VALUE" + ;; + affiliation) + AUTHOR_AFFILIATION="$VALUE" + ;; + author_handle) + AUTHOR_HANDLE="$VALUE" + ;; + entry_id) + ENTRY_ID_SET+=("$VALUE") + ;; + *) # ignore all other fields + ;; + esac + done <<< "$AUTHOR_DATA" + + # validate fields values + # + if [[ -z $FULL_NAME ]]; then + echo "$0: ERROR: cannot find name in: $filename" 1>&2 + exit 18 + fi + if [[ -z $LOCATION_CODE ]]; then + echo "$0: ERROR: cannot find location_code in: $filename" 1>&2 + exit 19 + fi + if [[ -z $AUTHOR_URL ]]; then + echo "$0: ERROR: cannot find url in: $filename" 1>&2 + exit 20 + fi + if [[ -z $AUTHOR_ALT_URL ]]; then + echo "$0: ERROR: cannot find alt_url in: $filename" 1>&2 + exit 21 + fi + if [[ -z $AUTHOR_MASTODON_HANDLE ]]; then + echo "$0: ERROR: cannot find mastodon in: $filename" 1>&2 + exit 22 + fi + if [[ -z $AUTHOR_MASTODON_URL ]]; then + echo "$0: ERROR: cannot find mastodon_url in: $filename" 1>&2 + exit 23 + fi + if [[ -z $AUTHOR_GITHUB ]]; then + echo "$0: ERROR: cannot find github in: $filename" 1>&2 + exit 24 + fi + if [[ -z $AUTHOR_AFFILIATION ]]; then + echo "$0: ERROR: cannot find affiliation in: $filename" 1>&2 + exit 25 + fi + if [[ -z $AUTHOR_HANDLE ]]; then + echo "$0: ERROR: cannot find author_handle in: $filename" 1>&2 + exit 26 + fi + + # use the location tool to obtain the location_name + # LOCATION_NAME=$("$LOCATION_TOOL" "$LOCATION_CODE" 2>/dev/null) status="$?" export LOCATION_NAME if [[ $status -ne 0 ]]; then - echo "$0: ERROR: cannot determine location name" \ - "for location ISO 3166 code: $LOCATION_CODE" 1>&2 - exit 18 - fi - if [[ -z $LOCATION_NAME ]]; then - echo "$0: ERROR: location name" \ - "for location ISO 3166 code: $LOCATION_CODE is empty: $LOCATION_NAME" 1>&2 - exit 19 + echo "$0: ERROR: cannot determine location name for" \ + "location ISO 3166 code: $LOCATION_CODE" 1>&2 + exit 1 fi + # use the location tool to obtain the location common name + # LOCATION_COMMON_NAME=$("$LOCATION_TOOL" -c "$LOCATION_CODE" 2>/dev/null) status="$?" export LOCATION_COMMON_NAME if [[ $status -ne 0 ]]; then echo "$0: ERROR: cannot determine location common name" \ "for location ISO 3166 code: $LOCATION_CODE" 1>&2 - exit 20 - fi - if [[ -z $LOCATION_COMMON_NAME ]]; then - echo "$0: ERROR: location common name" \ - "for location ISO 3166 code: $LOCATION_CODE is empty: $LOCATION_COMMON_NAME" 1>&2 - exit 21 + exit 1 fi - AUTHOR_HANDLE=$(output_author_handle "$filename") - export AUTHOR_HANDLE - if [[ -z $AUTHOR_HANDLE ]]; then - echo "$0: ERROR: author handle is empty in: $filename" 1>&2 - exit 22 - fi - if [[ $V_FLAG -ge 1 ]]; then - echo "$0: debug[1]: Processing author_handle $AUTHOR_HANDLE for: $AUTHORS_HTML" 1>&2 - fi - - # NOTE: AUTHOR_AFFILIATION will be empty if there is no affiliation - AUTHOR_AFFILIATION=$(output_affiliation "$filename") - export AUTHOR_AFFILIATION - - # NOTE: AUTHOR_URL will be empty if there is no url - AUTHOR_URL=$(output_url "$filename") - export AUTHOR_URL - - # NOTE: AUTHOR_ALT_URL will be empty if there is no alternate url - AUTHOR_ALT_URL=$(output_alt_url "$filename") - export AUTHOR_ALT_URL - - # NOTE: AUTHOR_MASTODON_HANDLE will be empty if there is no mastodon handle - AUTHOR_MASTODON_HANDLE=$(output_mastodon "$filename") - export AUTHOR_MASTODON_HANDLE - - # NOTE: AUTHOR_MASTODON_URL will be empty if there is no mastodon URL - AUTHOR_MASTODON_URL=$(output_mastodon_url "$filename") - export AUTHOR_MASTODON_URL - if [[ -n $AUTHOR_MASTODON_HANDLE && -z $AUTHOR_MASTODON_URL ]]; then - echo "$0: ERROR: mastodon url is null but mastodon handle is not null in: $filename" 1>&2 - exit 23 - fi - if [[ -z $AUTHOR_MASTODON_HANDLE && -n $AUTHOR_MASTODON_URL ]]; then - echo "$0: ERROR: mastodon handle is null but mastodon url is not null in: $filename" 1>&2 - exit 24 - fi - - # NOTE: AUTHOR_GITHUB will be empty if there is no GitHub handle - AUTHOR_GITHUB=$(output_github "$filename") - export AUTHOR_GITHUB + # form github username + # AUTHOR_GITHUB_USERNAME=${AUTHOR_GITHUB#@} - export AUTHOR_GITHUB_USERNAME + export AUTHOR_GITHUB_USERNAME + if [[ -z $AUTHOR_GITHUB_USERNAME ]]; then + echo "$0: ERROR: cannot determine github username from: $AUTHOR_GITHUB" 1>&2 + exit 27 + fi # output author information # @@ -1228,23 +883,24 @@ EOF echo "Location: $LOCATION_CODE -" \ "_${LOCATION_COMMON_NAME}_" fi - if [[ -n $AUTHOR_AFFILIATION ]]; then + if [[ -n $AUTHOR_AFFILIATION && $AUTHOR_AFFILIATION != null ]]; then echo "
" echo "Affiliation: $AUTHOR_AFFILIATION" fi - if [[ -n $AUTHOR_URL ]]; then + if [[ -n $AUTHOR_URL && $AUTHOR_URL != null ]]; then echo "
" echo "URL: $AUTHOR_URL" fi - if [[ -n $AUTHOR_ALT_URL ]]; then + if [[ -n $AUTHOR_ALT_URL && $AUTHOR_ALT_URL != null ]]; then echo "
" echo "Alternate URL: $AUTHOR_ALT_URL" fi - if [[ -n $AUTHOR_MASTODON_HANDLE && -n $AUTHOR_MASTODON_URL ]]; then + if [[ -n $AUTHOR_MASTODON_HANDLE && $AUTHOR_MASTODON_HANDLE != null && + -n $AUTHOR_MASTODON_URL && $AUTHOR_MASTODON_URL != null ]]; then echo "
" echo "Mastodon: $AUTHOR_MASTODON_HANDLE" fi - if [[ -n $AUTHOR_GITHUB ]]; then + if [[ -n $AUTHOR_GITHUB && $AUTHOR_GITHUB != null ]]; then echo "
" echo "GitHub: $AUTHOR_GITHUB" fi @@ -1254,23 +910,23 @@ EOF # output YYYY/dir set made by this author # - output_entry_ids "$filename" | while read -r ENTRY_ID; do + for ENTRY_ID in "${ENTRY_ID_SET[@]}"; do # verify the entry id # if [[ $ENTRY_ID =~ ^[^_][^_]*_[^_][^_]*$ ]]; then if [[ $V_FLAG -ge 7 ]]; then - echo "$0: debug[7]: entry_id: $entry_id" 1>&2 + echo "$0: debug[7]: entry_id: $ENTRY_ID" 1>&2 fi else - echo "$0: ERROR: entry_id is not in YYYY_dir formmat: $entry_id" 1>&2 - exit 25 + echo "$0: ERROR: entry_id is not in YYYY_dir formmat: $ENTRY_ID" 1>&2 + exit 28 fi YYYY_DIR=$(echo "$ENTRY_ID" | tr _ /) export YYYY_DIR if [[ ! -d $YYYY_DIR ]]; then echo "$0: ERROR: YYYY/dir is not a directory" 1>&2 - exit 26 + exit 29 fi # verify that the .entry.json file is a non-empty readable file @@ -1279,19 +935,19 @@ EOF export ENTRY_JSON if [[ ! -e $ENTRY_JSON ]]; then echo "$0: ERROR: .entry.json does not exist: $ENTRY_JSON" 1>&2 - exit 27 + exit 30 fi if [[ ! -f $ENTRY_JSON ]]; then echo "$0: ERROR: .entry.json is not a file: $ENTRY_JSON" 1>&2 - exit 28 + exit 31 fi if [[ ! -r $ENTRY_JSON ]]; then echo "$0: ERROR: .entry.json is not a readable file: $ENTRY_JSON" 1>&2 - exit 29 + exit 32 fi if [[ ! -s $ENTRY_JSON ]]; then echo "$0: ERROR: .entry.json is not a non-empty readable file: $ENTRY_JSON" 1>&2 - exit 30 + exit 33 fi if [[ $V_FLAG -ge 7 ]]; then echo "$0: debug[7]: .entry.json: $ENTRY_JSON" 1>&2 @@ -1299,11 +955,11 @@ EOF # collect the award for this entry id # - AWARD=$(output_award "$ENTRY_JSON") + AWARD=$(output_award "$ENTRY_JSON" | sed -f "$UNICODE_FIX_SED") export AWARD if [[ -z $AWARD ]]; then echo "$0: ERROR: award not found in .entry.json: $ENTRY_JSON" 1>&2 - exit 31 + exit 34 fi # output the YYYY/dir entry for this author diff --git a/bin/gen-location.sh b/bin/gen-location.sh index 25a727262..e945bad7a 100755 --- a/bin/gen-location.sh +++ b/bin/gen-location.sh @@ -85,7 +85,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an # set variables referenced in the usage message # -export VERSION="1.6 2024-08-18" +export VERSION="1.7 2024-09-23" NAME=$(basename "$0") export NAME export V_FLAG=0 @@ -397,6 +397,23 @@ fi export LOCATION_HTML="location.html" +# verify that the bin/unicode-fix.sed tool is executable +# +export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed" +if [[ ! -e $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -f $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -r $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi + + # print running info if verbose # # If -v 3 or higher, print exported variables in order that they were exported. @@ -430,6 +447,7 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: JVAL_WRAPPER=$JVAL_WRAPPER" 1>&2 echo "$0: debug[3]: LOCATION_TOOL=$LOCATION_TOOL" 1>&2 echo "$0: debug[3]: LOCATION_HTML=$LOCATION_HTML" 1>&2 + echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2 fi # -N stops early before any processing is performed @@ -529,14 +547,18 @@ if [[ $V_FLAG -ge 1 ]]; then echo "$0: debug[1]: about to obtain location codes from author_handle JSON files" 1>&2 fi export AUTHOR_COUNT=0 -export AUTHOR_COUNT_CYCLE=20 +export AUTHOR_COUNT_CYCLE=10 +export AUTHOR_COUNT_COUNT +AUTHOR_COUNT_COUNT=$(find "$AUTHOR_DIR" -mindepth 1 -maxdepth 1 -type f -name '*.json' -print | wc -l) find "$AUTHOR_DIR" -mindepth 1 -maxdepth 1 -type f -name '*.json' -print | while read -r AUTHOR_HANDLE_JSON; do # collect location_code # - LOCATION_CODE=$("$JVAL_WRAPPER" -b -w "$AUTHOR_HANDLE_JSON" '$..location_code') - if [[ -z $LOCATION_CODE ]]; then + PATTERN='$..location_code' + LOCATION_CODE=$("$JVAL_WRAPPER" -b -q "$AUTHOR_HANDLE_JSON" "$PATTERN") + status="$?" + if [[ $status -ne 0 || -z $LOCATION_CODE ]]; then echo "$0: ERROR: cannot obtain location_code from: $AUTHOR_HANDLE_JSON" 1>&2 exit 1 fi @@ -551,8 +573,10 @@ find "$AUTHOR_DIR" -mindepth 1 -maxdepth 1 -type f -name '*.json' -print | # obtain the author full name # - FULL_NAME=$("$JVAL_WRAPPER" -b -w "$AUTHOR_HANDLE_JSON" '$..full_name') - if [[ -z $FULL_NAME ]]; then + PATTERN='$..full_name' + FULL_NAME=$("$JVAL_WRAPPER" -b -q "$AUTHOR_HANDLE_JSON" "$PATTERN" | sed -f "$UNICODE_FIX_SED") + status_codes=("${PIPESTATUS[@]}") + if [[ ${status_codes[*]} =~ [1-9] || -z $FULL_NAME ]]; then echo "$0: ERROR: cannot full name from: $AUTHOR_HANDLE" 1>&2 exit 1 fi @@ -573,7 +597,8 @@ find "$AUTHOR_DIR" -mindepth 1 -maxdepth 1 -type f -name '*.json' -print | # ((++AUTHOR_COUNT)) if [[ $V_FLAG -ge 1 && $((AUTHOR_COUNT % AUTHOR_COUNT_CYCLE)) -eq 0 ]]; then - echo "$0: debug[1]: cycle $AUTHOR_COUNT finished processing: $AUTHOR_HANDLE_JSON" 1>&2 + echo "$0: debug[1]: cycle $AUTHOR_COUNT out of $AUTHOR_COUNT_COUNT" \ + "finished processing: $AUTHOR_HANDLE_JSON" 1>&2 fi done | LC_ALL=C sort -d -u > "$TMP_LOC_WORD_HANDLE_NAME_SET" diff --git a/bin/gen-status.sh b/bin/gen-status.sh index b76a7166a..8c7ae32e9 100755 --- a/bin/gen-status.sh +++ b/bin/gen-status.sh @@ -86,7 +86,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an # set variables referenced in the usage message # -export VERSION="1.6 2024-08-17" +export VERSION="1.7 2024-09-23" NAME=$(basename "$0") export NAME export V_FLAG=0 @@ -108,12 +108,6 @@ export REPO_TOP_URL="https://github.com/ioccc-src/temp-test-ioccc" # GitHub puts individual files under the "blob/master" sub-directory. export REPO_URL="$REPO_TOP_URL/blob/master" export SITE_URL="https://ioccc-src.github.io/temp-test-ioccc" -JPARSE_TOOL=$(type -P jparse) -export JPARSE_TOOL -if [[ -z "$JPARSE_TOOL" ]]; then - echo "$0: FATAL: jparse tool is not installed or not in \$PATH" 1>&2 - exit 5 -fi # STAT_TOOL=$(type -P stat) if [[ -z "$STAT_TOOL" ]]; then @@ -294,11 +288,23 @@ function output_status_json # determine the "contest_status" that was in status.json # - OLD_CONTEST_STATUS=$("$JVAL_WRAPPER" -w -b "$STATUS_PATH" '$..contest_status') + PATTERN='$..contest_status' + OLD_CONTEST_STATUS=$("$JVAL_WRAPPER" -b -q "$STATUS_PATH" "$PATTERN") + status="$?" + if [[ $status -ne 0 || -z $OLD_CONTEST_STATUS ]]; then + echo "$0: ERROR: cannot determine old contest_status from status.json: $STATUS_JSON" 1>&2 + return 11 + fi # determine the "status_update" that was in status.json # - OLD_STATUS_UPDATE=$("$JVAL_WRAPPER" -w -b "$STATUS_PATH" '$..status_update') + PATTERN='$..status_update' + OLD_STATUS_UPDATE=$("$JVAL_WRAPPER" -b -q "$STATUS_PATH" "$PATTERN") + status="$?" + if [[ $status -ne 0 || -z $OLD_STATUS_UPDATE ]]; then + echo "$0: ERROR: cannot determine old status_update from status.json: $STATUS_JSON" 1>&2 + return 12 + fi # update the status.json file modification date if the "contest_status" changed # @@ -306,11 +312,11 @@ function output_status_json status="$?" if [[ $status -ne 0 ]]; then echo "$0: ERROR: in output_status_json: modification date of $STATUS_PATH failed, error: $status" 1>&2 - return 11 + return 13 fi if [[ -z $NEW_STATUS_UPDATE ]]; then echo "$0: ERROR: in output_status_json: modification date of $STATUS_PATH is empty" 1>&2 - return 12 + return 14 fi # only update the "status_update" If the "contest_status" changed @@ -788,7 +794,6 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: REPO_TOP_URL=$REPO_TOP_URL" 1>&2 echo "$0: debug[3]: REPO_URL=$REPO_URL" 1>&2 echo "$0: debug[3]: SITE_URL=$SITE_URL" 1>&2 - echo "$0: debug[3]: JPARSE_TOOL=$JPARSE_TOOL" 1>&2 echo "$0: debug[3]: IOCCC_STATUS_VERSION=$IOCCC_STATUS_VERSION" 1>&2 echo "$0: debug[3]: NOOP=$NOOP" 1>&2 echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2 @@ -819,23 +824,16 @@ if [[ $V_FLAG -ge 3 ]]; then fi -# validate JSON in status.json -# -"$JPARSE_TOOL" -q "$STATUS_JSON" -status="$?" -if [[ $status -ne 0 ]]; then - echo "$0: ERROR: status.json is not valid JSON, jparse error code: $status" 1>&2 - exit 1 -fi - # obtain contest_status if CONTEST_STATUS was not set in the arg # if [[ -z $CONTEST_STATUS ]]; then # obtain the contest_status from status.json # - CONTEST_STATUS=$("$JVAL_WRAPPER" -w -b "$STATUS_JSON" '$..contest_status') - if [[ -z $CONTEST_STATUS ]]; then + PATTERN='$..contest_status' + CONTEST_STATUS=$("$JVAL_WRAPPER" -b -q "$STATUS_JSON" "$PATTERN") + status="$?" + if [[ $status -ne 0 || -z $CONTEST_STATUS ]]; then echo "$0: ERROR: cannot determine contest_status from status.json: $STATUS_JSON" 1>&2 exit 1 fi diff --git a/bin/gen-years.sh b/bin/gen-years.sh index 6a15a5ab1..906307f6e 100755 --- a/bin/gen-years.sh +++ b/bin/gen-years.sh @@ -88,7 +88,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an # set variables referenced in the usage message # -export VERSION="1.4 2024-08-17" +export VERSION="1.5 2024-09-23" NAME=$(basename "$0") export NAME export V_FLAG=0 @@ -213,8 +213,10 @@ function output_award # obtain the award string # - AWARD_STRING=$("$JVAL_WRAPPER" -w -b "$ENTRY_JSON_PATH" '$..award') - if [[ -z $AWARD_STRING ]]; then + PATTERN='$..award' + AWARD_STRING=$("$JVAL_WRAPPER" -b -q "$ENTRY_JSON_PATH" "$PATTERN") + status="$?" + if [[ $status -ne 0 || -z $AWARD_STRING ]]; then echo "$0: ERROR: in output_award: no award found in .entry.json file: $ENTRY_JSON_PATH" 1>&2 return 5 fi @@ -550,6 +552,23 @@ fi export YEARS_HTML="years.html" +# verify that the bin/unicode-fix.sed tool is executable +# +export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed" +if [[ ! -e $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -f $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -r $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi + + # print running info if verbose # # If -v 3 or higher, print exported variables in order that they were exported. @@ -582,6 +601,7 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: JVAL_WRAPPER=$JVAL_WRAPPER" 1>&2 echo "$0: debug[3]: TOP_FILE=$TOP_FILE" 1>&2 echo "$0: debug[3]: YEARS_HTML=$YEARS_HTML" 1>&2 + echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2 fi @@ -848,8 +868,9 @@ for YYYY in $("$TAC_TOOL" "$TOP_FILE"); do # determine the award for this entry # - AWARD=$(output_award "$ENTRY_JSON") - if [[ -z $AWARD ]]; then + AWARD=$(output_award "$ENTRY_JSON" | sed -f "$UNICODE_FIX_SED") + status_codes=("${PIPESTATUS[@]}") + if [[ ${status_codes[*]} =~ [1-9] || -z $AWARD ]]; then echo "$0: ERROR: cannot find award in .entry.json: $ENTRY_JSON" 1>&2 EXIT_CODE="7" # exit 7 echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 diff --git a/bin/jfmt-wrapper.sh b/bin/jfmt-wrapper.sh index 4b9af4b0f..9dd44a71b 100755 --- a/bin/jfmt-wrapper.sh +++ b/bin/jfmt-wrapper.sh @@ -2,15 +2,30 @@ # # jfmt-wrapper.sh - canonically format a JSON file # -# We form a wrapper for the jfmt(1), a tool that will format a JSON -# file into a canonical style. NOTE: As of 2024 July 28 the jfmt(1) -# tool has not been written, so jfmt-wrapper.sh uses the JSONPath.sh(1) -# tool from the recently forked and modified JSONPath.sh tool: +# If the JSON file is already on canonical form, the file is not modified. +# +# We form a wrapper for the jprint(1), a tool that will format a JSON +# file into a canonical style. +# +# NOTE: As of 2024 Sep 09 the jprint(1) tool has not been written, +# so jfmt-wrapper.sh uses with the jsp tool from: +# +# https://github.com/kjozsa/jsp +# +# FYI: For macOS, use homebrew to install pipx: +# +# brew install pipx +# +# Next, use pipx to install jsp: +# +# pipx --global install jsp +# +# and then be sure that ~/.local/bin is in your $PATH. +# +# or the JSONPath.sh(1) tool from: # # https://github.com/lcn2/JSONPath.sh # -# If the JSON file is already on canonical form, the file is not modified. -# # Copyright (c) 2024 by Landon Curt Noll. All Rights Reserved. # # Permission to use, copy, modify, and distribute this software and @@ -95,43 +110,50 @@ shopt -s globstar # enable ** to match all files and zero or more directories an # set variables referenced in the usage message # -export VERSION="1.1 2024-08-20" +export VERSION="1.2 2024-09-23" NAME=$(basename "$0") export NAME export V_FLAG=0 -# -JPARSE_TOOL=$(type -P jparse) -export JPARSE_TOOL - - -# Until we have a true jfmt tool, we will use the modified JSONPath.sh -# -# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX -# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX -# XXX - until we have the jfmt command, we must FAKE PARSE IOCCC JSON files - XXX -# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX -# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX -# -export JSONPATH_REPO="https://github.com/lcn2/JSONPath.sh" -JSONPATH_SH=$(type -P JSONPath.sh) -export JSONPATH_SH -if [[ -z $JSONPATH_SH ]]; then - echo "$0: FATAL: JSONPath.sh tool is not installed or not in \$PATH" 1>&2 - echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2 +GIT_TOOL=$(type -P git) +export GIT_TOOL +if [[ -z "$GIT_TOOL" ]]; then + echo "$0: FATAL: git tool is not installed or not in \$PATH" 1>&2 exit 5 fi -# verify JSONPath.sh supports -S -A +"$GIT_TOOL" rev-parse --is-inside-work-tree >/dev/null 2>&1 +status="$?" +if [[ $status -eq 0 ]]; then + TOPDIR=$("$GIT_TOOL" rev-parse --show-toplevel) +fi +export TOPDIR +export REPO_TOP_URL="https://github.com/ioccc-src/temp-test-ioccc" +# GitHub puts individual files under the "blob/master" sub-directory. +export REPO_URL="$REPO_TOP_URL/blob/master" + + +# Until we have a true jfmt tool, we will use wither jsp or the modified JSONPath.sh +# +# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX +# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX +# XXX - until we have the jprint command, we must use external tools to parse JSON - XXX +# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX +# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX +# +export JSP_REPO="https://github.com/kjozsa/jsp" +export JSONPATH_REPO="https://github.com/lcn2/JSONPath.sh" export FIZZBIN_JSON='"fizzbin"' -if ! "$JSONPATH_SH" -S -A -p >/dev/null 2>&1; then - echo "$0: FATAL: JSONPath.sh tool does not support -S -A: $FIZZBIN_JSON" 1>&2 - echo "$0: notice: we recommend you obtain and install JSONPath.sh from: $JSONPATH_REPO" 1>&2 - exit 5 -fi <<< "$FIZZBIN_JSON" +export JSP_TOOL="" +JSP_TOOL=$(type -P jsp) +export JSONPATH_SH="" +JSONPATH_SH=$(type -P JSONPath.sh) +# set the defaiult XPath for JSON +export XPATHJSON_USE="jsp" +export QUICK_CHECK="" # set usage message # -export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] [-j JSONPath.sh] file.json +export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] [-t tool] [-Q] file.json -h print help message and exit -v level set verbosity level (def level: 0) @@ -140,7 +162,8 @@ export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] [-j JSONPath.sh] file.jso -n do not modify file.json (def: modify file.json is not canonical) -N do not process file, just parse arguments and ignore the file (def: process the file) - -j JSONPath.sh path to the JSONPath.sh tool (not the wrapper) (def: $JSONPATH_SH) + -t tool tool to use: jsp or JSONPath.sh (def: try jsp, otherwise try JSONPath.sh) + -Q quick check tool using trivial input (def: do not) file.json update file.json in canonical form @@ -148,11 +171,11 @@ NOTE: file.json is not touched if it is already in canonical form Exit codes: 0 all OK - 1 JSONPath.sh exited non-zero + 1 some internal tool exited non-zero 2 -h and help string printed or -V and version string printed 3 command line error 4 bash version is too old - 5 internal tool not found + 5 some internal tool not found 6 file.json does not exist or is not readable 7 file.json is not valid JSON >= 10 internal error @@ -168,7 +191,7 @@ export DO_NOT_PROCESS= # parse command line # -while getopts :hv:VnNj: flag; do +while getopts :hv:VnNt:Q flag; do case "$flag" in h) echo "$USAGE" 1>&2 exit 2 @@ -182,8 +205,19 @@ while getopts :hv:VnNj: flag; do ;; N) DO_NOT_PROCESS="-N" ;; - j) JSONPATH_SH="$OPTARG" - ;; + t) # validate -t tool + case "$OPTARG" in + jsp) XPATHJSON_USE="jsp" ;; + JSONPath.sh) XPATHJSON_USE="JSONPath.sh" ;; + *) echo "$0: ERROR: unknown -t option: $OPTARG" 1>&2 + echo 1>&2 + echo "$USAGE" 1>&2 + exit 3 + ;; + esac + ;; + Q) QUICK_CHECK="true" + ;; \?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2 echo 1>&2 echo "$USAGE" 1>&2 @@ -237,6 +271,155 @@ if [[ ! -r $JSON_FILE ]]; then fi +# verify that we have a topdir directory +# +REPO_NAME=$(basename "$REPO_TOP_URL") +export REPO_NAME +if [[ -z $TOPDIR ]]; then + echo "$0: ERROR: cannot find top of git repo directory" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi +if [[ ! -e $TOPDIR ]]; then + echo "$0: ERROR: TOPDIR does not exist: $TOPDIR" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi +if [[ ! -d $TOPDIR ]]; then + echo "$0: ERROR: TOPDIR is not a directory: $TOPDIR" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi + + +# cd to topdir +# +if [[ ! -e $TOPDIR ]]; then + echo "$0: ERROR: cannot cd to non-existent path: $TOPDIR" 1>&2 + exit 6 +fi +if [[ ! -d $TOPDIR ]]; then + echo "$0: ERROR: cannot cd to a non-directory: $TOPDIR" 1>&2 + exit 6 +fi +export CD_FAILED +if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to: cd $TOPDIR" 1>&2 +fi +cd "$TOPDIR" || CD_FAILED="true" +if [[ -n $CD_FAILED ]]; then + echo "$0: ERROR: cd $TOPDIR failed" 1>&2 + exit 6 +fi +if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: now in directory: $(/bin/pwd)" 1>&2 +fi + + +# verify that we have a bin subdirectory +# +export BIN_PATH="$TOPDIR/bin" +if [[ ! -d $BIN_PATH ]]; then + echo "$0: ERROR: bin is not a directory under topdir: $BIN_PATH" 1>&2 + exit 6 +fi +export BIN_DIR="bin" + + +# verify that the bin/unicode-fix.sed tool is executable +# +export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed" +if [[ ! -e $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -f $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -r $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi + + +# validate the XPath for JSON tool +# +case "$XPATHJSON_USE" in + + # case: we will use JSONPath.sh + # + JSONPath.sh) + + # try JSONPath.sh + # + if [[ -z $JSONPATH_SH || ! -x $JSONPATH_SH ]]; then + echo "$0: ERROR: JSONPath.sh tool is not installed, is not executable, or not in \$PATH" 1>&2 + echo "$0: notice: obtain jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + fi + + # case: -Q - perform quick check + # + if [[ -n $QUICK_CHECK ]]; then + # verify JSONPath.sh supports -S -A -p + if ! "$JSONPATH_SH" -S -A -p >/dev/null 2>&1; then + echo "$0: ERROR: JSONPath.sh tool does not support -S -A: $FIZZBIN_JSON" 1>&2 + echo "$0: notice: we recommend you obtain and install jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: otherwise install JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + fi <<< "$FIZZBIN_JSON" + fi + ;; + + # case: we will use jsp + # + jsp) + + # try jsp + # + if [[ -z $JSP_TOOL || ! -x $JSP_TOOL ]]; then + echo "$0: ERROR: jsp.sh tool is not installed, is not executable, or not in \$PATH" 1>&2 + echo "$0: notice: obtain jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + fi + + # case: -Q - perform quick check + # + if [[ -n $QUICK_CHECK ]]; then + # verify jsp + if ! "$JSP_TOOL" --indent 4 --format --no-color >/dev/null 2>&1; then + echo "$0: ERROR: jsp tool does not support --indent 4 --format --no-color: $FIZZBIN_JSON" 1>&2 + echo "$0: notice: you might need to update and reinstall jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: or you might try to obtain and install JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + fi <<< "$FIZZBIN_JSON" + fi + ;; + + # case: we do not have an XPath for JSON tool + # + *) + echo "$0: ERROR: cannot find an XPath JSON tool to use" 1>&2 + echo "$0: notice: we recommend you obtain and install jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: otherwise install JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + ;; +esac + + # print running info if verbose # # If -v 3 or higher, print exported variables in order that they were exported. @@ -245,27 +428,25 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: VERSION=$VERSION" 1>&2 echo "$0: debug[3]: NAME=$NAME" 1>&2 echo "$0: debug[3]: V_FLAG=$V_FLAG" 1>&2 + echo "$0: debug[3]: GIT_TOOL=$GIT_TOOL" 1>&2 + echo "$0: debug[3]: TOPDIR=$TOPDIR" 1>&2 + echo "$0: debug[3]: REPO_TOP_URL=$REPO_TOP_URL" 1>&2 + echo "$0: debug[3]: REPO_URL=$REPO_URL" 1>&2 + echo "$0: debug[3]: JSP_REPO=$JSP_REPO" 1>&2 echo "$0: debug[3]: JSONPATH_REPO=$JSONPATH_REPO" 1>&2 - echo "$0: debug[3]: JSONPATH_SH=$JSONPATH_SH" 1>&2 echo "$0: debug[3]: FIZZBIN_JSON=$FIZZBIN_JSON" 1>&2 + echo "$0: debug[3]: JSP_TOOL=$JSP_TOOL" 1>&2 + echo "$0: debug[3]: JSONPATH_SH=$JSONPATH_SH" 1>&2 + echo "$0: debug[3]: XPATHJSON_USE=$XPATHJSON_USE" 1>&2 + echo "$0: debug[3]: QUICK_CHECK=$QUICK_CHECK" 1>&2 echo "$0: debug[3]: NOOP=$NOOP" 1>&2 echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2 echo "$0: debug[3]: JSON_FILE=$JSON_FILE" 1>&2 -fi - - -# verify the JSON of JSON_FILE -# -if [[ $V_FLAG -ge 7 ]]; then - echo "$0: debug[7]: about to run: $JPARSE_TOOL -q -- $JSON_FILE" 1>&2 -fi -if "$JPARSE_TOOL" -q -- "$JSON_FILE"; then - if [[ $V_FLAG -ge 5 ]]; then - echo "$0: debug[5]: valid JSON for: $JSON_FILE" 1>&2 - fi -else - echo "$0: ERROR: not valid JSON: $JSON_FILE" 1>&2 - exit 7 + echo "$0: debug[3]: REPO_NAME=$REPO_NAME" 1>&2 + echo "$0: debug[3]: CD_FAILED=$CD_FAILED" 1>&2 + echo "$0: debug[3]: BIN_PATH=$BIN_PATH" 1>&2 + echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2 + echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2 fi @@ -304,16 +485,49 @@ fi # canonically format the JSON file # -if [[ $V_FLAG -ge 3 ]]; then - echo "$0: debug[3]: about to: $JSONPATH_SH -S -A -j -f $JSON_FILE > $TMP_JSON_FILE" 1>&2 -fi -"$JSONPATH_SH" -S -A -j -f "$JSON_FILE" > "$TMP_JSON_FILE" -status="$?" -if [[ $status -ne 0 ]]; then - echo "$0: ERROR: $JSONPATH_SH -S -A -j -f $JSON_FILE > $TMP_JSON_FILE failed," \ - "error code: $status" 1>&2 - exit 1 -fi +case "$XPATHJSON_USE" in + + # case: use jsp + # + jsp) + + if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: about to: $JSP_TOOL --indent 4 --format --no-color < $JSON_FILE | sed ..." 1>&2 + fi + "$JSP_TOOL" --indent 4 --format --no-color < "$JSON_FILE" | + sed -f "$UNICODE_FIX_SED" -e 's/\(\S\): /\1 : /' > "$TMP_JSON_FILE" + status_codes=("${PIPESTATUS[@]}") + if [[ ${status_codes[*]} =~ [1-9] ]]; then + echo "$0: ERROR: $JSP_TOOL --indent 4 --format --no-color < $JSON_FILE | sed ... failed", \ + "error codes: ${status_codes[*]}" 1>&2 + exit 1 + fi + ;; + + # case: use JSONPath.sh + # + JSONPath.sh) + + if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: about to: $JSONPATH_SH -S -A -j -f $JSON_FILE > $TMP_JSON_FILE" 1>&2 + fi + "$JSONPATH_SH" -S -A -j -f "$JSON_FILE" > "$TMP_JSON_FILE" + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: $JSONPATH_SH -S -A -j -f $JSON_FILE > $TMP_JSON_FILE failed," \ + "error code: $status" 1>&2 + exit 1 + fi + ;; + + # case: we do not know which XPath JSON tool to use + # + *) + + echo "$0: ERROR: XPATHJSON_USE is neither jsp nor JSONPath.sh: $XPATHJSON_USE" 1>&2 + exit 12 + ;; +esac # replace JSON_FILE file with canonically formatted TMP_JSON_FILE if different @@ -344,13 +558,13 @@ if [[ -z $NOOP ]]; then if [[ $status -ne 0 ]]; then echo "$0: ERROR: mv -f -- $TMP_JSON_FILE $JSON_FILE filed," \ "error code: $status" 1>&2 - exit 12 + exit 13 elif [[ $V_FLAG -ge 1 ]]; then echo "$0: debug[1]: replaced JSON_FILE file: $JSON_FILE" 1>&2 fi if [[ ! -s $JSON_FILE ]]; then echo "$0: ERROR: not a non-empty canonical JSON_FILE file: $JSON_FILE" 1>&2 - exit 13 + exit 14 fi fi diff --git a/bin/jprint-wrapper.sh b/bin/jprint-wrapper.sh index b67d7bd0b..c3d2514de 100755 --- a/bin/jprint-wrapper.sh +++ b/bin/jprint-wrapper.sh @@ -2,9 +2,27 @@ # # jprint-wrapper.sh - canonically format JSON to stdout # -# We form a wrapper for the jprint(1), a tool that will format JSON in a canonical style to stdout. -# NOTE: jprint(1) is a planned tool based on jfmt(1). As of 2024 July 28 the jfmt(1) tool has not been written, -# so jprint-wrapper.sh uses the JSONPath.sh(1) tool from the recently forked and modified JSONPath.sh tool: +# Print JSON in canonical format to standard output. +# +# We form a wrapper for the jprint(1), a tool that will format a JSON +# file into a canonical style. +# +# NOTE: As of 2024 Sep 09 the jprint(1) tool has not been written, +# so jprint-wrapper.sh uses with the jsp tool from: +# +# https://github.com/kjozsa/jsp +# +# FYI: For macOS, use homebrew to install pipx: +# +# brew install pipx +# +# Next, use pipx to install jsp: +# +# pipx --global install jsp +# +# and then be sure that ~/.local/bin is in your $PATH. +# +# or the JSONPath.sh(1) tool from: # # https://github.com/lcn2/JSONPath.sh # @@ -92,43 +110,50 @@ shopt -s globstar # enable ** to match all files and zero or more directories an # set variables referenced in the usage message # -export VERSION="1.1 2024-08-20" +export VERSION="1.3 2024-09-23" NAME=$(basename "$0") export NAME export V_FLAG=0 - - -# Until we have a true jprint tool, we will use the modified JSONPath.sh -# -# BTW: A jprint tool might be a special case of a jfmt tool, with a command line -# optimized for just displaying JSON in a nicely formatted canonical way. -# -# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX -# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX -# XXX - until we have the jprint command, we must FAKE PARSE IOCCC JSON files - XXX -# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX -# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX -# -export JSONPATH_REPO="https://github.com/lcn2/JSONPath.sh" -JSONPATH_SH=$(type -P JSONPath.sh) -export JSONPATH_SH -if [[ -z $JSONPATH_SH ]]; then - echo "$0: FATAL: JSONPath.sh tool is not installed or not in \$PATH" 1>&2 - echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2 +GIT_TOOL=$(type -P git) +export GIT_TOOL +if [[ -z "$GIT_TOOL" ]]; then + echo "$0: FATAL: git tool is not installed or not in \$PATH" 1>&2 exit 5 fi -# verify JSONPath.sh supports -S -A +"$GIT_TOOL" rev-parse --is-inside-work-tree >/dev/null 2>&1 +status="$?" +if [[ $status -eq 0 ]]; then + TOPDIR=$("$GIT_TOOL" rev-parse --show-toplevel) +fi +export TOPDIR +export REPO_TOP_URL="https://github.com/ioccc-src/temp-test-ioccc" +# GitHub puts individual files under the "blob/master" sub-directory. +export REPO_URL="$REPO_TOP_URL/blob/master" + + +# Until we have a true jfmt tool, we will use wither jsp or the modified JSONPath.sh +# +# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX +# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX +# XXX - until we have the jprint command, we must use external tools to parse JSON - XXX +# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX +# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX +# +export JSP_REPO="https://github.com/kjozsa/jsp" +export JSONPATH_REPO="https://github.com/lcn2/JSONPath.sh" export FIZZBIN_JSON='"fizzbin"' -if ! "$JSONPATH_SH" -S -A -p >/dev/null 2>&1; then - echo "$0: FATAL: JSONPath.sh tool does not support -S -A -T: $FIZZBIN_JSON" 1>&2 - echo "$0: notice: we recommend you obtain and install JSONPath.sh from: $JSONPATH_REPO" 1>&2 - exit 5 -fi <<< "$FIZZBIN_JSON" +export JSP_TOOL="" +JSP_TOOL=$(type -P jsp) +export JSONPATH_SH="" +JSONPATH_SH=$(type -P JSONPath.sh) +# set the defaiult XPath for JSON +export XPATHJSON_USE="jsp" +export QUICK_CHECK="" # set usage message # -export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] [-j JSONPath.sh] [file.json] +export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] [-t tool] [-Q] [file.json] -h print help message and exit -v level set verbosity level (def level: 0) @@ -137,17 +162,18 @@ export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] [-j JSONPath.sh] [file.js -n do nothing (same as -N) (def: do the action) -N do not process file, just parse arguments and ignore the file (def: process the file) - -j JSONPath.sh path to the JSONPath.sh tool (not the wrapper) (def: $JSONPATH_SH) + -t tool tool to use: jsp or JSONPath.sh (def: try jsp, otherwise try JSONPath.sh) + -Q quick check tool using trivial input (def: do not) file.json The JSON file to write, in canonical form, to stdout (def: read stdin) Exit codes: 0 all OK - 1 JSONPath.sh exited non-zero + 1 some internal tool exited non-zero 2 -h and help string printed or -V and version string printed 3 command line error 4 bash version is too old - 5 internal tool not found + 5 some internal tool not found 6 file.json does not exist or is not readable >= 10 internal error @@ -162,7 +188,7 @@ export DO_NOT_PROCESS= # parse command line # -while getopts :hv:VnNj: flag; do +while getopts :hv:VnNt:Q flag; do case "$flag" in h) echo "$USAGE" 1>&2 exit 2 @@ -176,8 +202,19 @@ while getopts :hv:VnNj: flag; do ;; N) DO_NOT_PROCESS="-N" ;; - j) JSONPATH_SH="$OPTARG" - ;; + t) # validate -t tool + case "$OPTARG" in + jsp) XPATHJSON_USE="jsp" ;; + JSONPath.sh) XPATHJSON_USE="JSONPath.sh" ;; + *) echo "$0: ERROR: unknown -t option: $OPTARG" 1>&2 + echo 1>&2 + echo "$USAGE" 1>&2 + exit 3 + ;; + esac + ;; + Q) QUICK_CHECK="true" + ;; \?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2 echo 1>&2 echo "$USAGE" 1>&2 @@ -234,6 +271,155 @@ if [[ -n $JSON_FILE ]]; then fi +# verify that we have a topdir directory +# +REPO_NAME=$(basename "$REPO_TOP_URL") +export REPO_NAME +if [[ -z $TOPDIR ]]; then + echo "$0: ERROR: cannot find top of git repo directory" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi +if [[ ! -e $TOPDIR ]]; then + echo "$0: ERROR: TOPDIR does not exist: $TOPDIR" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi +if [[ ! -d $TOPDIR ]]; then + echo "$0: ERROR: TOPDIR is not a directory: $TOPDIR" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi + + +# cd to topdir +# +if [[ ! -e $TOPDIR ]]; then + echo "$0: ERROR: cannot cd to non-existent path: $TOPDIR" 1>&2 + exit 6 +fi +if [[ ! -d $TOPDIR ]]; then + echo "$0: ERROR: cannot cd to a non-directory: $TOPDIR" 1>&2 + exit 6 +fi +export CD_FAILED +if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to: cd $TOPDIR" 1>&2 +fi +cd "$TOPDIR" || CD_FAILED="true" +if [[ -n $CD_FAILED ]]; then + echo "$0: ERROR: cd $TOPDIR failed" 1>&2 + exit 6 +fi +if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: now in directory: $(/bin/pwd)" 1>&2 +fi + + +# verify that we have a bin subdirectory +# +export BIN_PATH="$TOPDIR/bin" +if [[ ! -d $BIN_PATH ]]; then + echo "$0: ERROR: bin is not a directory under topdir: $BIN_PATH" 1>&2 + exit 6 +fi +export BIN_DIR="bin" + + +# verify that the bin/unicode-fix.sed tool is executable +# +export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed" +if [[ ! -e $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -f $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -r $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi + + +# validate the XPath for JSON tool +# +case "$XPATHJSON_USE" in + + # case: we will use JSONPath.sh + # + JSONPath.sh) + + # try JSONPath.sh + # + if [[ -z $JSONPATH_SH || ! -x $JSONPATH_SH ]]; then + echo "$0: ERROR: JSONPath.sh tool is not installed, is not executable, or not in \$PATH" 1>&2 + echo "$0: notice: obtain jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + fi + + # case: -Q - perform quick check + # + if [[ -n $QUICK_CHECK ]]; then + # verify JSONPath.sh supports -S -A -p + if ! "$JSONPATH_SH" -S -A -p >/dev/null 2>&1; then + echo "$0: ERROR: JSONPath.sh tool does not support -S -A: $FIZZBIN_JSON" 1>&2 + echo "$0: notice: we recommend you obtain and install jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: otherwise install JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + fi <<< "$FIZZBIN_JSON" + fi + ;; + + # case: we will use jsp + # + jsp) + + # try jsp + # + if [[ -z $JSP_TOOL || ! -x $JSP_TOOL ]]; then + echo "$0: ERROR: jsp.sh tool is not installed, is not executable, or not in \$PATH" 1>&2 + echo "$0: notice: obtain jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + fi + + # case: -Q - perform quick check + # + if [[ -n $QUICK_CHECK ]]; then + # verify jsp + if ! "$JSP_TOOL" --indent 4 --format --no-color >/dev/null 2>&1; then + echo "$0: ERROR: jsp tool does not support --indent 4 --format --no-color: $FIZZBIN_JSON" 1>&2 + echo "$0: notice: you might need to update and reinstall jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: or you might try to obtain and install JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + fi <<< "$FIZZBIN_JSON" + fi + ;; + + # case: we do not have an XPath for JSON tool + # + *) + echo "$0: ERROR: cannot find an XPath JSON tool to use" 1>&2 + echo "$0: notice: we recommend you obtain and install jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: otherwise install JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + ;; +esac + + # print running info if verbose # # If -v 3 or higher, print exported variables in order that they were exported. @@ -242,12 +428,25 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: VERSION=$VERSION" 1>&2 echo "$0: debug[3]: NAME=$NAME" 1>&2 echo "$0: debug[3]: V_FLAG=$V_FLAG" 1>&2 + echo "$0: debug[3]: GIT_TOOL=$GIT_TOOL" 1>&2 + echo "$0: debug[3]: TOPDIR=$TOPDIR" 1>&2 + echo "$0: debug[3]: REPO_TOP_URL=$REPO_TOP_URL" 1>&2 + echo "$0: debug[3]: REPO_URL=$REPO_URL" 1>&2 + echo "$0: debug[3]: JSP_REPO=$JSP_REPO" 1>&2 echo "$0: debug[3]: JSONPATH_REPO=$JSONPATH_REPO" 1>&2 - echo "$0: debug[3]: JSONPATH_SH=$JSONPATH_SH" 1>&2 echo "$0: debug[3]: FIZZBIN_JSON=$FIZZBIN_JSON" 1>&2 + echo "$0: debug[3]: JSP_TOOL=$JSP_TOOL" 1>&2 + echo "$0: debug[3]: JSONPATH_SH=$JSONPATH_SH" 1>&2 + echo "$0: debug[3]: XPATHJSON_USE=$XPATHJSON_USE" 1>&2 + echo "$0: debug[3]: QUICK_CHECK=$QUICK_CHECK" 1>&2 echo "$0: debug[3]: NOOP=$NOOP" 1>&2 echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2 echo "$0: debug[3]: JSON_FILE=$JSON_FILE" 1>&2 + echo "$0: debug[3]: REPO_NAME=$REPO_NAME" 1>&2 + echo "$0: debug[3]: CD_FAILED=$CD_FAILED" 1>&2 + echo "$0: debug[3]: BIN_PATH=$BIN_PATH" 1>&2 + echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2 + echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2 fi @@ -271,38 +470,88 @@ if [[ -n $NOOP ]]; then fi -# case: processing JSON from standard input +# print the JSON file in canonical format # -if [[ -z $JSON_FILE ]]; then +case "$XPATHJSON_USE" in - if [[ $V_FLAG -ge 1 ]]; then - echo "$0: debug[1]: about to: $JSONPATH_SH -S -A -j" 1>&2 - fi - "$JSONPATH_SH" -S -A -j - status="$?" + # case: use jsp + # + jsp) - if [[ $status -ne 0 ]]; then - echo "$0: ERROR: $JSONPATH_SH -S -A -j failed," \ - "error code: $status" 1>&2 - exit 1 - fi + # case: processing JSON from stdin via jsp + # + if [[ -z $JSON_FILE ]]; then + if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: about to: $JSP_TOOL --indent 4 --format --no-color | sed ..." 1>&2 + fi + "$JSP_TOOL" --indent 4 --format --no-color | + sed -f "$UNICODE_FIX_SED" -e 's/\(\S\): /\1 : /' + status_codes=("${PIPESTATUS[@]}") + if [[ ${status_codes[*]} =~ [1-9] ]]; then + echo "$0: ERROR: $JSP_TOOL --indent 4 --format --no-color | sed ... failed", \ + "error codes: ${status_codes[*]}" 1>&2 + exit 1 + fi -# case: processing JSON from a file -# -else + # processing JSON from a file via jsp + # + else + if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: about to: $JSP_TOOL --indent 4 --format --no-color < $JSON_FILE | sed ..." 1>&2 + fi + "$JSP_TOOL" --indent 4 --format --no-color < "$JSON_FILE" | + sed -f "$UNICODE_FIX_SED" -e 's/\(\S\): /\1 : /' + status_codes=("${PIPESTATUS[@]}") + if [[ ${status_codes[*]} =~ [1-9] ]]; then + echo "$0: ERROR: $JSP_TOOL --indent 4 --format --no-color < $JSON_FILE | sed ... failed", \ + "error codes: ${status_codes[*]}" 1>&2 + exit 1 + fi + fi + ;; - if [[ $V_FLAG -ge 1 ]]; then - echo "$0: debug[1]: about to: $JSONPATH_SH -S -A -j -f $JSON_FILE" 1>&2 - fi - "$JSONPATH_SH" -S -A -j -f "$JSON_FILE" - status="$?" - if [[ $status -ne 0 ]]; then - echo "$0: ERROR: $JSONPATH_SH -S -A -j -f $JSON_FILE failed," \ - "error code: $status" 1>&2 - exit 1 - fi + # case: use JSONPath.sh + # + JSONPath.sh) -fi + # case: processing JSON from stdin via JSONPath.sh + # + if [[ -z $JSON_FILE ]]; then + if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: about to: $JSONPATH_SH -S -A -j" 1>&2 + fi + "$JSONPATH_SH" -S -A -j + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: $JSONPATH_SH -S -A -j failed," \ + "error code: $status" 1>&2 + exit 1 + fi + + # processing JSON from a file via JSONPath.sh + # + else + if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: about to: $JSONPATH_SH -S -A -j -f $JSON_FILE" 1>&2 + fi + "$JSONPATH_SH" -S -A -j -f "$JSON_FILE" + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: $JSONPATH_SH -S -A -j -f $JSON_FILE failed," \ + "error code: $status" 1>&2 + exit 1 + fi + fi + ;; + + # case: we do not know which XPath JSON tool to use + # + *) + + echo "$0: ERROR: XPATHJSON_USE is neither jsp nor JSONPath.sh: $XPATHJSON_USE" 1>&2 + exit 12 + ;; +esac # All Done!!! All Done!!! -- Jessica Noll, Age 2 diff --git a/bin/jval-wrapper.sh b/bin/jval-wrapper.sh index 05a4e14d5..d3e8c367a 100755 --- a/bin/jval-wrapper.sh +++ b/bin/jval-wrapper.sh @@ -106,47 +106,55 @@ shopt -s globstar # enable ** to match all files and zero or more directories an # set variables referenced in the usage message # -export VERSION="1.0.4 2024-08-20" +export VERSION="1.1 2024-09-23" NAME=$(basename "$0") export NAME export V_FLAG=0 - - -# Until we have a true jval tool, we will use the modified JSONPath.sh -# -# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX -# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX -# XXX - until we have the jval command, we must FAKE PARSE IOCCC JSON files - XXX -# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX -# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX -# -export JSONPATH_REPO="https://github.com/lcn2/JSONPath.sh" -JSONPATH_SH=$(type -P JSONPath.sh) -export JSONPATH_SH -if [[ -z $JSONPATH_SH ]]; then - echo "$0: FATAL: JSONPath.sh tool is not installed or not in \$PATH" 1>&2 - echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2 +GIT_TOOL=$(type -P git) +export GIT_TOOL +if [[ -z "$GIT_TOOL" ]]; then + echo "$0: FATAL: git tool is not installed or not in \$PATH" 1>&2 exit 5 fi -# verify JSONPath.sh supports -S -A -export FIZZBIN_JSON='"fizzbin"' -if ! "$JSONPATH_SH" -S -A -p >/dev/null 2>&1; then - echo "$0: FATAL: JSONPath.sh tool does not support -S -A: $FIZZBIN_JSON" 1>&2 - echo "$0: notice: we recommend you obtain and install JSONPath.sh from: $JSONPATH_REPO" 1>&2 - exit 5 -fi <<< "$FIZZBIN_JSON" -export JSONPATH_ARG -export PATTERN='$*' +"$GIT_TOOL" rev-parse --is-inside-work-tree >/dev/null 2>&1 +status="$?" +if [[ $status -eq 0 ]]; then + TOPDIR=$("$GIT_TOOL" rev-parse --show-toplevel) +fi +export TOPDIR +export REPO_TOP_URL="https://github.com/ioccc-src/temp-test-ioccc" +# GitHub puts individual files under the "blob/master" sub-directory. +export REPO_URL="$REPO_TOP_URL/blob/master" + + +# Until we have a true jval tool, we will use wither jsp or the modified JSONPath.sh # -unset OPTION -declare -ag OPTION +# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX +# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX +# XXX - until we have the jval command, we use external tools to get data from JSON - XXX +# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX +# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX +# +export JSP_REPO="https://github.com/kjozsa/jsp" +export JSONPATH_REPO="https://github.com/lcn2/JSONPath.sh" +export FIZZBIN_JSON='"fizzbin"' +export JSP_TOOL="" +JSP_TOOL=$(type -P jsp) +export JSONPATH_SH="" +JSONPATH_SH=$(type -P JSONPath.sh) +# set the defaiult XPath for JSON +export XPATHJSON_USE="jsp" +export JSONPATH_ARG +export PATTERN="" +# +export PRINT_ONLY_VALUES="" +export REMOVE_QUOTES="" +export QUICK_CHECK="" # set usage message # -export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] - [-w] [-b] [-i] [-u] - [-j JSONPath.sh] file.json [pattern] +export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] [-b] [-q] [-t tool] [-Q] file.json [pattern] -h print help message and exit -v level set verbosity level (def level: 0) @@ -155,12 +163,11 @@ export USAGE="usage: $0 [-h] [-v level] [-V] [-n] [-N] -n do nothing (same as -N) (def: do the action) -N do not process file, just parse arguments and ignore the file (def: process the file) - -w match whole words only for filter script expression (def: match substrings) - -b only print values (def: print both JSON member and JSON value) - -i match insensitive (def: match case) - -u strip unnecessary leading path element (def: print the entire element path) + -b print only values (def: print both JSON member and JSON value) + -q remove enclosing double quotes (def: keep any enclosing double quotes) - -j JSONPath.sh path to the JSONPath.sh tool (not the wrapper) (def: $JSONPATH_SH) + -t tool tool to use: jsp or JSONPath.sh (def: try jsp, otherwise try JSONPath.sh) + -Q quick check tool using trivial input (def: do not) file.json JSON file to read, - ==> read stdin pattern JSONPath query (def: $PATTERN} @@ -186,7 +193,7 @@ export DO_NOT_PROCESS= # parse command line # -while getopts :hv:VnNwbiuj: flag; do +while getopts :hv:VnNbqt:Q flag; do case "$flag" in h) echo "$USAGE" 1>&2 exit 2 @@ -200,16 +207,23 @@ while getopts :hv:VnNwbiuj: flag; do ;; N) DO_NOT_PROCESS="-N" ;; - j) JSONPATH_SH="$OPTARG" - ;; - w) OPTION+=("-w") - ;; - b) OPTION+=("-b") - ;; - i) OPTION+=("-i") - ;; - u) OPTION+=("-u") + b) PRINT_ONLY_VALUES="-b" ;; + q) REMOVE_QUOTES="true" + ;; + t) # validate -t tool + case "$OPTARG" in + jsp) XPATHJSON_USE="jsp" ;; + JSONPath.sh) XPATHJSON_USE="JSONPath.sh" ;; + *) echo "$0: ERROR: unknown -t option: $OPTARG" 1>&2 + echo 1>&2 + echo "$USAGE" 1>&2 + exit 3 + ;; + esac + ;; + Q) QUICK_CHECK="true" + ;; \?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2 echo 1>&2 echo "$USAGE" 1>&2 @@ -251,6 +265,9 @@ case "$#" in ;; esac # +if [[ $JSON_FILE == "-" ]]; then + JSON_FILE="" +fi if [[ -n $JSON_FILE ]]; then if [[ ! -e $JSON_FILE ]]; then echo "$0: ERROR: JSON_FILE does not exist: $JSON_FILE" 1>&2 @@ -267,6 +284,155 @@ if [[ -n $JSON_FILE ]]; then fi +# verify that we have a topdir directory +# +REPO_NAME=$(basename "$REPO_TOP_URL") +export REPO_NAME +if [[ -z $TOPDIR ]]; then + echo "$0: ERROR: cannot find top of git repo directory" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi +if [[ ! -e $TOPDIR ]]; then + echo "$0: ERROR: TOPDIR does not exist: $TOPDIR" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi +if [[ ! -d $TOPDIR ]]; then + echo "$0: ERROR: TOPDIR is not a directory: $TOPDIR" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi + + +# cd to topdir +# +if [[ ! -e $TOPDIR ]]; then + echo "$0: ERROR: cannot cd to non-existent path: $TOPDIR" 1>&2 + exit 6 +fi +if [[ ! -d $TOPDIR ]]; then + echo "$0: ERROR: cannot cd to a non-directory: $TOPDIR" 1>&2 + exit 6 +fi +export CD_FAILED +if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to: cd $TOPDIR" 1>&2 +fi +cd "$TOPDIR" || CD_FAILED="true" +if [[ -n $CD_FAILED ]]; then + echo "$0: ERROR: cd $TOPDIR failed" 1>&2 + exit 6 +fi +if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: now in directory: $(/bin/pwd)" 1>&2 +fi + + +# verify that we have a bin subdirectory +# +export BIN_PATH="$TOPDIR/bin" +if [[ ! -d $BIN_PATH ]]; then + echo "$0: ERROR: bin is not a directory under topdir: $BIN_PATH" 1>&2 + exit 6 +fi +export BIN_DIR="bin" + + +# verify that the bin/unicode-fix.sed tool is executable +# +export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed" +if [[ ! -e $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -f $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -r $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi + + +# validate the XPath for JSON tool +# +case "$XPATHJSON_USE" in + + # case: we will use JSONPath.sh + # + JSONPath.sh) + + # try JSONPath.sh + # + if [[ -z $JSONPATH_SH || ! -x $JSONPATH_SH ]]; then + echo "$0: ERROR: JSONPath.sh tool is not installed, is not executable, or not in \$PATH" 1>&2 + echo "$0: notice: obtain jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + fi + + # case: -Q - perform quick check + # + if [[ -n $QUICK_CHECK ]]; then + # verify JSONPath.sh supports -S -A -p + if ! "$JSONPATH_SH" -S -A -p >/dev/null 2>&1; then + echo "$0: ERROR: JSONPath.sh tool does not support -S -A: $FIZZBIN_JSON" 1>&2 + echo "$0: notice: we recommend you obtain and install jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: otherwise install JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + fi <<< "$FIZZBIN_JSON" + fi + ;; + + # case: we will use jsp + # + jsp) + + # try jsp + # + if [[ -z $JSP_TOOL || ! -x $JSP_TOOL ]]; then + echo "$0: ERROR: jsp.sh tool is not installed, is not executable, or not in \$PATH" 1>&2 + echo "$0: notice: obtain jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: obtain JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + fi + + # case: -Q - perform quick check + # + if [[ -n $QUICK_CHECK ]]; then + # verify jsp + if ! "$JSP_TOOL" --indent 4 --format --no-color >/dev/null 2>&1; then + echo "$0: ERROR: jsp tool does not support --indent 4 --format --no-color: $FIZZBIN_JSON" 1>&2 + echo "$0: notice: you might need to update and reinstall jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: or you might try to obtain and install JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + fi <<< "$FIZZBIN_JSON" + fi + ;; + + # case: we do not have an XPath for JSON tool + # + *) + echo "$0: ERROR: cannot find an XPath JSON tool to use" 1>&2 + echo "$0: notice: we recommend you obtain and install jsp from: $JSP_REPO" 1>&2 + echo "$0: notice: or obtain jsp via \"pipx install jsp\" and placing ~/.local/bin in \$PATH" 1>&2 + echo "$0: notice: otherwise install JSONPath.sh from: $JSONPATH_REPO" 1>&2 + echo "$0: notice: if possible install jsp instead of JSONPath.sh as jsp is faster" 1>&2 + exit 5 + ;; +esac + + # print running info if verbose # # If -v 3 or higher, print exported variables in order that they were exported. @@ -275,17 +441,28 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: VERSION=$VERSION" 1>&2 echo "$0: debug[3]: NAME=$NAME" 1>&2 echo "$0: debug[3]: V_FLAG=$V_FLAG" 1>&2 + echo "$0: debug[3]: GIT_TOOL=$GIT_TOOL" 1>&2 + echo "$0: debug[3]: TOPDIR=$TOPDIR" 1>&2 + echo "$0: debug[3]: REPO_TOP_URL=$REPO_TOP_URL" 1>&2 + echo "$0: debug[3]: REPO_URL=$REPO_URL" 1>&2 + echo "$0: debug[3]: JSP_REPO=$JSP_REPO" 1>&2 echo "$0: debug[3]: JSONPATH_REPO=$JSONPATH_REPO" 1>&2 - echo "$0: debug[3]: JSONPATH_SH=$JSONPATH_SH" 1>&2 echo "$0: debug[3]: FIZZBIN_JSON=$FIZZBIN_JSON" 1>&2 - echo "$0: debug[3]: JSONPATH_ARG=$JSONPATH_ARG" 1>&2 - for index in "${!OPTION[@]}"; do - echo "$0: debug[$V_FLAG]: $V_FLAG: OPTION[$index]=${OPTION[$index]}" 1>&2 - done + echo "$0: debug[3]: JSP_TOOL=$JSP_TOOL" 1>&2 + echo "$0: debug[3]: JSONPATH_SH=$JSONPATH_SH" 1>&2 + echo "$0: debug[3]: XPATHJSON_USE=$XPATHJSON_USE" 1>&2 + echo "$0: debug[3]: PATTERN=$PATTERN" 1>&2 + echo "$0: debug[3]: PRINT_ONLY_VALUES=$PRINT_ONLY_VALUES" 1>&2 + echo "$0: debug[3]: REMOVE_QUOTES=$REMOVE_QUOTES" 1>&2 + echo "$0: debug[3]: QUICK_CHECK=$QUICK_CHECK" 1>&2 echo "$0: debug[3]: NOOP=$NOOP" 1>&2 echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2 echo "$0: debug[3]: JSON_FILE=$JSON_FILE" 1>&2 - echo "$0: debug[3]: PATTERN=$PATTERN" 1>&2 + echo "$0: debug[3]: REPO_NAME=$REPO_NAME" 1>&2 + echo "$0: debug[3]: CD_FAILED=$CD_FAILED" 1>&2 + echo "$0: debug[3]: BIN_PATH=$BIN_PATH" 1>&2 + echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2 + echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2 fi @@ -309,36 +486,245 @@ if [[ -n $NOOP ]]; then fi -# case: processing JSON from standard input +# canonically process JSON from standard input via jsp # -if [[ -z $JSON_FILE ]]; then +case "$XPATHJSON_USE" in - if [[ $V_FLAG -ge 1 ]]; then - echo "$0: debug[1]: about to: $JSONPATH_SH ${OPTION[*]} -- $PATTERN" 1>&2 - fi - "$JSONPATH_SH" "${OPTION[@]}" -- "$PATTERN" - status="$?" - if [[ $status -ne 0 ]]; then - echo "$0: ERROR: $JSONPATH_SH ${OPTION[*]} -- $PATTERN failed," \ - "error code: $status" 1>&2 - exit 1 - fi - -# case: processing JSON from a file +# case: use jsp # -else +jsp) - if [[ $V_FLAG -ge 1 ]]; then - echo "$0: debug[1]: about to: $JSONPATH_SH ${OPTION[*]} -f $JSON_FILE -- $PATTERN" 1>&2 + # if not set, set the default jsp pattern + # + PATTERN="${PATTERN:=$}" + + # case: jsp processing from standard input + # + if [[ -z $JSON_FILE ]]; then + + if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: about to: $JSP_TOOL --no-color --format --indent 0 -- '$PATTERN'" 1>&2 + fi + "$JSP_TOOL" --no-color --format --indent 0 -- "$PATTERN" + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: $JSP_TOOL --no-color --format --indent 0 -- '$PATTERN' failed", \ + "error code: $status" 1>&2 + exit 1 + fi + + # case: jsp processing from a file + # + else + + if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: about to: $JSP_TOOL --no-color --format --indent 0 -- '$PATTERN' < $JSON_FILE" 1>&2 + fi + "$JSP_TOOL" --no-color --format --indent 0 -- "$PATTERN" < "$JSON_FILE" + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: $JSP_TOOL --no-color --format --indent 0 -- '$PATTERN' < $JSON_FILE failed", \ + "error code: $status" 1>&2 + exit 1 + fi + + # case: print only values from jsp output + # + fi | if [[ -n $PRINT_ONLY_VALUES ]]; then + + # case: -q -b (print only values and remove double quotes) via jsp + # + if [[ -n $REMOVE_QUOTES ]]; then + + sed -E -e '/^\s*[]{}[],?\s*$/d' \ + -e 's/^\s*("[^"]*[^\\]*"):\s+/\1\t/' \ + -e 's/^"[^\t]*"\t//' \ + -e 's/,?$//' \ + -e 's/^"//' \ + -e 's/",?$//' \ + -e '/^\[$/d' + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: pipe from -t jsp options: -q -b: sed -E -e ... failed", \ + "error code: $status" 1>&2 + exit 1 + fi + + # case: -b (print only values) via jsp + # + else + + sed -E -e '/^\s*[]{}[],?\s*$/d' \ + -e 's/^\s*("[^"]*[^\\]*"):\s+/\1\t/' \ + -e 's/^"[^\t]*"\t//' \ + -e 's/,?$//' \ + -e '/^\[$/d' + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: pipe from -t jsp options: -b: sed -E -e ... failed", \ + "error code: $status" 1>&2 + exit 1 + fi + fi + + # case: print both JSON member and JSON value from jsp output + # + else + + # case: -q without -b (print both JSON member and JSON value and remove double quotes) via jsp + # + if [[ -n $REMOVE_QUOTES ]]; then + + sed -E -e '/^\s*[]{}[],?\s*$/d' \ + -e 's/^\s*("[^"]*[^\\]*"):\s+/\1\t/' \ + -e 's/,?$//' \ + -e 's/^"//' \ + -e 's/"\t"?/\t/' \ + -e 's/"$//' \ + -e '/\t\[$/d' + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: pipe from -t jsp options: -q: sed -E -e ... failed", \ + "error code: $status" 1>&2 + exit 1 + fi + + # case: without -b (print only values) via jsp + # + else + + sed -E -e '/^\s*[]{}[],?\s*$/d' \ + -e 's/^\s*("[^"]*[^\\]*"):\s+/\1\t/' \ + -e 's/,?$//' \ + -e '/\t\[$/d' + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: pipe from -t jsp: sed -E -e ... failed", \ + "error code: $status" 1>&2 + exit 1 + fi + + fi fi - "$JSONPATH_SH" "${OPTION[@]}" -f "$JSON_FILE" -- "$PATTERN" - status="$?" - if [[ $status -ne 0 ]]; then - echo "$0: ERROR: $JSONPATH_SH ${OPTION[*]} -f $JSON_FILE -- $PATTERN failed," \ - "error code: $status" 1>&2 - exit 1 + ;; + + +# case: use JSONPath.sh +# +JSONPath.sh) + + # if not set, set the default JSONPath.sh pattern + # + PATTERN="${PATTERN:=$.*}" + + # case: JSONPath.sh processing from standard input + # + if [[ -z $JSON_FILE ]]; then + + if [[ $V_FLAG -ge 1 ]]; then + echo "$0: debug[1]: about to: $JSONPATH_SH -w -u -- '$PATTERN'" 1>&2 + fi + "$JSONPATH_SH" -w -u -- "$PATTERN" + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: $JSONPATH_SH -w -u -- '$PATTERN' failed", \ + "error code: $status" 1>&2 + exit 1 + fi + + # case: JSONPath.sh processing from a file + # + else + + if [[ $V_FLAG -ge 1 ]]; then + echo "$0: debug[1]: about to: $JSONPATH_SH -w -u -f $JSON_FILE -- '$PATTERN'" 1>&2 + fi + "$JSONPATH_SH" -w -u -f "$JSON_FILE" -- "$PATTERN" + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: $JSONPATH_SH -w -u -f $JSON_FILE -- '$PATTERN' failed", \ + "error code: $status" 1>&2 + exit 1 + fi + + # case: print only values from JSONPath.sh output + # + fi | if [[ -n $PRINT_ONLY_VALUES ]]; then + + # case: -q -b (print only values and remove double quotes and []'s) via JSONPath.sh + # + if [[ -n $REMOVE_QUOTES ]]; then + + sed -E -e 's/^\S*\s*//' \ + -e 's/^"//' \ + -e 's/"$//' + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: pipe from -t jJSONPath.sh options: -q -b: sed -E -e ... failed", \ + "error code: $status" 1>&2 + exit 1 + fi + + # case: -b (print only values) via JSONPath.sh + # + else + + sed -E -e 's/^\S*\s*//' + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: pipe from -t jJSONPath.sh options: -b: sed -E -e ... failed", \ + "error code: $status" 1>&2 + exit 1 + fi + fi + + # case: print both JSON member and JSON value from JSONPath.sh output + # + else + + # case: -q without -b (print both JSON member and JSON value and remove double quotes and []'s) via JSONPath.sh + # + if [[ -n $REMOVE_QUOTES ]]; then + + sed -E -e 's/^\s+//' \ + -e 's/\s+$//' \ + -e 's/^\["?//' \ + -e 's/^(\S*),"/\1,/' \ + -e 's/\]\t/\t/' \ + -e 's/"\t"?/\t/' \ + -e 's/"$//' + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: pipe from -t JSONPath.sh options: -q: sed -E -e ... failed", \ + "error code: $status" 1>&2 + exit 1 + fi + + # case: without -b (print only values) via JSONPath.sh + # + else + + sed -E -e 's/^\s+//' \ + -e 's/\s+$//' + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: pipe from -t JSONPath.sh: sed -E -e ... failed", \ + "error code: $status" 1>&2 + exit 1 + fi + fi fi -fi + ;; + + +# case: we do not know which XPath JSON tool to use +# +*) + + echo "$0: ERROR: XPATHJSON_USE is neither jsp nor JSONPath.sh: $XPATHJSON_USE" 1>&2 + exit 10 + ;; +esac # All Done!!! All Done!!! -- Jessica Noll, Age 2 diff --git a/bin/manifest.entry.json.awk b/bin/manifest.entry.json.awk index e055a08cd..49c39e912 100755 --- a/bin/manifest.entry.json.awk +++ b/bin/manifest.entry.json.awk @@ -23,7 +23,7 @@ BEGIN { # setup # - VERSION="1.1.2 2024-07-28" + VERSION="1.2 2024-09-17" found_manifest_array = 0; # 1 ==> we found the manifest ARRAY within_manifest_array = 0; # 1 ==> found start of "manifest" : [ JSON array begin_manifest_element = 0; # 1 ==> we have found the start of the manifest array element @@ -121,7 +121,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^" # $1 = ""; # remove 1st field $2 = ""; # remove 2nd field - gsub(/^[[:space:]]+/, "", $0); # any leading whitespace + gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace file_path = $0; # Remove any trailing , @@ -155,7 +155,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^" # $1 = ""; # remove 1st field $2 = ""; # remove 2nd field - gsub(/^[[:space:]]+/, "", $0); # any leading whitespace + gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace inventory_order = $0; # Remove any trailing , @@ -189,7 +189,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^" # $1 = ""; # remove 1st field $2 = ""; # remove 2nd field - gsub(/^[[:space:]]+/, "", $0); # any leading whitespace + gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace OK_to_edit = $0; # Remove any trailing , @@ -223,7 +223,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^" # $1 = ""; # remove 1st field $2 = ""; # remove 2nd field - gsub(/^[[:space:]]+/, "", $0); # any leading whitespace + gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace display_as = $0; # Remove any trailing , @@ -257,7 +257,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^" # $1 = ""; # remove 1st field $2 = ""; # remove 2nd field - gsub(/^[[:space:]]+/, "", $0); # any leading whitespace + gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace display_via_github = $0; # Remove any trailing , @@ -291,7 +291,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^" # $1 = ""; # remove 1st field $2 = ""; # remove 2nd field - gsub(/^[[:space:]]+/, "", $0); # any leading whitespace + gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace entry_text = $0; # Remove any trailing , @@ -300,12 +300,50 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^" entry_text = substr(entry_text, 1, length(entry_text)-1); } + # so we can single quote the entry_text as a single string, we protect any single quotes within it + # + jstrdecode_arg = entry_text; + gsub(/'/, "'\"'\"'", jstrdecode_arg); + + # form the jstrdecode command + # + if (jstrdecode_arg ~ /^".*"$/) { + jstrdecode_cmd = "jstrdecode -Q -- " "'" substr(jstrdecode_arg, 2, length(jstrdecode_arg)-2) "'"; + } else { + jstrdecode_cmd = "jstrdecode -Q -- " "'" jstrdecode_arg "'"; + } + jstrdecode_cmd = jstrdecode_cmd " 2>/dev/null; echo $?"; + + # JSON decode entry_text + # + numLines = 0; + result = ""; + exit_code = 0; + # process each pipe line + while ((jstrdecode_cmd | getline pipe_output) > 0) { + if (numLines++ == 0) { + # case: 1st line is from jstrdecode + result = pipe_output; + } else { + # case: 2nd/line line is the jstrdecode exit code + exit_code = pipe_output; + } + } + close(jstrdecode_cmd); + + # validate exit code from jstrdecode + # + if (length(exit_code) == 0 || exit_code != 0) { + print "ERROR: unable to jstrdecode entry_text:", entry_text > "/dev/stderr"; + exit 213; # END section will output ERROR message about cannot JSON decode entry_text + } + # remove any enclosing double quotes # - if (entry_text ~ /^".*"$/) { - unquote_entry_text = substr(entry_text, 2, length(entry_text)-2); + if (result ~ /^".*"$/) { + unquote_entry_text = substr(result, 2, length(result)-2); } else { - unquote_entry_text = entry_text; + unquote_entry_text = result; } } @@ -317,42 +355,42 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF == 1 && $1 ~ /^} # if (length(file_path) == 0) { print "ERROR: manifest array JSON member did not have a file_path JSON member in:", ARGV[1] > "/dev/stderr"; - exit 213; + exit 214; } # verify we have seen a inventory_order JSON member # if (length(inventory_order) == 0) { print "ERROR: manifest array JSON member did not have a inventory_order JSON member in:", ARGV[1] > "/dev/stderr"; - exit 214; + exit 215; } # verify we have seen a OK_to_edit JSON member # if (length(OK_to_edit) == 0) { print "ERROR: manifest array JSON member did not have a OK_to_edit JSON member in:", ARGV[1] > "/dev/stderr"; - exit 215; + exit 216; } # verify we have seen a display_as JSON member # if (length(display_as) == 0) { print "ERROR: manifest array JSON member did not have a display_as JSON member in:", ARGV[1] > "/dev/stderr"; - exit 216; + exit 217; } # verify we have seen a display_via_github JSON member # if (length(display_via_github) == 0) { print "ERROR: manifest array JSON member did not have a display_via_github JSON member in:", ARGV[1] > "/dev/stderr"; - exit 217; + exit 218; } # verify we have seen a entry_text JSON member # if (length(entry_text) == 0) { print "ERROR: manifest array JSON member did not have a entry_text JSON member in:", ARGV[1] > "/dev/stderr"; - exit 218; + exit 219; } # print manifest information @@ -395,39 +433,39 @@ END { # if (length(github) == 0) { print "ERROR: github variable not set, call with -v github=REPO_URL" > "/dev/stderr"; - exit 219; # use 200 to match exit code from BEGIN + exit 220; # use 200 to match exit code from BEGIN } # case: we quit because we could not determine dir from ARGV[1] # if (length(dir) == 0) { print "ERROR: cannot determine dir from:", ARGV[1] > "/dev/stderr"; - exit 220; # END section will output ERROR message about cannot find dir + exit 221; # END section will output ERROR message about cannot find dir } if (length(YYYY) == 0) { print "ERROR: cannot determine YYYY from:", ARGV[1] > "/dev/stderr"; - exit 221; # END section will output ERROR message about cannot find YYYY + exit 222; # END section will output ERROR message about cannot find YYYY } # case: no manifest array was found # if (!found_manifest_array) { print "ERROR: we did not find a manifest JSON array in:", ARGV[1] > "/dev/stderr"; - exit 222; + exit 223; } # case: manifest array was found, but manifest array element did not finish # if (begin_manifest_element) { print "ERROR: manifest JSON array element did not end in:", ARGV[1] > "/dev/stderr"; - exit 223; + exit 224; } # case: manifest array did not end # if (within_manifest_array) { print "ERROR: manifest JSON array did not end in:", ARGV[1] > "/dev/stderr"; - exit 224; + exit 225; } } diff --git a/bin/output-index-author.sh b/bin/output-index-author.sh index 34bb1d106..27f8d9def 100755 --- a/bin/output-index-author.sh +++ b/bin/output-index-author.sh @@ -104,7 +104,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an # set variables referenced in the usage message # -export VERSION="1.8 2024-08-17" +export VERSION="1.9 2024-09-23" NAME=$(basename "$0") export NAME export V_FLAG=0 @@ -188,6 +188,7 @@ export DO_NOT_PROCESS= function output_author_handles { local ENTRY_JSON_PATH; # the .entry.json path + local PATTERN; # XPath for JSON pattern # parse args # @@ -211,10 +212,11 @@ function output_author_handles # extract author handles from .entry.json # - "$JVAL_WRAPPER" -w -b "$ENTRY_JSON_PATH" '$..author_handle' + PATTERN='$..author_handle' + "$JVAL_WRAPPER" -b -q "$ENTRY_JSON_PATH" "$PATTERN" status="$?" if [[ $status -ne 0 ]]; then - echo "$0: ERROR: $JVAL_WRAPPER -w -b $ENTRY_JSON_PATH \$..author_handle failed," \ + echo "$0: ERROR: in output_author_handles: $JVAL_WRAPPER -b -q $ENTRY_JSON_PATH '$PATTERN' failed," \ "error: $status" 1>&2 return 5 fi @@ -236,6 +238,7 @@ function output_author_handles function output_entry_ids { local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path + local PATTERN; # XPath for JSON pattern # parse args # @@ -259,10 +262,11 @@ function output_entry_ids # extract entry ids from the author/author_handle.json file # - "$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..entry_id' + PATTERN='$..entry_id' + "$JVAL_WRAPPER" -b -q "$AUTHOR_HANDLE_JSON_PATH" "$PATTERN" status="$?" if [[ $status -ne 0 ]]; then - echo "$0: ERROR: $JVAL_WRAPPER -w -b $AUTHOR_HANDLE_JSON_PATH \$..entry_id failed," \ + echo "$0: ERROR: in output_entry_ids: $JVAL_WRAPPER -b -q $AUTHOR_HANDLE_JSON_PATH '$PATTERN' failed," \ "error: $status" 1>&2 return 5 fi @@ -283,6 +287,7 @@ function output_entry_ids function output_full_name { local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path + local PATTERN; # XPath for JSON pattern # parse args # @@ -306,11 +311,12 @@ function output_full_name # extract Full Name from the author/author_handle.json file # - "$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..full_name' - status="$?" - if [[ $status -ne 0 ]]; then - echo "$0: ERROR: $JVAL_WRAPPER -w -b $AUTHOR_HANDLE_JSON_PATH \$..full_name failed," \ - "error: $status" 1>&2 + PATTERN='$..full_name' + "$JVAL_WRAPPER" -b -q "$AUTHOR_HANDLE_JSON_PATH" "$PATTERN" | sed -f "$UNICODE_FIX_SED" + status_codes=("${PIPESTATUS[@]}") + if [[ ${status_codes[*]} =~ [1-9] ]]; then + echo "$0: ERROR: in output_full_name: $JVAL_WRAPPER -b -q $AUTHOR_HANDLE_JSON_PATH '$PATTERN' failed," \ + "error codes: ${status_codes[*]}" 1>&2 return 5 fi return 0 @@ -331,6 +337,7 @@ function output_full_name function output_location_code { local AUTHOR_HANDLE_JSON_PATH; # the .entry.json path + local PATTERN; # XPath for JSON pattern # parse args # @@ -354,10 +361,11 @@ function output_location_code # extract Location Code from the author/author_handle.json file # - "$JVAL_WRAPPER" -w -b "$AUTHOR_HANDLE_JSON_PATH" '$..location_code' + PATTERN='$..location_code' + "$JVAL_WRAPPER" -b -q "$AUTHOR_HANDLE_JSON_PATH" "$PATTERN" status="$?" if [[ $status -ne 0 ]]; then - echo "$0: ERROR: $JVAL_WRAPPER -w -b $ENTRY_JSON_PATH \$..location_code failed," \ + echo "$0: ERROR: in output_location_code: $JVAL_WRAPPER -b -q $ENTRY_JSON_PATH '$PATTERN' failed," \ "error: $status" 1>&2 return 5 fi @@ -638,6 +646,23 @@ if [[ ! -x $LOCATION_TOOL ]]; then fi +# verify that the bin/unicode-fix.sed tool is executable +# +export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed" +if [[ ! -e $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -f $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -r $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi + + # parameter debugging # if [[ $V_FLAG -ge 3 ]]; then @@ -673,6 +698,7 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: DOT_PATH_CONTENT=$DOT_PATH_CONTENT" 1>&2 echo "$0: debug[3]: ENTRY_JSON=$ENTRY_JSON" 1>&2 echo "$0: debug[3]: LOCATION_TOOL=$LOCATION_TOOL" 1>&2 + echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2 fi diff --git a/bin/output-year-index.sh b/bin/output-year-index.sh index 95fffc779..dd76b73fa 100755 --- a/bin/output-year-index.sh +++ b/bin/output-year-index.sh @@ -113,7 +113,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an # set variables referenced in the usage message # -export VERSION="1.4.4 2024-08-05" +export VERSION="1.5 2024-09-23" NAME=$(basename "$0") export NAME export V_FLAG=0 @@ -190,6 +190,7 @@ $NAME version: $VERSION" function output_award { local ENTRY_JSON_PATH; # the .entry.json path + local PATTERN; # XPath for JSON pattern # parse args # @@ -213,7 +214,17 @@ function output_award # obtain the award string # - "$JVAL_WRAPPER" -w -b "$ENTRY_JSON_PATH" '$..award' + PATTERN='$..award' + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN'" 1>&2 + fi + "$JVAL_WRAPPER" -b -q "$ENTRY_JSON_PATH" "$PATTERN" + status="$?" + if [[ $status -ne 0 ]]; then + echo "$0: ERROR: in output_award: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN' failed," \ + "error code: $status" 1>&2 + return 5 + fi return 0 } @@ -523,6 +534,23 @@ if [[ -z $ORDINAL ]]; then fi +# verify that the bin/unicode-fix.sed tool is executable +# +export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed" +if [[ ! -e $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -f $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -r $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi + + # parameter debugging # if [[ $V_FLAG -ge 3 ]]; then @@ -544,6 +572,7 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: YEAR_FILE=$YEAR_FILE" 1>&2 echo "$0: debug[3]: LINE_NUM=$LINE_NUM" 1>&2 echo "$0: debug[3]: ORDINAL=$ORDINAL" 1>&2 + echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2 fi @@ -668,9 +697,9 @@ for YYYY_DIR in $(< "$YEAR_FILE"); do # determine the award for this entry # - AWARD=$(output_award "$ENTRY_JSON") - export AWARD - if [[ -z $AWARD ]]; then + AWARD=$(output_award "$ENTRY_JSON" | sed -f "$UNICODE_FIX_SED") + status_codes=("${PIPESTATUS[@]}") + if [[ ${status_codes[*]} =~ [1-9] || -z $AWARD ]]; then echo "$0: ERROR: cannot find award in .entry.json: $ENTRY_JSON" 1>&2 exit 7 fi diff --git a/bin/subst.entry-index.sh b/bin/subst.entry-index.sh index 9bea1c16f..0db1efe30 100755 --- a/bin/subst.entry-index.sh +++ b/bin/subst.entry-index.sh @@ -106,7 +106,7 @@ shopt -s globstar # enable ** to match all files and zero or more directories an # set variables referenced in the usage message # -export VERSION="1.5 2024-08-13" +export VERSION="1.6 2024-09-23" NAME=$(basename "$0") export NAME export V_FLAG=0 @@ -180,6 +180,7 @@ $NAME version: $VERSION" function output_award { local ENTRY_JSON_PATH; # the .entry.json path + local PATTERN; # XPath for JSON pattern # parse args # @@ -203,7 +204,18 @@ function output_award # obtain the award string # - "$JVAL_WRAPPER" -w -b "$ENTRY_JSON_PATH" '$..award' + PATTERN='$..award' + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to run: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN' | sed ..." 1>&2 + fi + "$JVAL_WRAPPER" -b -q "$ENTRY_JSON_PATH" "$PATTERN" | sed -f "$UNICODE_FIX_SED" + status="$?" + status_codes=("${PIPESTATUS[@]}") + if [[ ${status_codes[*]} =~ [1-9] ]]; then + echo "$0: ERROR: in output_award: $JVAL_WRAPPER -b -q -- $ENTRY_JSON_PATH '$PATTERN | sed ... failed," \ + "error codes: ${status_codes[*]}" 1>&2 + return 5 + fi return 0 } @@ -381,6 +393,23 @@ if [[ ! -x $JVAL_WRAPPER ]]; then fi +# verify that the bin/unicode-fix.sed tool is executable +# +export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed" +if [[ ! -e $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -f $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -r $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi + + # verify we have our awk script # export ENTRY_NAVBAR_AWK="$BIN_DIR/subst.entry-navbar.awk" @@ -506,6 +535,7 @@ if [[ $V_FLAG -ge 3 ]]; then echo "$0: debug[3]: BIN_PATH=$BIN_PATH" 1>&2 echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2 echo "$0: debug[3]: JVAL_WRAPPER=$JVAL_WRAPPER" 1>&2 + echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2 echo "$0: debug[3]: ENTRY_JSON=$ENTRY_JSON" 1>&2 echo "$0: debug[3]: YEAR_DIR=$YEAR_DIR" 1>&2 echo "$0: debug[3]: ENTRY_DIR=$ENTRY_DIR" 1>&2 diff --git a/bin/unicode-chk.sh b/bin/unicode-chk.sh new file mode 100755 index 000000000..b531ffdf4 --- /dev/null +++ b/bin/unicode-chk.sh @@ -0,0 +1,690 @@ +#!/usr/bin/env bash +# +# unicode-chk.sh - verify all \\u[[:xdigit:]]{4} in entry and author JSON files +# +# Pass entry and author JSON files thru the following sed script and +# flag any \\u[[:xdigit:]]{4} that we might find as problems. +# +# bin/unicode-fix.sed +# +# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX +# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX +# XXX - until we have fixed jstrdecode, we must sed out \u[0-0A-fa-f]{4} from JSON - XXX +# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX +# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX +# +# Until this bug is the jparse repo is fixed, we need to avoid +# using \\u[[:xdigit:]]{4} within JSON encoded strings. +# +# https://github.com/xexyl/jparse/issues/13 +# +# The above mentioned bug results in certain tools that are based on the +# jparse repo: +# +# https://github.com/xexyl/jparse +# +# including code from the mkiocccentry repo's jparse sub-directory: +# +# https://github.com/ioccc-src/mkiocccentry/tree/master/jparse +# +# We need to avoid using \\u[[:xdigit:]]{4} within JSON encoded strings. +# Sorry (tm Canada :-)). +# +# The following sed script converts known problematic \\u[[:xdigit:]]{4} +# combinations into their RAW character form: +# +# bin/unicode-fix.sed +# +# Use of that sed script serves as a work-a-round to avoid to avoid any +# problematic \\u[[:xdigit:]]{4} combinations in JSON encoded strings. +# +# Copyright (c) 2024 by Landon Curt Noll. All Rights Reserved. +# +# Permission to use, copy, modify, and distribute this software and +# its documentation for any purpose and without fee is hereby granted, +# provided that the above copyright, this permission notice and text +# this comment, and the disclaimer below appear in all of the following: +# +# supporting documentation +# source copies +# source works derived from this source +# binaries derived from this source or from derived source +# +# LANDON CURT NOLL DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, +# INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO +# EVENT SHALL LANDON CURT NOLL BE LIABLE FOR ANY SPECIAL, INDIRECT OR +# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF +# USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR +# OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR +# PERFORMANCE OF THIS SOFTWARE. +# +# chongo (Landon Curt Noll, http://www.isthe.com/chongo/index.html) /\oo/\ +# +# Share and enjoy! :-) + + +# firewall - run only with a bash that is version 5.1.8 or later +# +# The "/usr/bin/env bash" command must result in using a bash that +# is version 5.1.8 or later. +# +# We could relax this version and insist on version 4.2 or later. Versions +# of bash between 4.2 and 5.1.7 might work. However, to be safe, we will require +# bash version 5.1.8 or later. +# +# WHY 5.1.8 and not 4.2? This safely is done because macOS Homebrew bash we +# often use is "version 5.2.26(1)-release" or later, and the RHEL Linux bash we +# use often use is "version 5.1.8(1)-release" or later. These versions are what +# we initially tested. We recommend you either upgrade bash or install a newer +# version of bash and adjust your $PATH so that "/usr/bin/env bash" finds a bash +# that is version 5.1.8 or later. +# +# NOTE: The macOS shipped, as of 2024 March 15, a version of bash is something like +# bash "version 3.2.57(1)-release". That macOS shipped version of bash +# will NOT work. For users of macOS we recommend you install Homebrew, +# (see https://brew.sh), and then run "brew install bash" which will +# typically install it into /opt/homebrew/bin/bash, and then arrange your $PATH +# so that "/usr/bin/env bash" finds "/opt/homebrew/bin" (or whatever the +# Homebrew bash is). +# +# NOTE: And while MacPorts might work, we noticed a number of subtle differences +# with some of their ported tools to suggest you might be better off +# with installing Homebrew (see https://brew.sh). No disrespect is intended +# to the MacPorts team as they do a commendable job. Nevertheless we ran +# into enough differences with MacPorts environments to suggest you +# might find a better experience with this tool under Homebrew instead. +# +if [[ -z ${BASH_VERSINFO[0]} || + ${BASH_VERSINFO[0]} -lt 5 || + ${BASH_VERSINFO[0]} -eq 5 && ${BASH_VERSINFO[1]} -lt 1 || + ${BASH_VERSINFO[0]} -eq 5 && ${BASH_VERSINFO[1]} -eq 1 && ${BASH_VERSINFO[2]} -lt 8 ]]; then + echo "$0: ERROR: bash version needs to be >= 5.1.8: $BASH_VERSION" 1>&2 + echo "$0: Warning: bash version >= 4.2 might work but 5.1.8 was the minimum we tested" 1>&2 + echo "$0: Notice: For macOS users: install Homebrew (see https://brew.sh), then run" \ + ""brew install bash" and then modify your \$PATH so that \"#!/usr/bin/env bash\"" \ + "finds the Homebrew installed (usually /opt/homebrew/bin/bash) version of bash" 1>&2 + exit 4 +fi + + +# setup bash file matching +# +# We must declare arrays with -ag or -Ag, and we need loops to "export" modified variables. +# This requires a bash with a version 4.2 or later. See the larger comment above about bash versions. +# +shopt -s nullglob # enable expanded to nothing rather than remaining unexpanded +shopt -u failglob # disable error message if no matches are found +shopt -u dotglob # disable matching files starting with . +shopt -u nocaseglob # disable strict case matching +shopt -u extglob # enable extended globbing patterns +shopt -s globstar # enable ** to match all files and zero or more directories and subdirectories + + +# set variables referenced in the usage message +# +export VERSION="1.0 2024-09-23" +NAME=$(basename "$0") +export NAME +export V_FLAG=0 +GIT_TOOL=$(type -P git) +export GIT_TOOL +if [[ -z "$GIT_TOOL" ]]; then + echo "$0: FATAL: git tool is not installed or not in \$PATH" 1>&2 + exit 5 +fi +"$GIT_TOOL" rev-parse --is-inside-work-tree >/dev/null 2>&1 +status="$?" +if [[ $status -eq 0 ]]; then + TOPDIR=$("$GIT_TOOL" rev-parse --show-toplevel) +fi +export TOPDIR +export DOCROOT_SLASH="../../" +export TAGLINE="unspecified" +export REPO_TOP_URL="https://github.com/ioccc-src/temp-test-ioccc" +# GitHub puts individual files under the "blob/master" sub-directory. +export REPO_URL="$REPO_TOP_URL/blob/master" +export SITE_URL="https://ioccc-src.github.io/temp-test-ioccc" +# +export NOOP= +export DO_NOT_PROCESS= +export EXIT_CODE="0" + + +# usage +# +export USAGE="usage: $0 [-h] [-v level] [-V] [-d topdir] [-D docroot/] [-n] [-N] [file.json ..] + + -h print help message and exit + -v level set verbosity level (def level: 0) + -V print version string and exit + + -d topdir set topdir (def: $TOPDIR) + -D docroot/ set the document root path followed by slash (def: $DOCROOT_SLASH) + NOTE: The '-D docroot/' is passed as leading options on tool command lines. + NOTE: 'docroot' must end in a slash + + -n go thru the actions, but do not update any files (def: do the action) + -N do not process anything, just parse arguments (def: process something) + + [file.json ...] check file.json files (def: check all entry and author JSON files) + +Exit codes: + 0 all OK + 1 found unexpected \\uHexHexHexHex strings in JSON files + 2 -h and help string printed or -V and version string printed + 3 command line error + 4 bash version is too old + 5 some internal tool is not found or not an executable file + 6 problems found with or in the topdir or topdir/YYYY directory + 7 problems found with or in the entry topdir/YYYY/dir directory + 8 file.json not a readable file + >= 10 internal error + +$NAME version: $VERSION" + + +# parse command line +# +while getopts :hv:Vd:D:j:nN flag; do + case "$flag" in + h) echo "$USAGE" 1>&2 + exit 2 + ;; + v) V_FLAG="$OPTARG" + ;; + V) echo "$VERSION" + exit 2 + ;; + d) TOPDIR="$OPTARG" + ;; + D) # parse -D docroot/ + case "$OPTARG" in + */) ;; + *) echo "$0: ERROR: in -D docroot/, the docroot must end in /" 1>&2 + echo 1>&2 + print_usage 1>&2 + exit 3 + ;; + esac + DOCROOT_SLASH="$OPTARG" + ;; + n) NOOP="-n" + ;; + N) DO_NOT_PROCESS="-N" + ;; + \?) echo "$0: ERROR: invalid option: -$OPTARG" 1>&2 + echo 1>&2 + print_usage 1>&2 + exit 3 + ;; + :) echo "$0: ERROR: option -$OPTARG requires an argument" 1>&2 + echo 1>&2 + echo "$USAGE" 1>&2 + exit 3 + ;; + *) echo "$0: ERROR: unexpected value from getopts: $flag" 1>&2 + echo 1>&2 + echo "$USAGE" 1>&2 + exit 3 + ;; + esac +done +# +# remove the options +# +shift $(( OPTIND - 1 )); +# +if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: file argument count: $#" 1>&2 +fi + + +# verify that we have a topdir directory +# +REPO_NAME=$(basename "$REPO_TOP_URL") +export REPO_NAME +if [[ -z $TOPDIR ]]; then + echo "$0: ERROR: cannot find top of git repo directory" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi +if [[ ! -e $TOPDIR ]]; then + echo "$0: ERROR: TOPDIR does not exist: $TOPDIR" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi +if [[ ! -d $TOPDIR ]]; then + echo "$0: ERROR: TOPDIR is not a directory: $TOPDIR" 1>&2 + echo "$0: Notice: if needed: $GIT_TOOL clone $REPO_TOP_URL; cd $REPO_NAME" 1>&2 + exit 6 +fi + + +# cd to topdir +# +if [[ ! -e $TOPDIR ]]; then + echo "$0: ERROR: cannot cd to non-existent path: $TOPDIR" 1>&2 + exit 6 +fi +if [[ ! -d $TOPDIR ]]; then + echo "$0: ERROR: cannot cd to a non-directory: $TOPDIR" 1>&2 + exit 6 +fi +export CD_FAILED +if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to: cd $TOPDIR" 1>&2 +fi +cd "$TOPDIR" || CD_FAILED="true" +if [[ -n $CD_FAILED ]]; then + echo "$0: ERROR: cd $TOPDIR failed" 1>&2 + exit 6 +fi +if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: now in directory: $(/bin/pwd)" 1>&2 +fi + + +# verify that we have an author subdirectory +# +export AUTHOR_PATH="$TOPDIR/author" +if [[ ! -d $AUTHOR_PATH ]]; then + echo "$0: ERROR: author is not a directory under topdir: $AUTHOR_PATH" 1>&2 + exit 6 +fi +export AUTHOR_DIR="author" + + +# verify that we have a bin subdirectory +# +export BIN_PATH="$TOPDIR/bin" +if [[ ! -d $BIN_PATH ]]; then + echo "$0: ERROR: bin is not a directory under topdir: $BIN_PATH" 1>&2 + exit 6 +fi +export BIN_DIR="bin" + + +# verify that the bin/unicode-fix.sed tool is executable +# +export UNICODE_FIX_SED="$BIN_DIR/unicode-fix.sed" +if [[ ! -e $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed does not exist: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -f $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not a regular file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi +if [[ ! -r $UNICODE_FIX_SED ]]; then + echo "$0: ERROR: bin/unicode-fix.sed is not an readable file: $UNICODE_FIX_SED" 1>&2 + exit 5 +fi + + +# verify we have a non-empty readable .top file +# +export TOP_FILE=".top" +if [[ ! -e $TOP_FILE ]]; then + echo "$0: ERROR: .top does not exist: $TOP_FILE" 1>&2 + exit 6 +fi +if [[ ! -f $TOP_FILE ]]; then + echo "$0: ERROR: .top is not a regular file: $TOP_FILE" 1>&2 + exit 6 +fi +if [[ ! -r $TOP_FILE ]]; then + echo "$0: ERROR: .top is not an readable file: $TOP_FILE" 1>&2 + exit 6 +fi +if [[ ! -s $TOP_FILE ]]; then + echo "$0: ERROR: .top is not a non-empty readable file: $TOP_FILE" 1>&2 + exit 6 +fi + + +# print running info if verbose +# +# If -v 3 or higher, print exported variables in order that they were exported. +# +if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: VERSION=$VERSION" 1>&2 + echo "$0: debug[3]: NAME=$NAME" 1>&2 + echo "$0: debug[3]: V_FLAG=$V_FLAG" 1>&2 + echo "$0: debug[3]: GIT_TOOL=$GIT_TOOL" 1>&2 + echo "$0: debug[3]: TOPDIR=$TOPDIR" 1>&2 + echo "$0: debug[3]: DOCROOT_SLASH=$DOCROOT_SLASH" 1>&2 + echo "$0: debug[3]: TAGLINE=$TAGLINE" 1>&2 + echo "$0: debug[3]: REPO_TOP_URL=$REPO_TOP_URL" 1>&2 + echo "$0: debug[3]: REPO_URL=$REPO_URL" 1>&2 + echo "$0: debug[3]: SITE_URL=$SITE_URL" 1>&2 + echo "$0: debug[3]: NOOP=$NOOP" 1>&2 + echo "$0: debug[3]: DO_NOT_PROCESS=$DO_NOT_PROCESS" 1>&2 + echo "$0: debug[3]: EXIT_CODE=$EXIT_CODE" 1>&2 + echo "$0: debug[3]: REPO_NAME=$REPO_NAME" 1>&2 + echo "$0: debug[3]: CD_FAILED=$CD_FAILED" 1>&2 + echo "$0: debug[3]: AUTHOR_PATH=$AUTHOR_PATH" 1>&2 + echo "$0: debug[3]: AUTHOR_DIR=$AUTHOR_DIR" 1>&2 + echo "$0: debug[3]: BIN_PATH=$BIN_PATH" 1>&2 + echo "$0: debug[3]: BIN_DIR=$BIN_DIR" 1>&2 + echo "$0: debug[3]: UNICODE_FIX_SED=$UNICODE_FIX_SED" 1>&2 + echo "$0: debug[3]: TOP_FILE=$TOP_FILE" 1>&2 +fi + + +# -N stops early before any processing is performed +# +if [[ -n $DO_NOT_PROCESS ]]; then + if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: arguments parsed, -N given, exiting 0" 1>&2 + fi + exit 0 +fi + + +# create a JSON problem file +# +export TMP_JSON_PROBLEM=".tmp.$NAME.JSON_PROBLEM.$$.tmp" +if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: JSON problem inventory file: $TMP_JSON_PROBLEM" 1>&2 +fi +trap 'rm -f $TMP_AUTHOR_HANDLE_INVENTORY; exit' 0 1 2 3 15 +rm -f "$TMP_JSON_PROBLEM" +if [[ -e $TMP_JSON_PROBLEM ]]; then + echo "$0: ERROR: cannot remove JSON problem inventory file: $TMP_JSON_PROBLEM" 1>&2 + exit 10 +fi +: > "$TMP_JSON_PROBLEM" +if [[ ! -e $TMP_JSON_PROBLEM ]]; then + echo "$0: ERROR: cannot create JSON problem inventory file: $TMP_JSON_PROBLEM" 1>&2 + exit 11 +fi + + +# case: check only file given as args +# +if [[ $# -gt 0 ]]; then + + # check specific JSON files + # + for JSON_FILE in "$@"; do + + # just be a readable file + # + if [[ ! -e $JSON_FILE ]]; then + echo "$0: ERROR: does not exist: $JSON_FILE" 1>&2 + exit 8 + fi + if [[ ! -f $JSON_FILE ]]; then + echo "$0: ERROR: is not a regular file: $JSON_FILE" 1>&2 + exit 8 + fi + if [[ ! -r $JSON_FILE ]]; then + echo "$0: ERROR: is not an readable file: $JSON_FILE" 1>&2 + exit 8 + fi + + # scan the author/author_handle.json file for unexpected \\u[[:xdigit:]]{4} strings + # + if [[ $V_FLAG -ge 1 ]]; then + echo "$0: debug[1]: about to run: sed -f $UNICODE_FIX_SED -- $JSON_FILE 2>&1 | grep -E ..." 1>&2 + fi + FOUND_UHEX=$(sed -f "$UNICODE_FIX_SED" -- "$JSON_FILE" 2>&1 | grep -E '\\u[[:xdigit:]]{4}') + if [[ -n $FOUND_UHEX ]]; then + echo "$JSON_FILE" + fi + + done >> "$TMP_JSON_PROBLEM" + + +# case: check all .entry.json files and author JSON files +# +else + + # check all .entry.json files + # + if [[ $V_FLAG -ge 1 ]]; then + echo "$0: debug[1]: starting to check .entry.json files under: $TOPDIR" 1>&2 + fi + export YYYY + for YYYY in $(< "$TOP_FILE"); do + + # debug YYYY + # + if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: starting to check year: $YYYY" 1>&2 + fi + + # verify that YYYY is a readable directory + # + if [[ ! -e $YYYY ]]; then + echo "$0: ERROR: YYYY does not exist: $YYYY" 1>&2 + EXIT_CODE="6" # exit 6 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + if [[ ! -d $YYYY ]]; then + echo "$0: ERROR: YYYY is not a directory: $YYYY" 1>&2 + EXIT_CODE="6" # exit 6 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + if [[ ! -r $YYYY ]]; then + echo "$0: ERROR: YYYY is not an readable directory: $YYYY" 1>&2 + EXIT_CODE="6" # exit 6 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + + # verify that YYYY has a non-empty readable .year file + # + export YEAR_FILE="$YYYY/.year" + if [[ ! -e $YEAR_FILE ]]; then + echo "$0: ERROR: YYYY/.year does not exist: $YEAR_FILE" 1>&2 + EXIT_CODE="6" # exit 6 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + if [[ ! -f $YEAR_FILE ]]; then + echo "$0: ERROR: YYYY/.year is not a regular file: $YEAR_FILE" 1>&2 + EXIT_CODE="6" # exit 6 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + if [[ ! -r $YEAR_FILE ]]; then + echo "$0: ERROR: YYYY/.year is not an readable file: $YEAR_FILE" 1>&2 + EXIT_CODE="6" # exit 6 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + if [[ ! -s $YEAR_FILE ]]; then + echo "$0: ERROR: YYYY/.year is not a non-empty readable file: $YEAR_FILE" 1>&2 + EXIT_CODE="6" # exit 6 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + + # process each entry directory under YYYY + # + export YYYY_DIR + for YYYY_DIR in $(< "$YEAR_FILE"); do + + # debug YYYY + # + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: starting to check year/dir: $YYYY_DIR" 1>&2 + fi + + # parse YYYY_DIR + # + if [[ ! -d $YYYY_DIR ]]; then + echo "$0: ERROR: YYYY_DIR is not a directory: $YYYY_DIR" 1>&2 + EXIT_CODE="6" # exit 6 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + if [[ ! -w $YYYY_DIR ]]; then + echo "$0: ERROR: YYYY_DIR is not a writable directory: $YYYY_DIR" 1>&2 + EXIT_CODE="6" # exit 6 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + export YEAR_DIR=${YYYY_DIR%%/*} + if [[ -z $YEAR_DIR ]]; then + echo "$0: ERROR: YYYY_DIR not in YYYY/dir form: $YYYY_DIR" 1>&2 + EXIT_CODE="6" # exit 6 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + export ENTRY_DIR=${YYYY_DIR#*/} + if [[ -z $ENTRY_DIR ]]; then + echo "$0: ERROR: YYYY_DIR not in $YEAR_DIR/dir form: $YYYY_DIR" 1>&2 + EXIT_CODE="6" # exit 6 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + if [[ $ENTRY_DIR = */* ]]; then + echo "$0: ERROR: YYYY_DIR: $YYYY_DIR dir contains a /: $ENTRY_DIR" 1>&2 + EXIT_CODE="6" # exit 6 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + + # verify that YYYY_DIR is a writable directory + # + if [[ ! -e $YYYY_DIR ]]; then + echo "$0: ERROR: YYYY_DIR does not exist: $YYYY_DIR" 1>&2 + EXIT_CODE="7" # exit 7 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + if [[ ! -d $YYYY_DIR ]]; then + echo "$0: ERROR: YYYY_DIR is not a directory: $YYYY_DIR" 1>&2 + EXIT_CODE="7" # exit 7 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + if [[ ! -w $YYYY_DIR ]]; then + echo "$0: ERROR: YYYY_DIR is not an writable directory: $YYYY_DIR" 1>&2 + EXIT_CODE="7" # exit 7 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + + # verify YYYY/dir/.path + # + export DOT_PATH="$YYYY_DIR/.path" + if [[ ! -s $DOT_PATH ]]; then + echo "$0: ERROR: not a non-empty file: $DOT_PATH" 1>&2 + EXIT_CODE="7" # exit 7 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + DOT_PATH_CONTENT=$(< "$DOT_PATH") + if [[ $YYYY_DIR != "$DOT_PATH_CONTENT" ]]; then + echo "$0: ERROR: arg: $YYYY_DIR does not match $DOT_PATH contents: $DOT_PATH_CONTENT" 1>&2 + EXIT_CODE="7" # exit 7 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + + # verify the YYYY/dir/.entry.json + # + export ENTRY_JSON="$YYYY_DIR/.entry.json" + if [[ ! -e $ENTRY_JSON ]]; then + echo "$0: ERROR: .entry.json does not exist: $ENTRY_JSON" 1>&2 + EXIT_CODE="7" # exit 7 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + if [[ ! -f $ENTRY_JSON ]]; then + echo "$0: ERROR: .entry.json is not a file: $ENTRY_JSON" 1>&2 + EXIT_CODE="7" # exit 7 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + if [[ ! -r $ENTRY_JSON ]]; then + echo "$0: ERROR: .entry.json is not a readable file: $ENTRY_JSON" 1>&2 + EXIT_CODE="7" # exit 7 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + continue + fi + + # scan the .entry.json file for unexpected \\u[[:xdigit:]]{4} strings + # + if [[ -z $NOOP ]]; then + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to run: sed -f $UNICODE_FIX_SED -- $YYYY_DIR/.entry.json 2>&1 | grep -E ..." 1>&2 + fi + FOUND_UHEX=$(sed -f "$UNICODE_FIX_SED" -- "$YYYY_DIR/.entry.json" 2>&1 | grep -E '\\u[[:xdigit:]]{4}' 2>&1) + export FOUND_UHEX + if [[ -n $FOUND_UHEX ]]; then + echo "$YYYY_DIR/.entry.json" + fi + fi + done + done >> "$TMP_JSON_PROBLEM" + + + # create a temporary author_handle inventory file + # + export TMP_AUTHOR_HANDLE_INVENTORY=".tmp.$NAME.AUTHOR_HANDLE_INVENTORY.$$.tmp" + if [[ $V_FLAG -ge 3 ]]; then + echo "$0: debug[3]: temporary author_handle inventory file: $TMP_AUTHOR_HANDLE_INVENTORY" 1>&2 + fi + trap 'rm -f $TMP_AUTHOR_HANDLE_INVENTORY $TMP_AUTHOR_HANDLE_INVENTORY; exit' 0 1 2 3 15 + rm -f "$TMP_AUTHOR_HANDLE_INVENTORY" + if [[ -e $TMP_AUTHOR_HANDLE_INVENTORY ]]; then + echo "$0: ERROR: cannot remove temporary author_handle inventory file: $TMP_AUTHOR_HANDLE_INVENTORY" 1>&2 + exit 12 + fi + : > "$TMP_AUTHOR_HANDLE_INVENTORY" + if [[ ! -e $TMP_AUTHOR_HANDLE_INVENTORY ]]; then + echo "$0: ERROR: cannot create temporary author_handle inventory file: $TMP_AUTHOR_HANDLE_INVENTORY" 1>&2 + exit 13 + fi + + # check author/author_handle.json files + # + if [[ $V_FLAG -ge 1 ]]; then + echo "$0: debug[1]: starting to check author_handle.json files under: $AUTHOR_DIR" 1>&2 + fi + find "$AUTHOR_DIR" -mindepth 1 -maxdepth 1 -type f -name '*.json' 2>/dev/null | + LC_ALL=C sort -d -u > "$TMP_AUTHOR_HANDLE_INVENTORY" + while read -r AUTHOR_HANDLE_JSON; do + + # scan the author/author_handle.json file for unexpected \\u[[:xdigit:]]{4} strings + # + if [[ $V_FLAG -ge 5 ]]; then + echo "$0: debug[5]: about to run: sed -f $UNICODE_FIX_SED -- $AUTHOR_HANDLE_JSON 2>&1 | grep -E ..." 1>&2 + fi + FOUND_UHEX=$(sed -f "$UNICODE_FIX_SED" -- "$AUTHOR_HANDLE_JSON" 2>&1 | grep -E '\\u[[:xdigit:]]{4}') + if [[ -n $FOUND_UHEX ]]; then + echo "$AUTHOR_HANDLE_JSON" + fi + + done < "$TMP_AUTHOR_HANDLE_INVENTORY" >> "$TMP_JSON_PROBLEM" +fi + + +# report any problems +# +if [[ -s $TMP_JSON_PROBLEM ]]; then + echo "$0: ERROR: found unexpected \\uHexHexHexHex strings in JSON files" 1>&2 + echo "$0: debug[5]: list of JSON files with unexpected \\uHexHexHexHex strings, starts below" 1>&2 + cat "$TMP_JSON_PROBLEM" 1>&2 + echo "$0: debug[5]: list of JSON files with unexpected \\uHexHexHexHex strings, ends above" 1>&2 + if [[ $EXIT_CODE -eq 0 ]]; then + EXIT_CODE="1" # exit 1 + echo "$0: Warning: EXIT_CODE set to: $EXIT_CODE" 1>&2 + fi +fi + + +# All Done!!! All Done!!! -- Jessica Noll, Age 2 +# +if [[ $EXIT_CODE -ne 0 ]]; then + echo "$0: Warning: about to exit non-zero: $EXIT_CODE" 1>&2 +elif [[ $V_FLAG -ge 1 ]]; then + echo "$0: debug[1]: about to exit $EXIT_CODE" 1>&2 +fi +exit "$EXIT_CODE" diff --git a/bin/unicode-fix.sed b/bin/unicode-fix.sed new file mode 100644 index 000000000..a04576555 --- /dev/null +++ b/bin/unicode-fix.sed @@ -0,0 +1,60 @@ +#!/usr/bin/env sed +# +# bin/unicode-fix.sed - work-a-round to pre-translate certain JSON /u codes +# +# Until the jsrdecode(1) is fixed so that it properly converts 1 or 2 consecutive +# JSON string \uHexHexHexHex values into the proper binary symbol, we pre-translate +# JSON string \uHexHexHexHex values used in this web site as a work-a-round to +# avoid the jsrdecode(1) bug. +# +# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX +# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX +# XXX - until we have fixed jstrdecode, we must FAKE PARSE IOCCC JSON files - XXX +# XXX - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - GROSS HACK - XXX +# XXX - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - XXX +# +# Until this bug is the jparse repo is fixed, we need to avoid +# using \\u[[:xdigit:]]{4} within JSON encoded strings. +# +# https://github.com/xexyl/jparse/issues/13 +# +# The above mentioned bug results in certain tools that are based on the +# jparse repo: +# +# https://github.com/xexyl/jparse +# +# including code from the mkiocccentry repo's jparse sub-directory: +# +# https://github.com/ioccc-src/mkiocccentry/tree/master/jparse +# +# We need to avoid using \\u[[:xdigit:]]{4} within JSON encoded strings. +# Sorry (tm Canada :-)). +# +# The following sed script converts known problematic \\u[[:xdigit:]]{4} +# combinations into their RAW character form: +# +# bin/unicode-fix.sed +# +# Use of that sed script serves as a work-a-round to avoid to avoid any +# problematic \\u[[:xdigit:]]{4} combinations in JSON encoded strings. +# +# For information on the jsrdecode(1) bug, see: +# +# https://github.com/xexyl/jparse/issues/13 +# +# usage: +# +# cmd | sed -f bin/unicode-fix.sed +# +# sed -f bin/unicode-fix.sed file > output +# +# sed -i -f bin/unicode-fix.sed file +# +# VERSION="1.0 2024-09-23" +# +s/\\u00c4/Ä/g +s/\\u00e1/á/g +s/\\u00eb/ë/g +s/\\u00fd/ý/g +s/\\u0159/ř/g +s/\\u2019/’/g