use jsp tool, workaround JSON Unicode bugs, misc fixes

If the [jsp tool](https://github.com/kjozsa/jsp) is
installed, it will be used in favor of the [JSONPath.sh
tool](https://github.com/lcn2/JSONPath.sh).

Updated `bin/jfmt-wrapper.sh`, `bin/jprint-wrapper.sh`, and
`bin/jval-wrapper.sh` to use jsp if found.  These 3 tools may be given
`-t tool` where tool is either "_jsp_" or "_JSONPath.sh_" for force the
use of the tool.  They also use `-Q` to perform a quick check tool using
trivial input.

The `bin/jval-wrapper.sh` tool had been simplified.  The `-w`,
`-i` and `-u` options are gone.  Instead it has:

```
    -b    print only values (def: print both JSON member and JSON value)
    -q    remove enclosing double quotes (def: keep any enclosing double quotes)
```

Add `make all_jfmt` to Makefile to canonically format all entry and
author handle JSON files.

A sed script `bin/unicode-fix.sed` has been added that converts
JSON string "_\u[0-9A-Fa-f]{4}_" encoding that may arise from known
JSON data.  This is done as a work-a-round for [Unicode / UTF-8
issues](https://github.com/xexyl/jparse/issues/13) with `jstrdecode(1)`.

The `bin/unicode-chk.sh` tool, if given on file arguments, scans
all `.entry.json` and `author/author_handle.json` JSON files to be
sure that, after `bin/unicode-fix.sed` processing, no JSON string
"_\u[0-9A-Fa-f]{4}_" encodings remain.

Removed all `-j jparse` args from commands.  Instead the `jparse(1)`
tool is looked for along $PATH.

Added `combine_author_handle.sh` tool to transform all of the
`author/author_handle.json` files into a temporary as a single JSON
file so that `bin/jval-wrapper.sh` tool may be used to extract all
"_sort_word_" or all "_entry_id_" values.

Changed `bin/all-jfmt.sh` VERSION from "1.0 2024-08-31"
to "1.1 2024-09-24".

Changed `bin/csv2entry.sh` VERSION from "1.2.3 2024-08-30"
to "1.3 2024-09-23".

Changed `bin/cvt-submission.sh` VERSION from "1.1 2024-08-31"
to "1.2 2024-09-23".

Changed `bin/entry2csv.sh` VERSION from "1.2.1 2024-08-17"
to "1.3 2024-09-23".

Changed `bin/gen-authors.sh` VERSION from "1.12 2024-08-18"
to "1.13 2024-09-23".

Changed `bin/gen-location.sh` VERSION from "1.6 2024-08-18"
to "1.7 2024-09-23".

Changed `bin/gen-status.sh` VERSION from "1.6 2024-08-17"
to "1.7 2024-09-23".

Changed `bin/gen-years.sh` VERSION from "1.4 2024-08-17"
to "1.5 2024-09-23".

Changed `bin/jfmt-wrapper.sh` VERSION from "1.1 2024-08-20"
to "1.2 2024-09-23".

Changed `bin/jprint-wrapper.sh` VERSION from "1.1 2024-08-20"
to "1.3 2024-09-23".

Changed `bin/jval-wrapper.sh` VERSION from "1.0.4 2024-08-20"
to "1.1 2024-09-23".

Changed `bin/manifest.entry.json.awk` VERSION from "1.1.2 2024-07-28"
to "1.2 2024-09-17".

Changed `bin/output-index-author.sh` VERSION from "1.8 2024-08-17"
to "1.9 2024-09-23".

Changed `bin/output-year-index.sh` VERSION from "1.4.4 2024-08-05"
to "1.5 2024-09-23".

Changed `bin/subst.entry-index.sh` VERSION from "1.5 2024-08-13"
to "1.6 2024-09-23".

Added `bin/combine_author_handle.sh` VERSION "1.0 2024-09-23".

Added `bin/unicode-chk.sh` VERSION "1.0 2024-09-23".

Added `bin/unicode-fix.sed` VERSION "1.0 2024-09-23".

Perform `make www` to perform the above and test under macOS.

FYI: On our MacBook Pro, `make www` took:

```
    make www  631.79s user 264.95s system 94% cpu 15:53.71 total
```

FYI: On our MacBook Pro, `make quick_www` took:

```
    make quick_www  81.43s user 58.26s system 102% cpu 2:16.02 total
```
This commit is contained in:
Landon Curt Noll committed 2024-09-24 04:04:27 -07:00
1 parent e84d6eb095
commit 499be5983d
19 files changed
+3044 -1213

No files matched your search

+60 -22
View File
@@ -23,7 +23,7 @@ BEGIN {
# setup
#
VERSION="1.1.2 2024-07-28"
VERSION="1.2 2024-09-17"
found_manifest_array = 0; # 1 ==> we found the manifest ARRAY
within_manifest_array = 0; # 1 ==> found start of "manifest" : [ JSON array
begin_manifest_element = 0; # 1 ==> we have found the start of the manifest array element
@@ -121,7 +121,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^"
#
$1 = ""; # remove 1st field
$2 = ""; # remove 2nd field
gsub(/^[[:space:]]+/, "", $0); # any leading whitespace
gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace
file_path = $0;
# Remove any trailing ,
@@ -155,7 +155,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^"
#
$1 = ""; # remove 1st field
$2 = ""; # remove 2nd field
gsub(/^[[:space:]]+/, "", $0); # any leading whitespace
gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace
inventory_order = $0;
# Remove any trailing ,
@@ -189,7 +189,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^"
#
$1 = ""; # remove 1st field
$2 = ""; # remove 2nd field
gsub(/^[[:space:]]+/, "", $0); # any leading whitespace
gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace
OK_to_edit = $0;
# Remove any trailing ,
@@ -223,7 +223,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^"
#
$1 = ""; # remove 1st field
$2 = ""; # remove 2nd field
gsub(/^[[:space:]]+/, "", $0); # any leading whitespace
gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace
display_as = $0;
# Remove any trailing ,
@@ -257,7 +257,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^"
#
$1 = ""; # remove 1st field
$2 = ""; # remove 2nd field
gsub(/^[[:space:]]+/, "", $0); # any leading whitespace
gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace
display_via_github = $0;
# Remove any trailing ,
@@ -291,7 +291,7 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^"
#
$1 = ""; # remove 1st field
$2 = ""; # remove 2nd field
gsub(/^[[:space:]]+/, "", $0); # any leading whitespace
gsub(/^[[:space:]]+/, "", $0); # remove any leading whitespace
entry_text = $0;
# Remove any trailing ,
@@ -300,12 +300,50 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF >= 3 && $1 ~ /^"
entry_text = substr(entry_text, 1, length(entry_text)-1);
}
# so we can single quote the entry_text as a single string, we protect any single quotes within it
#
jstrdecode_arg = entry_text;
gsub(/'/, "'\"'\"'", jstrdecode_arg);
# form the jstrdecode command
#
if (jstrdecode_arg ~ /^".*"$/) {
jstrdecode_cmd = "jstrdecode -Q -- " "'" substr(jstrdecode_arg, 2, length(jstrdecode_arg)-2) "'";
} else {
jstrdecode_cmd = "jstrdecode -Q -- " "'" jstrdecode_arg "'";
}
jstrdecode_cmd = jstrdecode_cmd " 2>/dev/null; echo $?";
# JSON decode entry_text
#
numLines = 0;
result = "";
exit_code = 0;
# process each pipe line
while ((jstrdecode_cmd | getline pipe_output) > 0) {
if (numLines++ == 0) {
# case: 1st line is from jstrdecode
result = pipe_output;
} else {
# case: 2nd/line line is the jstrdecode exit code
exit_code = pipe_output;
}
}
close(jstrdecode_cmd);
# validate exit code from jstrdecode
#
if (length(exit_code) == 0 || exit_code != 0) {
print "ERROR: unable to jstrdecode entry_text:", entry_text > "/dev/stderr";
exit 213; # END section will output ERROR message about cannot JSON decode entry_text
}
# remove any enclosing double quotes
#
if (entry_text ~ /^".*"$/) {
unquote_entry_text = substr(entry_text, 2, length(entry_text)-2);
if (result ~ /^".*"$/) {
unquote_entry_text = substr(result, 2, length(result)-2);
} else {
unquote_entry_text = entry_text;
unquote_entry_text = result;
}
}
@@ -317,42 +355,42 @@ within_manifest_array == 1 && begin_manifest_element == 1 && NF == 1 && $1 ~ /^}
#
if (length(file_path) == 0) {
print "ERROR: manifest array JSON member did not have a file_path JSON member in:", ARGV[1] > "/dev/stderr";
exit 213;
exit 214;
}
# verify we have seen a inventory_order JSON member
#
if (length(inventory_order) == 0) {
print "ERROR: manifest array JSON member did not have a inventory_order JSON member in:", ARGV[1] > "/dev/stderr";
exit 214;
exit 215;
}
# verify we have seen a OK_to_edit JSON member
#
if (length(OK_to_edit) == 0) {
print "ERROR: manifest array JSON member did not have a OK_to_edit JSON member in:", ARGV[1] > "/dev/stderr";
exit 215;
exit 216;
}
# verify we have seen a display_as JSON member
#
if (length(display_as) == 0) {
print "ERROR: manifest array JSON member did not have a display_as JSON member in:", ARGV[1] > "/dev/stderr";
exit 216;
exit 217;
}
# verify we have seen a display_via_github JSON member
#
if (length(display_via_github) == 0) {
print "ERROR: manifest array JSON member did not have a display_via_github JSON member in:", ARGV[1] > "/dev/stderr";
exit 217;
exit 218;
}
# verify we have seen a entry_text JSON member
#
if (length(entry_text) == 0) {
print "ERROR: manifest array JSON member did not have a entry_text JSON member in:", ARGV[1] > "/dev/stderr";
exit 218;
exit 219;
}
# print manifest information
@@ -395,39 +433,39 @@ END {
#
if (length(github) == 0) {
print "ERROR: github variable not set, call with -v github=REPO_URL" > "/dev/stderr";
exit 219; # use 200 to match exit code from BEGIN
exit 220; # use 200 to match exit code from BEGIN
}
# case: we quit because we could not determine dir from ARGV[1]
#
if (length(dir) == 0) {
print "ERROR: cannot determine dir from:", ARGV[1] > "/dev/stderr";
exit 220; # END section will output ERROR message about cannot find dir
exit 221; # END section will output ERROR message about cannot find dir
}
if (length(YYYY) == 0) {
print "ERROR: cannot determine YYYY from:", ARGV[1] > "/dev/stderr";
exit 221; # END section will output ERROR message about cannot find YYYY
exit 222; # END section will output ERROR message about cannot find YYYY
}
# case: no manifest array was found
#
if (!found_manifest_array) {
print "ERROR: we did not find a manifest JSON array in:", ARGV[1] > "/dev/stderr";
exit 222;
exit 223;
}
# case: manifest array was found, but manifest array element did not finish
#
if (begin_manifest_element) {
print "ERROR: manifest JSON array element did not end in:", ARGV[1] > "/dev/stderr";
exit 223;
exit 224;
}
# case: manifest array did not end
#
if (within_manifest_array) {
print "ERROR: manifest JSON array did not end in:", ARGV[1] > "/dev/stderr";
exit 224;
exit 225;
}
}