Merge pull request #1760 from xexyl/mills19

2019/mills/README.md

@@ -5,6 +5,18 @@ make
 ```
 
 
+### Bugs and (Mis)features:
+
+The current status of this entry is:
+
+```
+STATUS: INABIAF - please **DO NOT** fix
+```
+
+For more detailed information see [2019 mills in bugs.md](/bugs.md#2019-mills).
+
+
+
 ## To use:
 
 ```sh
@@ -17,17 +29,7 @@ make cpclean
 ### Try:
 
 ```sh
-make test-64bit
+./try.sh
-
-less IOCCC-Rules-Guidelines.output.txt
-
-less IOCCC-hints.output.txt
-
-less Eugene_Onegin.output.txt
-
-./prog < IOCCC-Rules-Guidelines.cp98_0.175 |head -n 100
-
-./prog < Shakespeare.cp04_1.633 | head -n 100
 ```
 
 However, as the binary model files used to produce the output are in an
@@ -51,25 +53,25 @@ You decide. :-)
 ### Welcome to OMLET! &#x1f373;:
 
 OMLET is the _Obfuscated Machine Learning Environment Toolkit_, a
-micro-framework for experimenting with [recurrent neural networks][1].  OMLET
+micro-framework for experimenting with [recurrent neural networks][1] (RNN).
-lets you build, train and evaluate [deep neural networks][2].  Why invest
+OMLET lets you build, train and evaluate [deep neural networks][2] (DNN).  Why
-hours reading documentation and megabytes of disk space on a full-featured DNN
+invest hours reading documentation and megabytes of disk space on a
-framework like [TensorFlow][3] or [Torch][4] when you can have full RNN
+full-featured DNN framework like [TensorFlow][3] or [Torch][4] when you can have
-functionality in less than 4 KB!
+full RNN functionality in less than 4 KB!
 
 OMLET has the following features:
 
-+ User-programmable network configurations and hyperparameters
++ User-programmable network configurations and hyperparameters.
 + Support for various types of recurrent and feed-forward neural networks
-including vanilla RNNs, LSTMs and GRUs of depths of up to 99 layers
+including vanilla RNNs, LSTMs and GRUs of depths of up to 99 layers.
-+ No limit on parameter size (except for those imposed by the system)
++ No limit on parameter size (except for those imposed by the system).
-+ Training and inference modes, with periodic checkpointing
++ Training and inference modes, with periodic checkpointing.
 + Advanced [Adam][42] optimizer with [weight decay][43] for simplified
-training
+training.
-+ Hyperparamer support for batch sizing, learning rate schedule, weight
++ Hyperparameters support for batch sizing, learning rate schedule, weight
-decay and gradient clipping
+decay and gradient clipping.
-+ Easily extensible (requires some expertise in the C programming language)
++ Easily extensible (requires some expertise in the C programming language).
-+ Friendly markdown documentation
++ Friendly markdown documentation.
 
 OMLET is based on [Andrej Karpathy's][40] [character-level language model][41]
 as described in his blog post [_The Unreasonable Effectiveness of Recurrent
@@ -84,12 +86,12 @@ training, but you can have even more fun by downloading some larger datasets:
 
 OMLET has three operating modes:
 
-  + _Training_, which trains a network from scratch
+  + _Training_, which trains a network from scratch.
-  + _Continuation_, which trains starting from an existing checkpoint
+  + _Continuation_, which trains starting from an existing checkpoint.
-  + _Inference_, which uses an existing trained network to make predictions
+  + _Inference_, which uses an existing trained network to make predictions.
 
 For your first OMLET experiment, we will try training a simple single-level
-RNN to write some [Shakespere][5] plays.  Start by typing
+RNN to write some [Shakespeare][5] plays.  Start by typing
 
 ```sh
 make
@@ -230,7 +232,7 @@ T1:0% 1.885619
 ```
 
 Validation cycles are used to test the network to see if it has learned to
-generalize -- how well performs on data it hasn't seen before (as opposed to
+generalize -- how well it performs on data it hasn't seen before (as opposed to
 the training data that the network will see many times as it trains).  Progress
 on the validation set is also displayed with a validation progress report that
 looks like
@@ -286,6 +288,7 @@ will want to delete all the checkpoint files because the format depends on
 both the network and the input -- using the wrong checkpoint is likely to
 cause a crash.
 
+
 ### Experimenting with different networks
 
 The default network for OMLET (the one you get if you type `make` with the
@@ -305,7 +308,7 @@ where
   * `h'` is the previous value of `h`
   * `Wxh`, `Whh`, and `Why` are *weight matrices*
   * `Bh` and `By` are *bias vectors*.
-  * `tanh` is the [hyperbolic tangent function][9]
+  * `tanh(3)` is the [hyperbolic tangent function][9]
 
 The `W`'s and `B`'s are the trainable parameters of the network, and the
 process of training is optimizing the values of these parameters to minimize
@@ -325,7 +328,7 @@ OMLET will create an [ADALINE][10] network that does
 y = Wxy * x + By
 ```
 
-This simple [linear][11] [feed-forward network][12].  You can run it with
+This is a simple [linear][11] [feed-forward network][12].  You can run it with
 
 ```sh
 ./lin1 Shakespeare.txt
@@ -369,15 +372,17 @@ It is able to guess at what character is likely to follow the current one
 (by doing a [linear regression][13]), but it lacks any history beyond that to
 guide it.
 
-You might be wondering about the role of the `tanh` function in the RNN.
+You might be wondering about the role of the `tanh(3)` function in the RNN.
-`tanh` acts as an [activation function][14] which adds [nonlinearity][15]
+`tanh(3)` acts as an [activation function][14] which adds [nonlinearity][15]
 to the network and allows it to solve complicated problems.  Without
 nonlinearity, all of the linear functions would fold together into a single
 matrix-vector multiply and you'd effectively regress to the linear network
 above.  Alas, even adding a nonlinearity to the feed-forward network
 (creating a [perceptron][16]) does not improve the performance because we
 still lack the history provided by the hidden state vector (although if you
-want to try it yourself, you can do so with `make per1`).
+want to try it yourself, you can do so with `make per1` - note that's not `perl`
+the language but `per` with the digit 1).
+
 
 ### Going deeper:
 
@@ -392,17 +397,21 @@ y = Why * h2 + By
 
 with `RNN(h, x)` defined as above.  Each RNN module has its own set of
 parameters and its own hidden state vector.  This will improve the network's
-performance, at the cost of a much larger parameter space
+performance, at the cost of a much larger parameter space.
 
-> **IMPORTANT NOTE**: Since OMLET uses the system stack for network storage,
+---
-> larger networks may cause OMLET to crash (typically with a message like
+
-> `Segmentation fault`) unless the system stack size is first increased.
+**IMPORTANT NOTE**: Since OMLET uses the system stack for network storage,
-> The exact command for doing so depends on your shell and your system's
+larger networks may cause OMLET to crash (typically with a message like
-> hard limits.  On `sh`/`ksh`/`bash` shells, you can view the hard limit
+`Segmentation fault`) unless the system stack size is first increased.
-> with `ulimit -Hs` and set it with `ulimit -s 65532` (replacing `65532`
+The exact command for doing so depends on your shell and your system's
-> with the actual hard limit).  On `csh`/`tcsh` shells, you can view
+hard limits.  On `sh`/`ksh`/`bash` shells, you can view the hard limit
-> the hard limit with `limit -h stacksize` and set it with
+with `ulimit -Hs` and set it with `ulimit -s 65532` (replacing `65532`
-> `limit stacksize 65532` (replacing `65532` with the actual hard limit).
+with the actual hard limit).  On `csh`/`tcsh` shells, you can view
+the hard limit with `limit -h stacksize` and set it with
+`limit stacksize 65532` (replacing `65532` with the actual hard limit).
+
+---
 
 You can try the deeper network by doing
 
@@ -418,14 +427,15 @@ make rnn2
 
 The additional depth should allow the network to make better predictions (it
 can represent more complicated history), but it may take a long time to train
--- both because the network (being larger) now requires more time to train and
+\-\- both because the network (being larger) now requires more time to train and
 because of the [vanishing and exploding gradient problem][17], which might
 keep it from ever reaching its potential.
 
+
 ### LSTMs and GRUs
 
 RNNs are particularly hard to train because they are trained using
-[bankpropagation through time][18].  The RNN is trained by effectively
+[backpropagation through time][18].  The RNN is trained by effectively
 converting it into a non-recurrent network by making many copies of it
 and propagating the hidden state through the copies.  During training,
 the backpropagation through many clones of the network amplifies the
@@ -478,7 +488,7 @@ Where
   * `Wxf`, `Whf`, `Wxi`, `Whi`, `Wxo`, `Who`, `Wxc`, and `Whc` are trainable
     parameter matrices
   * `Bf`, `Bi`, `Bo`, and `Bc` are trainable bias vectors
-  * `tanh` is the hyperbolic tangent function
+  * `tanh(3)` is the hyperbolic tangent function
   * `sigmoid` is the [logistic function][21]
 
 There are several LSTM variants (see [C. Olah's blog post][20] for more
@@ -502,7 +512,7 @@ a `-DNW='...'` command which consists of a series of comma-separated
 assignments.  For example, the simple one-layer RNN could be defined like
 
 ```make
--DNW=` x  = I(n), hp = I(128),                  \
+-DNW=' x  = I(n), hp = I(128),                  \
        h  = C(hp, T(A(L(128, x), L(128, hp)))), \
        y  = L(n, h)'
 ```
@@ -527,7 +537,7 @@ takes two parameters -- the output vector size (which must match the size of
 has its own `W` (weight) and `B` (bias) training parameters.  Both `x` and
 `hp` are sent through `L` and the result passed through the `A` function,
 which does vector addition.  That result is passed through `T` which does
-element-wise `tanh` activation.
+element-wise `tanh(3)` activation.
 
 Next, we wrap the whole thing with the `C` function.  `C` connects `hp`
 with `h`, causing the new value of `h` to be passed to the `hp` vector on
@@ -595,6 +605,7 @@ Note: even if you don't use `MD` in your network, you should still define
 `BK` by adding `-DBK='y=x'` to the command line, otherwise you will get a
 compile-time error.
 
+
 ### Hyperparameters
 
 OMLET has a large number of training and inference parameters which can be
@@ -606,7 +617,7 @@ The list of hyperparameters follows:
     safer but more boring choices.  A high temperature takes more risks but
     makes more mistakes.  Default is 1.0, which uses the computed
     probabilities.
-  * 'N': Batch size.  This is the number of times the RNN is unrolled, so it
+  * `N`: Batch size.  This is the number of times the RNN is unrolled, so it
     controls how far back in the past the RNN can see.  The default is 50.
     Larger batch sizes update the weights less frequently and allow the RNN to
     see farther back in time, but at a cost of proportionally more memory.
@@ -668,7 +679,7 @@ The list of hyperparameters follows:
  [31]: https://cs.stanford.edu/people/karpathy/char-rnn/warpeace_input.txt
  [32]: https://github.com/ryanmcdermott/trump-speeches
 
- [40]: https://cs.stanford.edu/people/karpathy/
+ [40]: https://karpathy.ai
  [41]: https://github.com/karpathy/char-rnn
  [42]: https://arxiv.org/abs/1412.6980
  [43]: https://www.fast.ai/2018/07/02/adam-weight-decay/

2019/mills/try.sh

@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+#
+# try.sh - demonstrate IOCCC winner 2019/mills
+#
+
+# make sure CC is set so that when we do make CC="$CC" it isn't empty. Doing it
+# this way allows us to have the user specify a different compiler in an easy
+# way.
+if [[ -z "$CC" ]]; then
+    CC="cc"
+fi
+
+make clobber CC="$CC" all >/dev/null || exit 1
+
+
+read -r -n 1 -p "Press any key to run: make test-64bit: "
+echo 1>&2
+echo "This may take a while:"
+make test-64bit
+
+
+echo "$ less IOCCC-Rules-Guidelines.output.txt" 1>&2
+read -r -n 1 -p "Press any key to continue (space = next page, q = quit): "
+echo 1>&2
+less -rEXF IOCCC-Rules-Guidelines.output.txt
+echo 1>&2
+
+echo "$ less IOCCC-hints.output.txt" 1>&2
+read -r -n 1 -p "Press any key to continue (space = next page, q = quit): "
+echo 1>&2
+less -rEXF IOCCC-hints.output.txt
+echo 1>&2
+
+echo "$ less Eugene_Onegin.output.txt" 1>&2
+echo "You should be able to safely ignore the warnings about this file" 1>&2
+echo "being a binary file." 1>&2
+read -r -n 1 -p "Press any key to continue (space = next page, q = quit): "
+echo 1>&2
+less -rEXF Eugene_Onegin.output.txt
+echo 1>&2
+
+echo "$ ./prog < IOCCC-Rules-Guidelines.cp98_0.175 |head -n 100"
+read -r -n 1 -p "Press any key to continue (space = next page, q = quit): "
+echo 1>&2
+./prog < IOCCC-Rules-Guidelines.cp98_0.175 |head -n 100
+echo 1>&2
+
+echo "$ ./prog < Shakespeare.cp04_1.633 | head -n 100" 1>&2
+read -r -n 1 -p "Press any key to continue (space = next page, q = quit): "
+echo 1>&2
+./prog < Shakespeare.cp04_1.633 | head -n 100
+echo 1>&2

bugs.md

@@ -3504,6 +3504,34 @@ remarks in the sections [Syntax](/2019/lynn/README.md#syntax) and
 [Caveats](/2019/lynn/README.md#caveats).
 
 
+## 2019 mills
+
+### STATUS: INABIAF - please **DO NOT** fix
+### Source code: [2019/mills/prog.c](2019/mills/prog.c)
+### Information: [2019/mills/README.md](2019/mills/README.md)
+
+The author wrote that if you decide to change networks or use a different input
+file you should delete all the checkpoint files because the format depends on
+both the network and the input -- using the wrong checkpoint is likely to cause
+a crash.
+
+The author also wrote:
+
+---
+
+**IMPORTANT NOTE**: Since OMLET uses the system stack for network storage,
+larger networks may cause OMLET to crash (typically with a message like
+`Segmentation fault`) unless the system stack size is first increased.
+The exact command for doing so depends on your shell and your system's
+hard limits.  On `sh`/`ksh`/`bash` shells, you can view the hard limit
+with `ulimit -Hs` and set it with `ulimit -s 65532` (replacing `65532`
+with the actual hard limit).  On `csh`/`tcsh` shells, you can view
+the hard limit with `limit -h stacksize` and set it with
+`limit stacksize 65532` (replacing `65532` with the actual hard limit).
+
+---
+
+
 ## 2019 poikola
 
 ### STATUS: INABIAF - please **DO NOT** fix

thanks-for-help.md

@@ -4243,6 +4243,11 @@ Cody also added the [example-1.txt](/2019/lynn/example-1.txt) and
 entry existing.
 
 
+## <a name="2019_mills"></a>[2019/mills](/2019/mills/prog.c) ([README.md](/2019/mills/README.md]))
+
+[Cody](#cody) added the [try.sh](/2019/mills/try.sh) script.
+
+
 ## <a name="2019_poikola"></a>[2019/poikola](/2019/poikola/prog.c) ([README.md[(/2019/poikola/README.md))
 
 [Cody](#cody) added the missing `docs` rule to the Makefile that forms a PDF