mirror/ioccc
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fixes in 2019/mills

Browse Source 
Format fixes and typo fixes in README.md. Some of the typo fixes were
very subtle (like a ' outside of `` where it should be in) but I
probably missed some as there are many terms I'm unfamiliar with. I'll
have to go through this file again but it's a good start.

Add to .gitignore file name referenced README.md.

I believe this completes the first pass of formatting and typos in 2019
entries (I haven't really looked at the year README.md files).
This commit is contained in:
Cody Boone Ferguson committed 2023-04-06 09:05:07 -07:00
1 parent 437329f18c
commit 03bbc129a5
2 files changed
+32 -31
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

2019/mills/.gitignore
+1
View File
@@ -20,3 +20,4 @@ rlstm3
rnn1 rnn1
rnn2 rnn2
rnn3 rnn3
IOCCC-hints.output.txt
2019/mills/README.md
+31 -31
View File
@@ -1,7 +1,7 @@
# Most in need to be tweeted # Most in need to be tweeted
Christopher Mills Christopher Mills
Twitter: @MisterXopher Twitter: @MisterXopher
## To build: ## To build:
@@ -30,7 +30,8 @@ less IOCCC-hints.output.txt
less Eugene_Onegin.output.txt less Eugene_Onegin.output.txt
``` ```
However, as the binary model files used to produce the output are in an implementation-specific format, your mileage may vary. However, as the binary model files used to produce the output are in an
implementation-specific format, your mileage may vary.
## Judges' comments: ## Judges' comments:
@@ -39,7 +40,6 @@ Can a machine learn?
Some say so. Some say so.
But can a machine learn to write like Shakespeare, But can a machine learn to write like Shakespeare,
or tweet like [Trump](http://trumptwitterarchive.com)? or tweet like [Trump](http://trumptwitterarchive.com)?
Can it write rules and guidelines for the IOCCC? Can it write rules and guidelines for the IOCCC?
@@ -48,7 +48,7 @@ You decide. :-)
## Author's comments: ## Author's comments:
### Welcome to OMLET! 🍳 ### Welcome to OMLET! 🍳:
OMLET is the _Obfuscated Machine Learning Environment Toolkit_, a OMLET is the _Obfuscated Machine Learning Environment Toolkit_, a
micro-framework for experimenting with [recurrent neural networks][1]. OMLET micro-framework for experimenting with [recurrent neural networks][1]. OMLET
@@ -59,17 +59,17 @@ functionality in less than 4 KB!
OMLET has the following features: OMLET has the following features:
+ User-programmable network configurations and hyperparameters + User-programmable network configurations and hyperparameters
+ Support for various types of recurrent and feed-forward neural networks + Support for various types of recurrent and feed-forward neural networks
including vanilla RNNs, LSTMs and GRUs of depths of up to 99 layers including vanilla RNNs, LSTMs and GRUs of depths of up to 99 layers
+ No limit on parameter size (except for those imposed by the system) + No limit on parameter size (except for those imposed by the system)
+ Training and inference modes, with periodic checkpointing + Training and inference modes, with periodic checkpointing
+ Advanced [Adam][42] optimizer with [weight decay][43] for simplified + Advanced [Adam][42] optimizer with [weight decay][43] for simplified
training training
+ Hyperparamer support for batch sizing, learning rate schedule, weight + Hyperparamer support for batch sizing, learning rate schedule, weight
decay and gradient clipping decay and gradient clipping
+ Easily extensible (requires some expertise in the C programming language) + Easily extensible (requires some expertise in the C programming language)
+ Friendly markdown documentation + Friendly markdown documentation
OMLET is based on [Andrej Karpathy's][40] [character-level language model][41] OMLET is based on [Andrej Karpathy's][40] [character-level language model][41]
as described in his blog post [_The Unreasonable Effectiveness of Recurrent as described in his blog post [_The Unreasonable Effectiveness of Recurrent
@@ -98,13 +98,13 @@ make
After it builds, train it using After it builds, train it using
```sh ```sh
./prog shakespere.txt ./prog Shakespere.txt
``` ```
This will immediately start outputting gibberish to the output, e.g. This will immediately start outputting gibberish to the output, e.g.
```sh ```sh
./prog shakespere.txt ./prog Shakespere.txt
``` ```
produces: produces:
@@ -178,7 +178,7 @@ reduce and the generated snippets will improve quickly:
He brich nom dove worthan then wise, He brich nom dove worthan then wise,
T0:13% 2.254926 T0:13% 2.254926
It's already started to figure out things about Shakespere plays -- how to It's already started to figure out things about Shakespeare plays -- how to
spell short English words, how long lines tend to be, and that characters take spell short English words, how long lines tend to be, and that characters take
turns speaking with their names capitalized. The training loss has dropped to turns speaking with their names capitalized. The training loss has dropped to
2.25 and the improvement is noticeable. 2.25 and the improvement is noticeable.
@@ -235,7 +235,7 @@ training loss will give you an idea of how well the network is learning and
can let you know if the network is [overfitting or underfitting][7]. can let you know if the network is [overfitting or underfitting][7].
As part of the training process, the data set (which for OMLET is the As part of the training process, the data set (which for OMLET is the
`shakespere.txt` file you gave on the command line) is divided into [Shakespeare.txt](Shakespeare.txt) file you gave on the command line) is divided into
[training and validation sets][8] (by default, 95% of the data is used for [training and validation sets][8] (by default, 95% of the data is used for
training, but like most OMLET parameters, you can change this at compile-time). training, but like most OMLET parameters, you can change this at compile-time).
@@ -249,7 +249,7 @@ You can continue training from a previous checkpoint by providing the
checkpoint file name as the second parameter, for example: checkpoint file name as the second parameter, for example:
```sh ```sh
./prog shakespere.txt cp01_1.970 ./prog Shakespeare.txt cp01_1.970
``` ```
After the validation cycle finishes, OMLET begins the next epoch by restarting After the validation cycle finishes, OMLET begins the next epoch by restarting
@@ -318,14 +318,14 @@ y = Wxy * x + By
This simple [linear][11] [feed-forward network][12]. You can run it with This simple [linear][11] [feed-forward network][12]. You can run it with
```sh ```sh
./lin1 shakespeare.txt ./lin1 Shakespeare.txt
``` ```
The linear network won't be able to get past the gibberish stage, because it The linear network won't be able to get past the gibberish stage, because it
lacks history: lacks history:
```sh ```sh
./lin1 shakespeare.txt ./lin1 Shakespeare.txt
``` ```
produces: produces:
@@ -367,7 +367,7 @@ above. Alas, even adding a nonlinearity to the feed-forward network
still lack the history provided by the hidden state vector (although if you still lack the history provided by the hidden state vector (although if you
want to try it yourself, you can do so with `make per1`). want to try it yourself, you can do so with `make per1`).
### Going deeper ### Going deeper:
We can try to improve the RNN's performance by stacking RNN modules atop each We can try to improve the RNN's performance by stacking RNN modules atop each
other: other:
@@ -401,7 +401,7 @@ make rnn2
(or even `make rnn3` if you want a three-layer RNN) and train it with (or even `make rnn3` if you want a three-layer RNN) and train it with
```sh ```sh
./rnn2 shakespeare.txt ./rnn2 Shakespeare.txt
``` ```
The additional depth should allow the network to make better predictions (it The additional depth should allow the network to make better predictions (it
@@ -412,7 +412,7 @@ keep it from ever reaching its potential.
### LSTMs and GRUs ### LSTMs and GRUs
RNNs are particularly hard to train because the they are trained using RNNs are particularly hard to train because they are trained using
[bankpropagation through time][18]. The RNN is trained by effectively [bankpropagation through time][18]. The RNN is trained by effectively
converting it into a non-recurrent network by making many copies of it converting it into a non-recurrent network by making many copies of it
and propagating the hidden state through the copies. During training, and propagating the hidden state through the copies. During training,
@@ -430,7 +430,7 @@ make lstm2
and train with it with and train with it with
```sh ```sh
./lstm2 shakespeare.txt ./lstm2 Shakespeare.txt
``` ```
The LSTM is much easier to train because it explicitly decides how to update The LSTM is much easier to train because it explicitly decides how to update
@@ -456,7 +456,7 @@ Where
* `x` is the input vector * `x` is the input vector
* `h` is the hidden state (and the output to the next layer) * `h` is the hidden state (and the output to the next layer)
* `c` is the _cell state_ which represents the "memory" of the LSTM * `c` is the _cell state_ which represents the "memory" of the LSTM
* `h'` and `c`' are the previous values of `h` and `c` respectively * `h'` and `c'` are the previous values of `h` and `c` respectively
* `f` is the _forget gate_ that tells the LSTM what portion of the hidden * `f` is the _forget gate_ that tells the LSTM what portion of the hidden
state to forget state to forget
* `i` is the _input gate_ that tells the LSTM what portion of the input * `i` is the _input gate_ that tells the LSTM what portion of the input
@@ -548,7 +548,7 @@ GRU network:
``` ```
We are using a few new tricks here -- first, we are defining `HS` as the size We are using a few new tricks here -- first, we are defining `HS` as the size
of the hidden and cell vectors. There's nothing special about this name, its of the hidden and cell vectors. There's nothing special about this name, it's
just convenient to specify it so we don't have a bunch of constants in the just convenient to specify it so we don't have a bunch of constants in the
code. Second, the network itself is very simple -- it declares `x` and has code. Second, the network itself is very simple -- it declares `x` and has
the matrix that converts the `HS`-sized hidden vector back to the `n`-sized the matrix that converts the `HS`-sized hidden vector back to the `n`-sized
@@ -600,12 +600,12 @@ The list of hyperparameters follows:
see farther back in time, but at a cost of proportionally more memory. see farther back in time, but at a cost of proportionally more memory.
* `TR`: The percentage of batches in the input data set that will be used * `TR`: The percentage of batches in the input data set that will be used
for training. The default value of 0.95 sets this as 95%. for training. The default value of 0.95 sets this as 95%.
* `LR`: The initial learning rate, The default is 0.002. * `LR`: The initial learning rate, the default is 0.002.
* `LE`: The epoch where the learning rate will start decaying. Defaults to * `LE`: The epoch where the learning rate will start decaying. Defaults to
epoch 10. epoch 10.
* `LD`: Learning rate decay, per epoch (after `LE` epochs). The learning * `LD`: Learning rate decay, per epoch (after `LE` epochs). The learning
rate is scaled by this number. Default is 0.97 rate is scaled by this number. Default is 0.97
* `WD`: Weight-decay parameter, to promote [regularizaiton][27]. The default * `WD`: Weight-decay parameter, to promote [regularization][27]. The default
is 0.00008. is 0.00008.
* `RS`: The random scale for weight initialization. Weight parameters will * `RS`: The random scale for weight initialization. Weight parameters will
be initialized to be between `-RS` to `+RS`. The default is 0.15. be initialized to be between `-RS` to `+RS`. The default is 0.15.