mirror/ioccc
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

improved indentation and whitespace for markdown code blocks

Browse Source 
All code blocks in markdown have been intended by at least 4 spaces.

We add `--tab-stop=8` to all `pandoc(1)` executions in order to set
a better tab stop indentation, especially for tab intended code blocks.

Changed `bin/pandoc-wrapper.sh` VERSION from "1.5.3 2024-03-26"
to "1.6 2024-03-31".

Fixed `1991/brnstnd/sorta.README.md` so that the sorta code
is code blocks.

Removed unneeded paragraph group in copyright in `inc/footer.default.html`.

Fixed excess whitespace after list Author on entry `index.html` page.

Changed `bin/output-index-author.sh` VERSION from "1.5.2 2024-03-26"
to "1.6 2024-03-31".

Performed `make www` to perform the above and test under macOS.

TODO: Fix the problem of the initial line of code being extra intended.
TODO: Fix excess whitespace around code that is inline.
This commit is contained in:
Landon Curt Noll committed 2024-04-01 04:48:48 -07:00
1 parent b84c27ce02
commit a996f8d2b8
734 files changed
+16723 -16963
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

2019/mills/README.md
+141 -141
View File
@@ -1,7 +1,7 @@
## To build:
```sh
make
make
```
@@ -10,7 +10,7 @@ make
The current status of this entry is:
```
STATUS: INABIAF - please **DO NOT** fix
STATUS: INABIAF - please **DO NOT** fix
```
For more detailed information see [2019 mills bugs](../../bugs.html#2019_mills).
@@ -19,16 +19,16 @@ For more detailed information see [2019 mills bugs](../../bugs.html#2019_mills).
## To use:
```sh
make cpclean
# Let this run for about about an hour and then kill it:
./prog Shakespeare.txt
make cpclean
# Let this run for about about an hour and then kill it:
./prog Shakespeare.txt
```
## Try:
```sh
./try.sh
./try.sh
```
However, as the binary model files used to produce the output are in an
@@ -93,56 +93,56 @@ For your first OMLET experiment, we will try training a simple single-level
RNN to write some [Shakespeare][5] plays. Start by typing
```sh
make
make
```
After it builds, train it using
```sh
./prog Shakespere.txt
./prog Shakespere.txt
```
This will immediately start outputting gibberish to the output, e.g.
```sh
./prog Shakespere.txt
./prog Shakespere.txt
```
produces:
```
sins ohennAu
T-teooclelp tiThoWy
sins ohennAu
T-teooclelp tiThoWy
g
nlakuafy
e
sselW usnsofueB Aoee pasfUsuslhe ooM ot Wou moy
me neltAl -no IoyI mhuyakse inT-l chu ghenn ffo? fnsoe yhyye
ue nnfrlass heUthole saounlcesyee pee
t,
T0:0% 3.210888
o',,vU
An ,hTf lnm Far rur:s moilt WoEgrv wonds mith Aog thernw
Rni So
co Nnd :
For an bImy pgafoun:
Wf'r hom wortiverita
int fod mous Eheledet,
Tho he theket nonS wnu-ang dorlaMSp
nrocWiSe tflg 'o.
T0:0% 2.995950
d whecedhencrysesil yr bn,
we hh y thiwt
g
nlakuafy
e
sselW usnsofueB Aoee pasfUsuslhe ooM ot Wou moy
me neltAl -no IoyI mhuyakse inT-l chu ghenn ffo? fnsoe yhyye
ue nnfrlass heUthole saounlcesyee pee
t,
T0:0% 3.210888
o',,vU
An ,hTf lnm Far rur:s moilt WoEgrv wonds mith Aog thernw
Rni So
co Nnd :
For an bImy pgafoun:
Wf'r hom wortiverita
int fod mous Eheledet,
Tho he theket nonS wnu-ang dorlaMSp
nrocWiSe tflg 'o.
T0:0% 2.995950
d whecedhencrysesil yr bn,
we hh y thiwt
hut fithlot,
Fmdy s he alt
Vh th no dh foud bobt werw:s Aotnf Fhwi't whe, eusu
hut fithlot,
Fmdy s he alt
Vh th no dh foud bobt werw:s Aotnf Fhwi't whe, eusu
lhh thele wewcond ary soupfy wind tDont couc ths:
er fucwald oncli hen bos, f
T0:1% 2.878945
lhh thele wewcond ary soupfy wind tDont couc ths:
er fucwald oncli hen bos, f
T0:1% 2.878945
```
The gibberish is the networks attempt to write a play. So far, it's not very
@@ -150,7 +150,7 @@ successful! Between the chunks of gibberish are training progress reports
that look like
```
T0:1% 2.878945
T0:1% 2.878945
```
This tells you that we are 1% through training epoch 0, and that the training
@@ -158,31 +158,31 @@ This tells you that we are 1% through training epoch 0, and that the training
reduce and the generated snippets will improve quickly:
```
ses, kuth
LAs of the wish,
As, I nos you,
Yov not to nalll,
Tr tot wonds.
ses, kuth
LAs of the wish,
As, I nos you,
Yov not to nalll,
Tr tot wonds.
First Sondy, llt lrte, our, tw.
First Sondy, llt lrte, our, tw.
First.
First.
BRUTUS:
Helsting the kith gops of hoch Whay, fars surd what to,
The cownens golt te.
T0:12% 2.259114
eplerrotrur tandans one wiok thy or thach and cullice ded yourssting
And wours:
Whed ur surt.
BRUTUS:
Helsting the kith gops of hoch Whay, fars surd what to,
The cownens golt te.
T0:12% 2.259114
eplerrotrur tandans one wiok thy or thach and cullice ded yourssting
And wours:
Whed ur surt.
SINENIUS:
On we lain bith
rerytund: tich lon hyivetetgor.
SINENIUS:
On we lain bith
rerytund: tich lon hyivetetgor.
VOLUONIA:
He brich nom dove worthan then wise,
T0:13% 2.254926
VOLUONIA:
He brich nom dove worthan then wise,
T0:13% 2.254926
```
It's already started to figure out things about Shakespeare plays -- how to
@@ -194,40 +194,40 @@ Eventually, we will finish with the training data set and move on to a
validation cycle:
```
y fath onother,
I sucess,
For I me the west crare.
y fath onother,
I sucess,
For I me the west crare.
TRANIO:
Whow and'd have not to had you you one in my lapteny she very ame come me a gut and shourd aghir you as ignested; shend to make I strem
To h
T0:99% 1.924063
notnce gaud and is nicked thou day, ha the dusing you disaid: in thim, you things in ere thee thus erile Iht that tare theme my hast thesp thou shay: thou not eaten-or-ho-bess resing: I the but had d
T0:99% 1.923128
V0:0% 1.678907
V0:9% 1.700527
V0:18% 1.733179
V0:27% 1.714891
V0:36% 1.716672
V0:45% 1.782946
V0:54% 1.835629
V0:63% 1.876108
V0:72% 1.906814
V0:81% 1.924096
V0:90% 1.954492
V0:99% 1.969287
serfs you'll alliencseard:
We
got you? before
I say.
TRANIO:
Whow and'd have not to had you you one in my lapteny she very ame come me a gut and shourd aghir you as ignested; shend to make I strem
To h
T0:99% 1.924063
notnce gaud and is nicked thou day, ha the dusing you disaid: in thim, you things in ere thee thus erile Iht that tare theme my hast thesp thou shay: thou not eaten-or-ho-bess resing: I the but had d
T0:99% 1.923128
V0:0% 1.678907
V0:9% 1.700527
V0:18% 1.733179
V0:27% 1.714891
V0:36% 1.716672
V0:45% 1.782946
V0:54% 1.835629
V0:63% 1.876108
V0:72% 1.906814
V0:81% 1.924096
V0:90% 1.954492
V0:99% 1.969287
serfs you'll alliencseard:
We
got you? before
I say.
Farstred dentlentecaly, sir, I it one bosticield
All me the backnour mino,
Whith capitaned mid! but stell the ifvemion
Willerity.
Farstred dentlentecaly, sir, I it one bosticield
All me the backnour mino,
Whith capitaned mid! but stell the ifvemion
Willerity.
First Cumfol of
T1:0% 1.885619
First Cumfol of
T1:0% 1.885619
```
Validation cycles are used to test the network to see if it has learned to
@@ -237,7 +237,7 @@ on the validation set is also displayed with a validation progress report that
looks like
```
V0:36% 1.716672
V0:36% 1.716672
```
which means we are 36% of the way through the validation for epoch 0, and the
@@ -260,7 +260,7 @@ You can continue training from a previous checkpoint by providing the
checkpoint file name as the second parameter, for example:
```sh
./prog Shakespeare.txt cp01_1.970
./prog Shakespeare.txt cp01_1.970
```
After the validation cycle finishes, OMLET begins the next epoch by restarting
@@ -276,7 +276,7 @@ parameters to generate data. Inference mode takes the checkpoint file as
command like
```sh
./prog < cp55_1.807
./prog < cp55_1.807
```
Running it produces an infinite amount of generated output, until you hit
@@ -295,8 +295,8 @@ executable name `prog`) is the simplest recurrent neural network. It looks
like
```c
h = tanh(Wxh * x + Whh * h' + Bh)
y = Why * h + By
h = tanh(Wxh * x + Whh * h' + Bh)
y = Why * h + By
```
where
@@ -318,53 +318,53 @@ It is the presence of the hidden state vector that allows the RNN to
state. If you type
```sh
make lin1
make lin1
```
OMLET will create an [ADALINE][10] network that does
```c
y = Wxy * x + By
y = Wxy * x + By
```
This is a simple [linear][11] [feed-forward network][12]. You can run it with
```sh
./lin1 Shakespeare.txt
./lin1 Shakespeare.txt
```
The linear network won't be able to get past the gibberish stage, because it
lacks history:
```sh
./lin1 Shakespeare.txt
./lin1 Shakespeare.txt
```
produces:
```
UERond w,
UERond w,
Gir:
KINof s, mesther s thouth.
E:
KINTret, at fu,
GOMy t, as sth kesewit sooos atse ang k, ck,
Sotheouserivesthecowhet been's, t he, h nre; t and, har wiread of pincer cedst sur has, ut:
Gir:
KINof s, mesther s thouth.
E:
KINTret, at fu,
GOMy t, as sth kesewit sooos atse ang k, ck,
Sotheouserivesthecowhet been's, t he, h nre; t and, har wiread of pincer cedst sur has, ut:
T14:67% 2.465115
UKESpan,
NGaromy soreate e m esewfoure pamitherarjulthengeoly tl.
NG s at e! w.
T14:67% 2.465115
UKESpan,
NGaromy soreate e m esewfoure pamitherarjulthengeoly tl.
NG s at e! w.
WAllinoully?
Wamisw ofilem:
I'delandinarrstath har aksubly s cath Whern t Is, weciss:
WAllinoully?
Wamisw ofilem:
I'delandinarrstath har aksubly s cath Whern t Is, weciss:
GLat s; llde.
Y aterit dsthence
T14:67% 2.465404
GLat s; llde.
Y aterit dsthence
T14:67% 2.465404
```
It is able to guess at what character is likely to follow the current one
@@ -389,9 +389,9 @@ We can try to improve the RNN's performance by stacking RNN modules atop each
other:
```c
h1 = RNN(h1', x)
h2 = RNN(h2', h1)
y = Why * h2 + By
h1 = RNN(h1', x)
h2 = RNN(h2', h1)
y = Why * h2 + By
```
with `RNN(h, x)` defined as above. Each RNN module has its own set of
@@ -415,13 +415,13 @@ the hard limit with `limit -h stacksize` and set it with
You can try the deeper network by doing
```sh
make rnn2
make rnn2
```
(or even `make rnn3` if you want a three-layer RNN) and train it with
```sh
./rnn2 Shakespeare.txt
./rnn2 Shakespeare.txt
```
The additional depth should allow the network to make better predictions (it
@@ -445,13 +445,13 @@ solve this problem. Christopher Olah gives a good description of them
at his [blog posting][20]. You can build a two-level LSTM by doing
```sh
make lstm2
make lstm2
```
and train with it with
```sh
./lstm2 Shakespeare.txt
./lstm2 Shakespeare.txt
```
The LSTM is much easier to train because it explicitly decides how to update
@@ -465,11 +465,11 @@ its hidden state via "gates". These gates are called
The basic LSTM equations are
```c
f = sigmoid(Wxf * x + Whf * h' + Bf)
i = sigmoid(Wxi * x + Whi * h' + Bi)
o = sigmoid(Wxo * x + Who * h' + Bo)
c = f * c' + i * tanh(Wxc * x + Whc * h' + Bc)
h = o * tanh(c)
f = sigmoid(Wxf * x + Whf * h' + Bf)
i = sigmoid(Wxi * x + Whi * h' + Bi)
o = sigmoid(Wxo * x + Who * h' + Bo)
c = f * c' + i * tanh(Wxc * x + Whc * h' + Bc)
h = o * tanh(c)
```
Where
@@ -497,7 +497,7 @@ require fewer learned parameters. This allows them to train faster than a
generic LSTM. You can build a two-layer GRU with
```sh
make gru2
make gru2
```
### Building your own networks
@@ -511,9 +511,9 @@ a `-DNW='...'` command which consists of a series of comma-separated
assignments. For example, the simple one-layer RNN could be defined like
```make
-DNW=' x = I(n), hp = I(128), \
h = C(hp, T(A(L(128, x), L(128, hp)))), \
y = L(n, h)'
-DNW=' x = I(n), hp = I(128), \
h = C(hp, T(A(L(128, x), L(128, hp)))), \
y = L(n, h)'
```
The network declares `x` as an input vector (there must be a declaration for
@@ -556,16 +556,16 @@ As an example of a more complicated network, we can look at a two-layer
GRU network:
```make
-DHS=128, \
-DNW=' x = I(n), \
y = L(n, MD(MD(x)))' \
-DBK=' hp = I(HS), \
z = S(A(L(HS, x), L(HS, hp))), \
r = S(A(L(HS, x), L(HS, hp))), \
c = T(A(L(HS, x), L(HS, hp))), \
zc = OG(1, -1, z), \
h = C(hp, A(M(zc, hp), M(z, c))), \
y = h'
-DHS=128, \
-DNW=' x = I(n), \
y = L(n, MD(MD(x)))' \
-DBK=' hp = I(HS), \
z = S(A(L(HS, x), L(HS, hp))), \
r = S(A(L(HS, x), L(HS, hp))), \
c = T(A(L(HS, x), L(HS, hp))), \
zc = OG(1, -1, z), \
h = C(hp, A(M(zc, hp), M(z, c))), \
y = h'
```
We are using a few new tricks here -- first, we are defining `HS` as the size
2019/mills/index.html
+142 -143
View File
@@ -132,8 +132,7 @@
<h2 id="author"><a name="author"></a>Author:</h2>
<ul>
<li>Name: <a href="../../authors.html#Christopher_Mills">Christopher Mills</a><br>
Location: <a href="../../location.html#US">US</a> - <em>United States of America</em> (<em>United States</em>)
<br><br></li>
Location: <a href="../../location.html#US">US</a> - <em>United States of America</em> (<em>United States</em>)</li>
</ul>
<!-- END: next line ends content generated by: bin/output-index-author.sh -->
@@ -142,17 +141,17 @@ Location: <a href="../../location.html#US">US</a> - <em>United States of America
<!-- START: this line starts content for HTML phase 21 by: bin/pandoc-wrapper.sh via bin/md2html.sh -->
<h2 id="to-build">To build:</h2>
<div class="sourceCode" id="cb1"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="fu">make</span></span></code></pre></div>
<div class="sourceCode" id="cb1"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a> <span class="fu">make</span></span></code></pre></div>
<h3 id="bugs-and-misfeatures">Bugs and (Mis)features:</h3>
<p>The current status of this entry is:</p>
<pre><code>STATUS: INABIAF - please **DO NOT** fix</code></pre>
<pre><code> STATUS: INABIAF - please **DO NOT** fix</code></pre>
<p>For more detailed information see <a href="../../bugs.html#2019_mills">2019 mills bugs</a>.</p>
<h2 id="to-use">To use:</h2>
<div class="sourceCode" id="cb3"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="fu">make</span> cpclean</span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="co"># Let this run for about about an hour and then kill it:</span></span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a><span class="ex">./prog</span> Shakespeare.txt</span></code></pre></div>
<div class="sourceCode" id="cb3"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a> <span class="fu">make</span> cpclean</span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a> <span class="co"># Let this run for about about an hour and then kill it:</span></span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a> <span class="ex">./prog</span> Shakespeare.txt</span></code></pre></div>
<h2 id="try">Try:</h2>
<div class="sourceCode" id="cb4"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="ex">./try.sh</span></span></code></pre></div>
<div class="sourceCode" id="cb4"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a> <span class="ex">./try.sh</span></span></code></pre></div>
<p>However, as the binary model files used to produce the output are in an
implementation-specific format, your mileage may vary.</p>
<h2 id="judges-remarks">Judges’ remarks:</h2>
@@ -201,122 +200,122 @@ training, but you can have even more fun by downloading some larger datasets:</p
</ul>
<p>For your first OMLET experiment, we will try training a simple single-level
RNN to write some <a href="https://en.wikipedia.org/wiki/William_Shakespeare">Shakespeare</a> plays. Start by typing</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="fu">make</span></span></code></pre></div>
<div class="sourceCode" id="cb5"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a> <span class="fu">make</span></span></code></pre></div>
<p>After it builds, train it using</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="ex">./prog</span> Shakespere.txt</span></code></pre></div>
<div class="sourceCode" id="cb6"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a> <span class="ex">./prog</span> Shakespere.txt</span></code></pre></div>
<p>This will immediately start outputting gibberish to the output, e.g.</p>
<div class="sourceCode" id="cb7"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a><span class="ex">./prog</span> Shakespere.txt</span></code></pre></div>
<div class="sourceCode" id="cb7"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a> <span class="ex">./prog</span> Shakespere.txt</span></code></pre></div>
<p>produces:</p>
<pre><code>sins ohennAu
T-teooclelp tiThoWy
<pre><code> sins ohennAu
T-teooclelp tiThoWy
g
nlakuafy
e
sselW usnsofueB Aoee pasfUsuslhe ooM ot Wou moy
me neltAl -no IoyI mhuyakse inT-l chu ghenn ffo? fnsoe yhyye
ue nnfrlass heUthole saounlcesyee pee
t,
T0:0% 3.210888
o&#39;,,vU
An ,hTf lnm Far rur:s moilt WoEgrv wonds mith Aog thernw
Rni So
co Nnd :
For an bImy pgafoun:
Wf&#39;r hom wortiverita
int fod mous Eheledet,
Tho he theket nonS wnu-ang dorlaMSp
nrocWiSe tflg &#39;o.
T0:0% 2.995950
d whecedhencrysesil yr bn,
we hh y thiwt
g
nlakuafy
e
sselW usnsofueB Aoee pasfUsuslhe ooM ot Wou moy
me neltAl -no IoyI mhuyakse inT-l chu ghenn ffo? fnsoe yhyye
ue nnfrlass heUthole saounlcesyee pee
t,
T0:0% 3.210888
o&#39;,,vU
An ,hTf lnm Far rur:s moilt WoEgrv wonds mith Aog thernw
Rni So
co Nnd :
For an bImy pgafoun:
Wf&#39;r hom wortiverita
int fod mous Eheledet,
Tho he theket nonS wnu-ang dorlaMSp
nrocWiSe tflg &#39;o.
T0:0% 2.995950
d whecedhencrysesil yr bn,
we hh y thiwt
hut fithlot,
Fmdy s he alt
Vh th no dh foud bobt werw:s Aotnf Fhwi&#39;t whe, eusu
hut fithlot,
Fmdy s he alt
Vh th no dh foud bobt werw:s Aotnf Fhwi&#39;t whe, eusu
lhh thele wewcond ary soupfy wind tDont couc ths:
er fucwald oncli hen bos, f
T0:1% 2.878945</code></pre>
lhh thele wewcond ary soupfy wind tDont couc ths:
er fucwald oncli hen bos, f
T0:1% 2.878945</code></pre>
<p>The gibberish is the networks attempt to write a play. So far, it’s not very
successful! Between the chunks of gibberish are training progress reports
that look like</p>
<pre><code>T0:1% 2.878945</code></pre>
<pre><code> T0:1% 2.878945</code></pre>
<p>This tells you that we are 1% through training epoch 0, and that the training
<a href="https://en.wikipedia.org/wiki/Loss_function">loss</a> is about 2.88. As the training continues, the training loss will
reduce and the generated snippets will improve quickly:</p>
<pre><code>ses, kuth
LAs of the wish,
As, I nos you,
Yov not to nalll,
Tr tot wonds.
<pre><code> ses, kuth
LAs of the wish,
As, I nos you,
Yov not to nalll,
Tr tot wonds.
First Sondy, llt lrte, our, tw.
First Sondy, llt lrte, our, tw.
First.
First.
BRUTUS:
Helsting the kith gops of hoch Whay, fars surd what to,
The cownens golt te.
T0:12% 2.259114
eplerrotrur tandans one wiok thy or thach and cullice ded yourssting
And wours:
Whed ur surt.
BRUTUS:
Helsting the kith gops of hoch Whay, fars surd what to,
The cownens golt te.
T0:12% 2.259114
eplerrotrur tandans one wiok thy or thach and cullice ded yourssting
And wours:
Whed ur surt.
SINENIUS:
On we lain bith
rerytund: tich lon hyivetetgor.
SINENIUS:
On we lain bith
rerytund: tich lon hyivetetgor.
VOLUONIA:
He brich nom dove worthan then wise,
T0:13% 2.254926</code></pre>
VOLUONIA:
He brich nom dove worthan then wise,
T0:13% 2.254926</code></pre>
<p>It’s already started to figure out things about Shakespeare plays – how to
spell short English words, how long lines tend to be, and that characters take
turns speaking with their names capitalized. The training loss has dropped to
2.25 and the improvement is noticeable.</p>
<p>Eventually, we will finish with the training data set and move on to a
validation cycle:</p>
<pre><code>y fath onother,
I sucess,
For I me the west crare.
<pre><code> y fath onother,
I sucess,
For I me the west crare.
TRANIO:
Whow and&#39;d have not to had you you one in my lapteny she very ame come me a gut and shourd aghir you as ignested; shend to make I strem
To h
T0:99% 1.924063
notnce gaud and is nicked thou day, ha the dusing you disaid: in thim, you things in ere thee thus erile Iht that tare theme my hast thesp thou shay: thou not eaten-or-ho-bess resing: I the but had d
T0:99% 1.923128
V0:0% 1.678907
V0:9% 1.700527
V0:18% 1.733179
V0:27% 1.714891
V0:36% 1.716672
V0:45% 1.782946
V0:54% 1.835629
V0:63% 1.876108
V0:72% 1.906814
V0:81% 1.924096
V0:90% 1.954492
V0:99% 1.969287
serfs you&#39;ll alliencseard:
We
got you? before
I say.
TRANIO:
Whow and&#39;d have not to had you you one in my lapteny she very ame come me a gut and shourd aghir you as ignested; shend to make I strem
To h
T0:99% 1.924063
notnce gaud and is nicked thou day, ha the dusing you disaid: in thim, you things in ere thee thus erile Iht that tare theme my hast thesp thou shay: thou not eaten-or-ho-bess resing: I the but had d
T0:99% 1.923128
V0:0% 1.678907
V0:9% 1.700527
V0:18% 1.733179
V0:27% 1.714891
V0:36% 1.716672
V0:45% 1.782946
V0:54% 1.835629
V0:63% 1.876108
V0:72% 1.906814
V0:81% 1.924096
V0:90% 1.954492
V0:99% 1.969287
serfs you&#39;ll alliencseard:
We
got you? before
I say.
Farstred dentlentecaly, sir, I it one bosticield
All me the backnour mino,
Whith capitaned mid! but stell the ifvemion
Willerity.
Farstred dentlentecaly, sir, I it one bosticield
All me the backnour mino,
Whith capitaned mid! but stell the ifvemion
Willerity.
First Cumfol of
T1:0% 1.885619</code></pre>
First Cumfol of
T1:0% 1.885619</code></pre>
<p>Validation cycles are used to test the network to see if it has learned to
generalize – how well it performs on data it hasn’t seen before (as opposed to
the training data that the network will see many times as it trains). Progress
on the validation set is also displayed with a validation progress report that
looks like</p>
<pre><code>V0:36% 1.716672</code></pre>
<pre><code> V0:36% 1.716672</code></pre>
<p>which means we are 36% of the way through the validation for epoch 0, and the
validation loss is about 1.72. Comparing the validation loss and the
training loss will give you an idea of how well the network is learning and
@@ -332,7 +331,7 @@ helpful if you need to stop and restart training. You can stop training by
doing <code>Control-C</code>.</p>
<p>You can continue training from a previous checkpoint by providing the
checkpoint file name as the second parameter, for example:</p>
<div class="sourceCode" id="cb13"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb13-1"><a href="#cb13-1" aria-hidden="true" tabindex="-1"></a><span class="ex">./prog</span> Shakespeare.txt cp01_1.970</span></code></pre></div>
<div class="sourceCode" id="cb13"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb13-1"><a href="#cb13-1" aria-hidden="true" tabindex="-1"></a> <span class="ex">./prog</span> Shakespeare.txt cp01_1.970</span></code></pre></div>
<p>After the validation cycle finishes, OMLET begins the next epoch by restarting
training at the beginning of the training set. Training continues forever,
until you quit it with <code>Control-C</code>. You should monitor the checkpoints to
@@ -343,7 +342,7 @@ OMLET to run the network in inference mode which uses the frozen checkpoint
parameters to generate data. Inference mode takes the checkpoint file as
<em>standard input</em> (not on the command line) and hence must be run with a
command like</p>
<div class="sourceCode" id="cb14"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb14-1"><a href="#cb14-1" aria-hidden="true" tabindex="-1"></a><span class="ex">./prog</span> <span class="op">&lt;</span> cp55_1.807</span></code></pre></div>
<div class="sourceCode" id="cb14"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb14-1"><a href="#cb14-1" aria-hidden="true" tabindex="-1"></a> <span class="ex">./prog</span> <span class="op">&lt;</span> cp55_1.807</span></code></pre></div>
<p>Running it produces an infinite amount of generated output, until you hit
<code>Control-C</code> to stop it.</p>
<p>Note that if you decide to change networks or use a different input file, you
@@ -354,8 +353,8 @@ cause a crash.</p>
<p>The default network for OMLET (the one you get if you type <code>make</code> with the
executable name <code>prog</code>) is the simplest recurrent neural network. It looks
like</p>
<div class="sourceCode" id="cb15"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb15-1"><a href="#cb15-1" aria-hidden="true" tabindex="-1"></a>h <span class="op">=</span> tanh<span class="op">(</span>Wxh <span class="op">*</span> x <span class="op">+</span> Whh <span class="op">*</span> h<span class="ch">&#39; </span><span class="er">+ Bh)</span></span>
<span id="cb15-2"><a href="#cb15-2" aria-hidden="true" tabindex="-1"></a>y <span class="op">=</span> Why <span class="op">*</span> h <span class="op">+</span> By</span></code></pre></div>
<div class="sourceCode" id="cb15"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb15-1"><a href="#cb15-1" aria-hidden="true" tabindex="-1"></a> h <span class="op">=</span> tanh<span class="op">(</span>Wxh <span class="op">*</span> x <span class="op">+</span> Whh <span class="op">*</span> h<span class="ch">&#39; </span><span class="er">+ Bh)</span></span>
<span id="cb15-2"><a href="#cb15-2" aria-hidden="true" tabindex="-1"></a> y <span class="op">=</span> Why <span class="op">*</span> h <span class="op">+</span> By</span></code></pre></div>
<p>where</p>
<ul>
<li><code>x</code> is the <em>input vector</em></li>
@@ -372,37 +371,37 @@ the loss of the network across the training set.</p>
<p>It is the presence of the hidden state vector that allows the RNN to
“remember” the past. We can see what would happen if we removed this hidden
state. If you type</p>
<div class="sourceCode" id="cb16"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb16-1"><a href="#cb16-1" aria-hidden="true" tabindex="-1"></a><span class="fu">make</span> lin1</span></code></pre></div>
<div class="sourceCode" id="cb16"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb16-1"><a href="#cb16-1" aria-hidden="true" tabindex="-1"></a> <span class="fu">make</span> lin1</span></code></pre></div>
<p>OMLET will create an <a href="https://en.wikipedia.org/wiki/ADALINE">ADALINE</a> network that does</p>
<div class="sourceCode" id="cb17"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb17-1"><a href="#cb17-1" aria-hidden="true" tabindex="-1"></a>y <span class="op">=</span> Wxy <span class="op">*</span> x <span class="op">+</span> By</span></code></pre></div>
<div class="sourceCode" id="cb17"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb17-1"><a href="#cb17-1" aria-hidden="true" tabindex="-1"></a> y <span class="op">=</span> Wxy <span class="op">*</span> x <span class="op">+</span> By</span></code></pre></div>
<p>This is a simple <a href="https://en.wikipedia.org/wiki/Linear_map">linear</a> <a href="https://en.wikipedia.org/wiki/Feedforward_neural_network">feed-forward network</a>. You can run it with</p>
<div class="sourceCode" id="cb18"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb18-1"><a href="#cb18-1" aria-hidden="true" tabindex="-1"></a><span class="ex">./lin1</span> Shakespeare.txt</span></code></pre></div>
<div class="sourceCode" id="cb18"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb18-1"><a href="#cb18-1" aria-hidden="true" tabindex="-1"></a> <span class="ex">./lin1</span> Shakespeare.txt</span></code></pre></div>
<p>The linear network won’t be able to get past the gibberish stage, because it
lacks history:</p>
<div class="sourceCode" id="cb19"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb19-1"><a href="#cb19-1" aria-hidden="true" tabindex="-1"></a><span class="ex">./lin1</span> Shakespeare.txt</span></code></pre></div>
<div class="sourceCode" id="cb19"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb19-1"><a href="#cb19-1" aria-hidden="true" tabindex="-1"></a> <span class="ex">./lin1</span> Shakespeare.txt</span></code></pre></div>
<p>produces:</p>
<pre><code>UERond w,
<pre><code> UERond w,
Gir:
KINof s, mesther s thouth.
E:
KINTret, at fu,
GOMy t, as sth kesewit sooos atse ang k, ck,
Sotheouserivesthecowhet been&#39;s, t he, h nre; t and, har wiread of pincer cedst sur has, ut:
Gir:
KINof s, mesther s thouth.
E:
KINTret, at fu,
GOMy t, as sth kesewit sooos atse ang k, ck,
Sotheouserivesthecowhet been&#39;s, t he, h nre; t and, har wiread of pincer cedst sur has, ut:
T14:67% 2.465115
UKESpan,
NGaromy soreate e m esewfoure pamitherarjulthengeoly tl.
NG s at e! w.
T14:67% 2.465115
UKESpan,
NGaromy soreate e m esewfoure pamitherarjulthengeoly tl.
NG s at e! w.
WAllinoully?
Wamisw ofilem:
I&#39;delandinarrstath har aksubly s cath Whern t Is, weciss:
WAllinoully?
Wamisw ofilem:
I&#39;delandinarrstath har aksubly s cath Whern t Is, weciss:
GLat s; llde.
Y aterit dsthence
T14:67% 2.465404</code></pre>
GLat s; llde.
Y aterit dsthence
T14:67% 2.465404</code></pre>
<p>It is able to guess at what character is likely to follow the current one
(by doing a <a href="https://en.wikipedia.org/wiki/Linear_regression">linear regression</a>), but it lacks any history beyond that to
guide it.</p>
@@ -419,9 +418,9 @@ the language but <code>per</code> with the digit 1).</p>
<h3 id="going-deeper">Going deeper:</h3>
<p>We can try to improve the RNN’s performance by stacking RNN modules atop each
other:</p>
<div class="sourceCode" id="cb21"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb21-1"><a href="#cb21-1" aria-hidden="true" tabindex="-1"></a>h1 <span class="op">=</span> RNN<span class="op">(</span>h1<span class="ch">&#39;,</span><span class="er"> x)</span></span>
<span id="cb21-2"><a href="#cb21-2" aria-hidden="true" tabindex="-1"></a>h2 <span class="op">=</span> RNN<span class="op">(</span>h2<span class="ch">&#39;,</span><span class="er"> h1)</span></span>
<span id="cb21-3"><a href="#cb21-3" aria-hidden="true" tabindex="-1"></a>y <span class="op">=</span> Why <span class="op">*</span> h2 <span class="op">+</span> By</span></code></pre></div>
<div class="sourceCode" id="cb21"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb21-1"><a href="#cb21-1" aria-hidden="true" tabindex="-1"></a> h1 <span class="op">=</span> RNN<span class="op">(</span>h1<span class="ch">&#39;,</span><span class="er"> x)</span></span>
<span id="cb21-2"><a href="#cb21-2" aria-hidden="true" tabindex="-1"></a> h2 <span class="op">=</span> RNN<span class="op">(</span>h2<span class="ch">&#39;,</span><span class="er"> h1)</span></span>
<span id="cb21-3"><a href="#cb21-3" aria-hidden="true" tabindex="-1"></a> y <span class="op">=</span> Why <span class="op">*</span> h2 <span class="op">+</span> By</span></code></pre></div>
<p>with <code>RNN(h, x)</code> defined as above. Each RNN module has its own set of
parameters and its own hidden state vector. This will improve the network’s
performance, at the cost of a much larger parameter space.</p>
@@ -437,9 +436,9 @@ the hard limit with <code>limit -h stacksize</code> and set it with
<code>limit stacksize 65532</code> (replacing <code>65532</code> with the actual hard limit).</p>
<hr />
<p>You can try the deeper network by doing</p>
<div class="sourceCode" id="cb22"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb22-1"><a href="#cb22-1" aria-hidden="true" tabindex="-1"></a><span class="fu">make</span> rnn2</span></code></pre></div>
<div class="sourceCode" id="cb22"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb22-1"><a href="#cb22-1" aria-hidden="true" tabindex="-1"></a> <span class="fu">make</span> rnn2</span></code></pre></div>
<p>(or even <code>make rnn3</code> if you want a three-layer RNN) and train it with</p>
<div class="sourceCode" id="cb23"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb23-1"><a href="#cb23-1" aria-hidden="true" tabindex="-1"></a><span class="ex">./rnn2</span> Shakespeare.txt</span></code></pre></div>
<div class="sourceCode" id="cb23"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb23-1"><a href="#cb23-1" aria-hidden="true" tabindex="-1"></a> <span class="ex">./rnn2</span> Shakespeare.txt</span></code></pre></div>
<p>The additional depth should allow the network to make better predictions (it
can represent more complicated history), but it may take a long time to train
-- both because the network (being larger) now requires more time to train and
@@ -455,9 +454,9 @@ gradient, worsening the exploding and vanishing gradient problem.</p>
<p><a href="https://en.wikipedia.org/wiki/Long_short-term_memory">Long Short Term Memory</a> networks (also called LSTMs) were developed to
solve this problem. Christopher Olah gives a good description of them
at his <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">blog posting</a>. You can build a two-level LSTM by doing</p>
<div class="sourceCode" id="cb24"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb24-1"><a href="#cb24-1" aria-hidden="true" tabindex="-1"></a><span class="fu">make</span> lstm2</span></code></pre></div>
<div class="sourceCode" id="cb24"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb24-1"><a href="#cb24-1" aria-hidden="true" tabindex="-1"></a> <span class="fu">make</span> lstm2</span></code></pre></div>
<p>and train with it with</p>
<div class="sourceCode" id="cb25"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb25-1"><a href="#cb25-1" aria-hidden="true" tabindex="-1"></a><span class="ex">./lstm2</span> Shakespeare.txt</span></code></pre></div>
<div class="sourceCode" id="cb25"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb25-1"><a href="#cb25-1" aria-hidden="true" tabindex="-1"></a> <span class="ex">./lstm2</span> Shakespeare.txt</span></code></pre></div>
<p>The LSTM is much easier to train because it explicitly decides how to update
its hidden state via “gates”. These gates are called</p>
<ul>
@@ -467,11 +466,11 @@ its hidden state via “gates”. These gates are called</p>
produce the output</li>
</ul>
<p>The basic LSTM equations are</p>
<div class="sourceCode" id="cb26"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb26-1"><a href="#cb26-1" aria-hidden="true" tabindex="-1"></a>f <span class="op">=</span> sigmoid<span class="op">(</span>Wxf <span class="op">*</span> x <span class="op">+</span> Whf <span class="op">*</span> h<span class="ch">&#39; </span><span class="er">+ Bf)</span></span>
<span id="cb26-2"><a href="#cb26-2" aria-hidden="true" tabindex="-1"></a>i <span class="op">=</span> sigmoid<span class="op">(</span>Wxi <span class="op">*</span> x <span class="op">+</span> Whi <span class="op">*</span> h<span class="ch">&#39; </span><span class="er">+ Bi)</span></span>
<span id="cb26-3"><a href="#cb26-3" aria-hidden="true" tabindex="-1"></a>o <span class="op">=</span> sigmoid<span class="op">(</span>Wxo <span class="op">*</span> x <span class="op">+</span> Who <span class="op">*</span> h<span class="ch">&#39; </span><span class="er">+ Bo)</span></span>
<span id="cb26-4"><a href="#cb26-4" aria-hidden="true" tabindex="-1"></a>c <span class="op">=</span> f <span class="op">*</span> c<span class="ch">&#39; </span><span class="er">+ i * tanh(Wxc * x + Whc * h</span><span class="ch">&#39;</span> <span class="op">+</span> Bc<span class="op">)</span></span>
<span id="cb26-5"><a href="#cb26-5" aria-hidden="true" tabindex="-1"></a>h <span class="op">=</span> o <span class="op">*</span> tanh<span class="op">(</span>c<span class="op">)</span></span></code></pre></div>
<div class="sourceCode" id="cb26"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb26-1"><a href="#cb26-1" aria-hidden="true" tabindex="-1"></a> f <span class="op">=</span> sigmoid<span class="op">(</span>Wxf <span class="op">*</span> x <span class="op">+</span> Whf <span class="op">*</span> h<span class="ch">&#39; </span><span class="er">+ Bf)</span></span>
<span id="cb26-2"><a href="#cb26-2" aria-hidden="true" tabindex="-1"></a> i <span class="op">=</span> sigmoid<span class="op">(</span>Wxi <span class="op">*</span> x <span class="op">+</span> Whi <span class="op">*</span> h<span class="ch">&#39; </span><span class="er">+ Bi)</span></span>
<span id="cb26-3"><a href="#cb26-3" aria-hidden="true" tabindex="-1"></a> o <span class="op">=</span> sigmoid<span class="op">(</span>Wxo <span class="op">*</span> x <span class="op">+</span> Who <span class="op">*</span> h<span class="ch">&#39; </span><span class="er">+ Bo)</span></span>
<span id="cb26-4"><a href="#cb26-4" aria-hidden="true" tabindex="-1"></a> c <span class="op">=</span> f <span class="op">*</span> c<span class="ch">&#39; </span><span class="er">+ i * tanh(Wxc * x + Whc * h</span><span class="ch">&#39;</span> <span class="op">+</span> Bc<span class="op">)</span></span>
<span id="cb26-5"><a href="#cb26-5" aria-hidden="true" tabindex="-1"></a> h <span class="op">=</span> o <span class="op">*</span> tanh<span class="op">(</span>c<span class="op">)</span></span></code></pre></div>
<p>Where</p>
<ul>
<li><code>x</code> is the input vector</li>
@@ -495,7 +494,7 @@ examples). One important one is the <a href="https://en.wikipedia.org/wiki/Gated
simplified versions of an LSTM which combine the gates together, meaning they
require fewer learned parameters. This allows them to train faster than a
generic LSTM. You can build a two-layer GRU with</p>
<div class="sourceCode" id="cb27"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb27-1"><a href="#cb27-1" aria-hidden="true" tabindex="-1"></a><span class="fu">make</span> gru2</span></code></pre></div>
<div class="sourceCode" id="cb27"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb27-1"><a href="#cb27-1" aria-hidden="true" tabindex="-1"></a> <span class="fu">make</span> gru2</span></code></pre></div>
<h3 id="building-your-own-networks">Building your own networks</h3>
<p>The OMLET <code>Makefile</code> comes with one-, two- and three-layer RNNs, LSTMs and
GRUs, along with simpler feed-forward networks like multi-layer perceptrons and
@@ -504,9 +503,9 @@ own networks by modifying the <code>Makefile</code>. Networks are passed in on t
compiler’s command-line by using <code>-D</code> directives. The network is defined by
a <code>-DNW='...'</code> command which consists of a series of comma-separated
assignments. For example, the simple one-layer RNN could be defined like</p>
<pre class="make"><code>-DNW=&#39; x = I(n), hp = I(128), \
h = C(hp, T(A(L(128, x), L(128, hp)))), \
y = L(n, h)&#39;</code></pre>
<pre class="make"><code> -DNW=&#39; x = I(n), hp = I(128), \
h = C(hp, T(A(L(128, x), L(128, hp)))), \
y = L(n, h)&#39;</code></pre>
<p>The network declares <code>x</code> as an input vector (there must be a declaration for
<code>x</code>). It is declared as <code>I(n)</code>, which is an input vector of size <code>n</code>, which
is the number of characters of the input alphabet (OMLET computes this from
@@ -539,16 +538,16 @@ inference mode, this is used to select the next character to emit. In
training mode, this is used to generate the loss which is backpropagated.</p>
<p>As an example of a more complicated network, we can look at a two-layer
GRU network:</p>
<pre class="make"><code>-DHS=128, \
-DNW=&#39; x = I(n), \
y = L(n, MD(MD(x)))&#39; \
-DBK=&#39; hp = I(HS), \
z = S(A(L(HS, x), L(HS, hp))), \
r = S(A(L(HS, x), L(HS, hp))), \
c = T(A(L(HS, x), L(HS, hp))), \
zc = OG(1, -1, z), \
h = C(hp, A(M(zc, hp), M(z, c))), \
y = h&#39;</code></pre>
<pre class="make"><code> -DHS=128, \
-DNW=&#39; x = I(n), \
y = L(n, MD(MD(x)))&#39; \
-DBK=&#39; hp = I(HS), \
z = S(A(L(HS, x), L(HS, hp))), \
r = S(A(L(HS, x), L(HS, hp))), \
c = T(A(L(HS, x), L(HS, hp))), \
zc = OG(1, -1, z), \
h = C(hp, A(M(zc, hp), M(z, c))), \
y = h&#39;</code></pre>
<p>We are using a few new tricks here – first, we are defining <code>HS</code> as the size
of the hidden and cell vectors. There’s nothing special about this name, it’s
just convenient to specify it so we don’t have a bunch of constants in the