Remove hint.html and index.html
As per discussions on GitHub the index.html and hint.html files should be deleted in favour of README.md files (from any .text/.txt/.markdown/.md files which will be turned into README.md in an upcoming commit).
This commit is contained in:
Cody Boone Ferguson committed
1 parent
ee8a36310f
commit
77fb0f11a2
143 files changed
-28606
No files matched your search
@@ -1,273 +0,0 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta name="GENERATOR" content="mkd2html 2.2.0">
|
||||
<meta http-equiv="Content-Type"
|
||||
content="text/html; charset=utf-8"> <link rel="stylesheet"
|
||||
type="text/css"
|
||||
href="../hint.css">
|
||||
<title>Most functional interpreter</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Most functional interpreter</h1>
|
||||
|
||||
<p>Ondřej Adamovský <a href="mailto:oa@cmail.cz">oa@cmail.cz</a></p>
|
||||
|
||||
<h2>Judges' comments:</h2>
|
||||
|
||||
<h3>To use:</h3>
|
||||
|
||||
<pre><code>make prog
|
||||
./prog file.unl
|
||||
</code></pre>
|
||||
|
||||
<h3>Try:</h3>
|
||||
|
||||
<pre><code>./prog spam.unl
|
||||
|
||||
./prog sierpinski.unl | less
|
||||
|
||||
./prog advent.unl
|
||||
</code></pre>
|
||||
|
||||
<h3>Selected Judges Remarks:</h3>
|
||||
|
||||
<p>Even after pre-processing and indenting, the C code of this entry
|
||||
is about as understandable as the Unlambda code.</p>
|
||||
|
||||
<p>Pretending that we don’t know a bit of functional programming :) and lacking a better way
|
||||
to understand the entry but to fuzz it,
|
||||
we stumbled on a string of bytes which crashed it:</p>
|
||||
|
||||
<pre><code>````.0`.0`.0`c.0``sssss
|
||||
</code></pre>
|
||||
|
||||
<p>Functional programming is not a panacea against core dumps, after all.</p>
|
||||
|
||||
<h2>Author’s comments:</h2>
|
||||
|
||||
<p><strong>»A highly structured state state machine«</strong></p>
|
||||
|
||||
<h3>Purpose</h3>
|
||||
|
||||
<p>The purpose of this program is to allow you to play Colossal Cave Adventure as
|
||||
implemented by <a href="https://github.com/irori/advent-unlambda" title="Adventure">Kunihiko Sakamoto</a>. As that was written in <a href="http://www.madore.org/~david/programs/unlambda/" title="Unlambda programming language">Unlambda programming
|
||||
language</a> designed by David Alexander Madore it has to be an Unlambda interpreter.</p>
|
||||
|
||||
<h3>Aesthetics</h3>
|
||||
|
||||
<p>The program grew a bit too long so I had to use several macros to downsize it.
|
||||
That unfortunately reduced my options for program layout. I decided to separate
|
||||
it into two portions. The initial macro section where I had limited resources and
|
||||
you should disregard its shape, unless you like it, and the rest is a separator in the
|
||||
form of a single line followed by a section signifying the function of the program,
|
||||
a block with removed shape of lambda character. Even the official IOCCC size tool
|
||||
acknowledges this: When you call <code>./iocccsize -i <prog.c</code>, using the
|
||||
<a href="/2019/iocccsize.c">2019 version of iocccsize</a>,
|
||||
iocccsize returns <strong>955</strong>, which is the Unicode codepoint of the <strong>λ</strong> character.</p>
|
||||
|
||||
<p>It might be a surprise for you that according to the IOCCC size tool the program is
|
||||
actually a <em>oneliner</em>. It certainly was for me. I like to think the tool is overwhelmed
|
||||
by the sheer length of the dividing line and totally overlooks the rest of the program.</p>
|
||||
|
||||
<h3>Usage</h3>
|
||||
|
||||
<p>When built directly</p>
|
||||
|
||||
<pre><code>make
|
||||
</code></pre>
|
||||
|
||||
<p>the program (<em>prog</em>) accepts a single parameter with the name of an Unlambda program. You can
|
||||
download the <a href="https://github.com/irori/advent-unlambda" title="Adventure">Colossal Cave Adventure</a> (the <em>advent.unl</em> file) and run it like this</p>
|
||||
|
||||
<pre><code>./prog advent.unl
|
||||
</code></pre>
|
||||
|
||||
<p>You can also try other programs from the Web. Many example programs are in the official
|
||||
<a href="http://www.madore.org/~david/programs/unlambda/#distrib" title="Reference Unlambda distribution">Unlambda distribution</a> on the Unlambda homepage. The most complex programs there are
|
||||
entries from <a href="http://www.madore.org/~david/programs/unlambda/#quine" title="Unlambda quine contest">quine contest</a>.</p>
|
||||
|
||||
<p>There is an alternative build path that requires IOCCC size tool to complete. Put the tool
|
||||
source <em>iocccsize.c</em> in the project directory and call</p>
|
||||
|
||||
<pre><code>make identify
|
||||
</code></pre>
|
||||
|
||||
<p>It will build the tool and use it to build an alternative program <em>prog2</em>, which in turn will
|
||||
produce an Unlambda program <em>identify.unl</em>. <code>make identify</code> runs this program to identify
|
||||
itself.</p>
|
||||
|
||||
<p>The base program (<em>prog</em>) implements an interpreter of Unlambda 2.0. It recognises all its
|
||||
builtins (k, s, i, v, c, d, .<em>x</em>, e, @, ?<em>x</em> and |) including the syntactic sugar function
|
||||
(​r​) and formating properties in the form of ignoring whitespace and comments
|
||||
to the end of line introduced by the <em>#</em> character.</p>
|
||||
|
||||
<p>In case of wrong usage (missing Unlambda program file or missing parameter) or use of an unseekable
|
||||
Unlambda program source if it uses the <em>c</em> function, the program returns message <em>fail</em>. It
|
||||
also informs you about syntactic errors in the Unlambda program should there be any (but only
|
||||
when the execution reaches them).</p>
|
||||
|
||||
<p>The Unlambda source file may contain several Unlambda programs. The interpreter will run
|
||||
them all in sequence.</p>
|
||||
|
||||
<h3>Unlambda</h3>
|
||||
|
||||
<p>If you want to familiarise yourself with the language, visit its <a href="https://en.wikipedia.org/wiki/Unlambda" title="Unlambda on Wikipedia">Wikipedia page</a> for
|
||||
a brief introduction or its <a href="http://www.madore.org/~david/programs/unlambda/" title="Unlambda programming language">homepage</a> for an indepth discussion. For further reading you
|
||||
can visit Wikipedia pages on <a href="https://en.wikipedia.org/wiki/Lambda_calculus" title="Lambda Calculus on Wikipedia">Lambda Calculus</a> and <a href="https://en.wikipedia.org/wiki/Combinatory_logic" title="Combinatory Logic on Wikipedia">Combinatory Logic</a>.</p>
|
||||
|
||||
<p>You can also play with an <a href="https://inazz.jp/unlambda/" title="Online Unlambda interpreter">online interpreter</a> by Github user inazz. There you can
|
||||
debug one of example programs provided or try your wits in writing your own.</p>
|
||||
|
||||
<h3>Hints</h3>
|
||||
|
||||
<p>Here are some pointers where to start when trying to understand the interpreter:</p>
|
||||
|
||||
<ul>
|
||||
<li>Where is syntax error detected?</li>
|
||||
<li>How is the memory managed?</li>
|
||||
<li>How does a program character turn into a function?</li>
|
||||
<li>How does exit function <em>e</em> work?</li>
|
||||
<li>Can fD0 function be called?</li>
|
||||
<li>Are the function names familiar? (warning: some are misleading)</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<h3>Obfuscation sources</h3>
|
||||
|
||||
<ul>
|
||||
<li>The elephant in the room is the Unlambda language. As we all know, functional
|
||||
programming defies sanity. Furthermore, Unlambda was specifically designed
|
||||
to be unintelligible and hard to code in. Most of its functions are quite
|
||||
simple and straightforward to implement, but two make it a mess: the promise
|
||||
operator <em>d</em> and the call with current continuation <em>c</em>.</li>
|
||||
<li>Most of the additional obfuscation is derived from basic design choices. That
|
||||
is, how the state of the program is stored and how are the Unlambda functions
|
||||
implemented.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>More on this in the Spoilers section.</p>
|
||||
|
||||
<h3>Build notes</h3>
|
||||
|
||||
<p>The program does not require any special treatment by compilers. Some warnings
|
||||
had to be suppressed when compiling with <code>-Wall -Wextra -pedantic</code>. Those were
|
||||
produced because of the formatting of the source code, not any programming
|
||||
technique. Compilation with <code>-Weverything</code> produced many more warnings, but those
|
||||
are only relevant for multi-source-file projects and some are really questionable
|
||||
(e.g. <code>-Wdisabled-macro-expansion</code>).</p>
|
||||
|
||||
<h3>Portability</h3>
|
||||
|
||||
<p>Development was done on Debian 9 on x86-64 architecture using mainly Clang 3.8.1-24
|
||||
and GCC 6.3.0. The source is C11 compliant and does not use any special properties
|
||||
except the following:</p>
|
||||
|
||||
<ul>
|
||||
<li>The interpreter sort of relies on tail call optimization. It works well even
|
||||
without it, but the stack runs out quickly with some Unlambda programs (e.g. <em>trivial.unl</em>
|
||||
example from the <a href="http://www.madore.org/~david/programs/unlambda/#distrib" title="Reference Unlambda distribution">reference Unlambda distribution</a> produces segfault).
|
||||
Clang with <code>-O3</code> option does this optimization well and said example does
|
||||
not segfault with it. I was not able to force GCC to optimize it out (probably
|
||||
an operator fault).</li>
|
||||
<li>The only data structure of the program (the union structure) had much more
|
||||
members originally. I joined all integers to a single long long member in the name
|
||||
of obfuscation. It is used for reference counting (only 1 for every reference) among
|
||||
other uses. size_t should be used for that, but practically, I think the memory
|
||||
would run out sooner than the reference counter in any real world situation
|
||||
even when size_t is bigger than long long. It is also used for storing chars
|
||||
for later printing. This might produce some warnings, but since it was developed
|
||||
using chars it should be always used safely.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<h3>Spoilers</h3>
|
||||
|
||||
<ul>
|
||||
<li><p>You can examine the <em>progSpoiler.c</em> file. It is the original final version of
|
||||
the interpreter, before I started the downsizing. It accepts one more optional
|
||||
parameter that specifies filename for the output Unlambda function (it’s the
|
||||
Unlambda function, that remains after all function applications are done).
|
||||
When no filename is provided, <em>stderr</em> is used.</p>
|
||||
|
||||
<p>The function <em>output</em> is used for this additional functionality. You can use
|
||||
it while debugging to visualise current variables.</p></li>
|
||||
<li><p>The Unlambda functions are represented by a record consisting of an array of five
|
||||
multipurpose arguments. The first one refers to the function implementing its
|
||||
application, the second and the third ones are for arguments of the function,
|
||||
the fourth one is for stack building, and the fifth one is the reference counter.</p>
|
||||
|
||||
<p>The function argument is used not only for evaluation of the functions, but also
|
||||
as a state variable to decide whether we want to call it, or do something else.</p></li>
|
||||
<li><p>The interpreter reads the Unlambda source as it interprets it, translating
|
||||
the incoming character to a function using a constructor stored in translation array <em>T</em>.</p>
|
||||
|
||||
<p>This means the interpreter must reposition in the source file whenever current
|
||||
continuation is called.</p></li>
|
||||
<li><p>The state of the interpreter is represented by trees representing evaluated
|
||||
Unlambda functions stored on the C stack as local variables, plus the source file
|
||||
position. When a current continuation is created, the stack is unwound using the
|
||||
fourth argument of the nodes, stored in the continuation, and rebuilt back resuming
|
||||
the evaluation. When its called, the stack is unwound and thrown away, and the
|
||||
stack stored in the continuation is restored.</p>
|
||||
|
||||
<p>This makes implementation of a complete debugger a bit more challenging.</p>
|
||||
|
||||
<p>For this to work, the functions that do the work (i.e. make series of applications
|
||||
and/or evaluations each of which may be a call to current continuation), the <em>s</em>
|
||||
combinator and the <em>`</em> application operation had to be broken down to pieces, one
|
||||
for each application/evaluation and corresponding helper functions for stack
|
||||
rebuilding. (the helper functions are named beginning with <em>r</em> instead of <em>f</em>)</p></li>
|
||||
<li><p>The <em>`</em> application operation does the reading of the Unlambda source when needed. It
|
||||
also implements the promise function. Whenever it would apply it, it overwrites
|
||||
the Apply constructor in translation array <em>T</em>, changing its function to build
|
||||
unevaluated subtree that is later evaluated whenever the promise is applied.</p></li>
|
||||
<li><p>To further the obfuscation, I shortened the function names while downsizing,
|
||||
but preserved their meaning (with the exception of Apply functions, which I
|
||||
renamed to Y to possibly confuse with Y combinator). Simple functions retained
|
||||
their names to misguide the reader to believe to know what is what. The names
|
||||
of continuation and exit functions were swapped. The names of S and Apply
|
||||
functions were shuffled both for S functions and Apply functions in the same way
|
||||
to hide the meaning of <em>f</em> and <em>r</em> prefixes.</p></li>
|
||||
</ul>
|
||||
|
||||
|
||||
<h3>Identification spoiler</h3>
|
||||
|
||||
<p>The alternative build path <code>make identify</code> output commemorates a <a href="http://ioccc.org/2014/birken/hint.html" title="Best use of port 1701">previous IOCCC winner</a> which uses
|
||||
similar methods to mess with the size tool.</p>
|
||||
|
||||
<h3>Judges remarks spoilers</h3>
|
||||
|
||||
<p>It is not a bug, it is a feature. :)</p>
|
||||
|
||||
<p>The crash is caused by a stack overflow. Conventional Unlabmda interpreters implement
|
||||
the Unlambda execution stack using data structures on the heap. This interpreter uses
|
||||
the C call stack for this purpose. In a conventional interpreter, the judges program would
|
||||
consume memory until the user (or the operating system) would lose patience. The C call stack
|
||||
is usually of a limited size and protected, so this interpreter crashes, when the judges
|
||||
program depletes the stack. You can always increase the maximal stack size through
|
||||
the system settings (on Linux) or compiler options (on Windows) if your Unlambda program
|
||||
needs it. Nevertheless, it will not help the judges program which quickly depletes any limit
|
||||
you define.</p>
|
||||
|
||||
<p>The judges, pretending not to know what they are doing, created a program that grows
|
||||
Unlambda execution stack indefinitely. You may consider it an Unlambda antipattern.
|
||||
Normally, infinite cycles in Unlambda work by creating two functions and applying one
|
||||
to the other which creates further functions to apply. The application of the function
|
||||
frees its position on the stack. This program builds one ever growing function that will
|
||||
never be complete and able to be applied.</p>
|
||||
|
||||
<hr />
|
||||
|
||||
<TABLE><TR>
|
||||
<TD><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/"><img alt="Creative Commons License" style="border-width:0" src="/png/by-sa-3.0-88x31.png" /></a></TD>
|
||||
<TD><P>© Copyright 1984-2019,
|
||||
<A HREF="/judges.html">Leo Broukhis, Simon Cooper, Landon Curt Noll</A>
|
||||
- All rights reserved<BR>
|
||||
This work is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.</P></TD>
|
||||
<TD> <!--<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a>--></TD>
|
||||
</TR></TABLE>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,101 +0,0 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta name="GENERATOR" content="mkd2html 2.2.0">
|
||||
<meta http-equiv="Content-Type"
|
||||
content="text/html; charset=utf-8"> <link rel="stylesheet"
|
||||
type="text/css"
|
||||
href="../hint.css">
|
||||
<title>Best one-liner</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Best one-liner</h1>
|
||||
|
||||
<p>Dave Burton</p>
|
||||
|
||||
<h2>Judges' comments:</h2>
|
||||
|
||||
<h3>To use:</h3>
|
||||
|
||||
<pre><code>make
|
||||
</code></pre>
|
||||
|
||||
<h3>Try:</h3>
|
||||
|
||||
<pre><code>./prog < prog.c
|
||||
</code></pre>
|
||||
|
||||
<h3>Selected Judges Remarks:</h3>
|
||||
|
||||
<p>A tiny, but punctually correct way of POSIX counting for text files. This one
|
||||
line, one word, 127 character beauty continues to expand the number of classic
|
||||
unix utilities available as winning entries.</p>
|
||||
|
||||
<h2>Author’s comments:</h2>
|
||||
|
||||
<h2>Synopsis</h2>
|
||||
|
||||
<p>Feed it anything on stdin. <br>
|
||||
The output should look familiar.</p>
|
||||
|
||||
<p>Compiles cleanly with:</p>
|
||||
|
||||
<pre><code>WARN="-Wno-implicit-int -Wno-missing-variable-declarations -Wno-parentheses"
|
||||
clang -Wall -Wextra -Weverything -pedantic $WARN -include stdio.h -o prog prog.c
|
||||
</code></pre>
|
||||
|
||||
<h2>Description</h2>
|
||||
|
||||
<p>Included is a test script, useful for the corner cases.
|
||||
As in all good Unix programs, it is silent upon success; any output demonstrates an error.
|
||||
Instead of checking correctness, the check script can simply output the results: <br>
|
||||
(The Makefile contains the spoiler):</p>
|
||||
|
||||
<pre><code>out=cat ./check ./prog
|
||||
</code></pre>
|
||||
|
||||
<p>Interestingly, and a spoiler here: MacOS and FreeBSD implement the common utility “correctly”.
|
||||
Centos and Ubuntu both share a “flawed” utility that changes output format based upon input redirection or pipeline(!).
|
||||
As well, the latter two also behave slightly differently on what they count as whitespace:
|
||||
\a and \b are both counted whitespace(!), even though the relevant man pages on these systems
|
||||
omit these characters from the description.</p>
|
||||
|
||||
<p>This program follows FreeBSD and MacOS, the arbiters of correct for these purposes.</p>
|
||||
|
||||
<h2>Limits</h2>
|
||||
|
||||
<p>Requires the C locale and ASCII character set. <br>
|
||||
Input should be less than ten million octets toavoidthisproblem.</p>
|
||||
|
||||
<h2>See Also</h2>
|
||||
|
||||
<p>You can enjoy this code from beginning to end. <br>
|
||||
No whitespace was harmed in the creation of this program. <br>
|
||||
All your base belong to us, but the constants seem a bit … off?</p>
|
||||
|
||||
<h2>Bonus</h2>
|
||||
|
||||
<p>The script <code>cow</code> provides a closer analog, and perhaps a more useful interface.</p>
|
||||
|
||||
<p>Included is a version without compilation warnings:</p>
|
||||
|
||||
<pre><code>clang -Wall -Wextra -Weverything -pedantic -include stdio.h -o prog.clean prog.clean.c
|
||||
</code></pre>
|
||||
|
||||
<h2>Bugs</h2>
|
||||
|
||||
<p>The flags are not supported. Although they could easily be added into <code>cow</code>.
|
||||
This exercise left to the reader.</p>
|
||||
|
||||
<hr />
|
||||
|
||||
<TABLE><TR>
|
||||
<TD><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/"><img alt="Creative Commons License" style="border-width:0" src="/png/by-sa-3.0-88x31.png" /></a></TD>
|
||||
<TD><P>© Copyright 1984-2019,
|
||||
<A HREF="/judges.html">Leo Broukhis, Simon Cooper, Landon Curt Noll</A>
|
||||
- All rights reserved<BR>
|
||||
This work is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.</P></TD>
|
||||
<TD> <!--<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a>--></TD>
|
||||
</TR></TABLE>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,183 +0,0 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta name="GENERATOR" content="mkd2html 2.2.0">
|
||||
<meta http-equiv="Content-Type"
|
||||
content="text/html; charset=utf-8"> <link rel="stylesheet"
|
||||
type="text/css"
|
||||
href="../hint.css">
|
||||
<title>Most alphabetic</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Most alphabetic</h1>
|
||||
|
||||
<p>Marcin Ciura<br>
|
||||
Twitter: @mciura</p>
|
||||
|
||||
<h2>Judges' comments:</h2>
|
||||
|
||||
<h3>To use:</h3>
|
||||
|
||||
<pre><code>make
|
||||
|
||||
./getwords.sh en | grep .. | ./prog aeiouvwxyz
|
||||
</code></pre>
|
||||
|
||||
<h3>Try:</h3>
|
||||
|
||||
<pre><code>./getwords.sh en | grep .. | ./prog aeiouhjklmnvwxyz
|
||||
|
||||
./getwords.sh en | grep .. | ./prog mrjocktvquizphdbagsfewlynx
|
||||
|
||||
./en.sh
|
||||
|
||||
./fr.sh
|
||||
|
||||
./ru.sh
|
||||
</code></pre>
|
||||
|
||||
<h3>Selected Judges Remarks:</h3>
|
||||
|
||||
<p>A few letters, one at a time, with no repeats.
|
||||
How many different ways can this be done?</p>
|
||||
|
||||
<p>While a quick brown fox might jump over the lazy dog,
|
||||
it has too many repeat letters to allow this entry to repeat.
|
||||
It is better that Mr Jock, TV quiz PhD, bags few lynx.</p>
|
||||
|
||||
<p>Speaking of jumping, can you rewrite the code to
|
||||
remote all of the goto jumps in this code?</p>
|
||||
|
||||
<h2>Author’s comments:</h2>
|
||||
|
||||
<h3>What is this? <a name="wit"></a></h3>
|
||||
|
||||
<p>This entry reads from standard input a list of words in any language
|
||||
written with an alphabet and outputs perfect pangrams composed of these
|
||||
words. A perfect pangram is a series of words that contains every letter
|
||||
of the alphabet exactly once.</p>
|
||||
|
||||
<p>Example execution:</p>
|
||||
|
||||
<pre><code>$ grep .. /usr/share/dict/words | ./prog abcdefghijklmnopqrstuvwxyz
|
||||
qualm fjord wiz pyx vs beg kc nth
|
||||
quartz jinx vs fed kc womb glyph
|
||||
quid jamb fez vs pyx kc growl nth
|
||||
quiz fjord pyx vs bag kc mewl nth
|
||||
quiz fjord pyx vs gab kc mewl nth
|
||||
quiz fjord pyx vs gem bawl kc nth
|
||||
quiz fjord pyx vs meg bawl kc nth
|
||||
quiz fjord vex bawl kc gyp ms nth
|
||||
quiz jamb flex vs kc gyp word nth
|
||||
quiz jamb pyx vs kc flog drew nth
|
||||
quiz jamb pyx vs kc frog lewd nth
|
||||
quiz jamb pyx vs kc frog weld nth
|
||||
quiz jamb pyx vs kc golf drew nth
|
||||
quiz jamb pyx vs kc grew fold nth
|
||||
quiz jamb pyx vs kc grow nth fled
|
||||
quiz jamb pyx vs kc growl fed nth
|
||||
</code></pre>
|
||||
|
||||
<p>Rearranging the words into more or less meaningful expressions is the
|
||||
user’s duty.</p>
|
||||
|
||||
<p>This entry implements Algorithm X (<em>Exact cover via dancing links</em>)
|
||||
from Section 7.2.2.1 in fascicle 5C of <em>The Art of Computer
|
||||
Programming</em> by Donald E. Knuth. As of June 2019, you can download an
|
||||
incomplete draft of the fascicle from <a href="https://www-cs-faculty.stanford.edu/~knuth/fasc5c.ps.gz">Knuth’s
|
||||
website</a>.</p>
|
||||
|
||||
<h3>Usage</h3>
|
||||
|
||||
<p><strong><code>prog LETTERS [N]</code></strong>
|
||||
reads allowed words from standard input and writes perfect pangrams
|
||||
with at most <em>N</em> words composed of <em>LETTERS</em> to standard output, one
|
||||
per line. If <em>N</em> is not given, the number of words in the pangrams is
|
||||
unlimited. The characters in <em>LETTERS</em> must not repeat. The length of
|
||||
<em>LETTERS</em> must not exceed 97 characters.</p>
|
||||
|
||||
<p>The exit status is 0 if no error occurred, 1 if the input word list
|
||||
is too long, and 2 if the command-line arguments are missing.</p>
|
||||
|
||||
<h3>Helper scripts</h3>
|
||||
|
||||
<p><strong><code>getwords.sh LANGUAGE_CODE</code></strong>
|
||||
outputs a sorted list of unique words in a given language, one per
|
||||
line. It requires <strong><code>aspell</code></strong> with a dictionary for <em>LANGUAGE_CODE</em>.</p>
|
||||
|
||||
<p><strong><code>wrapprog.py LETTERS [N]</code></strong>
|
||||
works similarly to <strong><code>prog LETTERS [N]</code></strong> except that it merges
|
||||
anagrams before passing them to <strong><code>prog</code></strong> and expands them in
|
||||
<strong><code>prog</code></strong>’s output. This way, it finishes the job faster than
|
||||
<strong><code>prog</code></strong> when there are millions of perfect pangrams. It requires a
|
||||
UTF-8 locale.</p>
|
||||
|
||||
<p><strong><code>de.sh</code></strong>, <strong><code>en.sh</code></strong>, <strong><code>fr.sh</code></strong>, <strong><code>it.sh</code></strong>, <strong><code>pl.sh</code></strong>,
|
||||
and <strong><code>ru.sh</code></strong> output German, English, French, Italian, Polish,
|
||||
and Russian perfect pangrams, respectively. They require a UTF-8
|
||||
locale.</p>
|
||||
|
||||
<h3>Miscellaneous remarks</h3>
|
||||
|
||||
<p>I dare submit this entry to categories <em>algorithms</em>,
|
||||
<em>internationalization</em> (I have not found active use of <strong><code><wchar.h></code></strong>
|
||||
among the programs that won IOCCC in the past), and <em>obscure bugs</em> (see
|
||||
the questions about Linux core dumps below).</p>
|
||||
|
||||
<p>To a casual eye, this entry may look similar to
|
||||
<a href="/years.html#2005_klausler">2005 klausler</a>.
|
||||
However, <strong><code>prog</code></strong> is internationalized and way faster. On my machine,
|
||||
<strong><code>./klausler abcdefghijklmnopqrstuvwxyz</code></strong> printed only 3,965 perfect
|
||||
pangrams until I killed it after 144 hours running, while the
|
||||
equivalent
|
||||
<strong><code>tr A-Z a-z < /usr/share/dict/words | egrep -v '^[^ais]$' | ./prog abcdefghijklmnopqrstuvwxyz | wc -l</code></strong>
|
||||
finished in 62 minutes, reporting 1,640,569,315 perfect pangrams, and
|
||||
the example from section <a href="#wit">What is this?</a> finished in 1.35
|
||||
seconds. On the other hand, <strong><code>prog</code></strong> can only output series of words
|
||||
with nonrepeating characters, unlike <strong><code>klausler</code></strong>.</p>
|
||||
|
||||
<p>In contrast to <strong><code>prog.orig.c</code></strong>, <strong><code>prog.c</code></strong> does not call
|
||||
<strong><code>fflush(stdout)</code></strong> after outputting each line, thus running faster.
|
||||
I am grateful to Witold Jarnicki for suggesting this change.
|
||||
The original <strong><code>prog</code></strong> finished the example above in 100 minutes
|
||||
instead of 62 minutes.</p>
|
||||
|
||||
<p>If your word list contains meaningless one-letter words, pipe
|
||||
<strong><code>grep .. [FILE]</code></strong> through <strong><code>prog</code></strong> to eliminate the pangrams that
|
||||
contain them.</p>
|
||||
|
||||
<p>According to <strong><code>prog</code></strong> and <strong><code>aspell</code></strong>, some languages, e.g. Spanish,
|
||||
have no perfect pangrams.</p>
|
||||
|
||||
<p>Why does <strong><code>yes | ./prog y</code></strong> hang on Linux? Change <strong><code>static int</code></strong> to
|
||||
<strong><code>int</code></strong> and <strong><code>static wchar_t</code></strong> to <strong><code>wchar_t</code></strong>, and recompile
|
||||
<strong><code>prog</code></strong>. Why does <strong><code>yes | ./prog y</code></strong> dump core now? Change the line
|
||||
<strong><code>#define P 9<<23^1</code></strong> to <strong><code>#define P 9<<23</code></strong> and recompile
|
||||
<strong><code>prog</code></strong>. Why does <strong><code>yes | ./prog y</code></strong> exit gracefully?</p>
|
||||
|
||||
<p>Answer: Nf bs Whar 2019, guhf znavsrfg ohtf va <strong><code>tyvop</code></strong> 2.22–2.29:
|
||||
<a href="https://sourceware.org/bugzilla/show_bug.cgi?id=20568">1</a> naq
|
||||
<a href="https://sourceware.org/bugzilla/show_bug.cgi?id=20632">2</a>.</p>
|
||||
|
||||
<p>The source code uses <strong><code>goto</code></strong> three times. While it could easily get
|
||||
by without <strong><code>goto M</code></strong> and <strong><code>goto H</code></strong>, I challenge the adherents of
|
||||
structured programming to refactor <strong><code>goto T</code></strong>, which jumps back into a
|
||||
nested <strong><code>if</code></strong> block.</p>
|
||||
|
||||
<p>With the supplied <strong><code>Makefile</code></strong>, both <strong><code>gcc</code></strong> and <strong><code>clang</code></strong> compile
|
||||
<strong><code>prog.c</code></strong> without warnings in C11 and C99 mode. For a clean
|
||||
compilation with <strong><code>gcc -std=c90</code></strong>, add <strong><code>-Wno-format</code></strong> to
|
||||
<strong><code>CSILENCE</code></strong> in <strong><code>Makefile</code></strong>.</p>
|
||||
|
||||
<hr />
|
||||
|
||||
<TABLE><TR>
|
||||
<TD><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/"><img alt="Creative Commons License" style="border-width:0" src="/png/by-sa-3.0-88x31.png" /></a></TD>
|
||||
<TD><P>© Copyright 1984-2019,
|
||||
<A HREF="/judges.html">Leo Broukhis, Simon Cooper, Landon Curt Noll</A>
|
||||
- All rights reserved<BR>
|
||||
This work is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.</P></TD>
|
||||
<TD> <!--<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a>--></TD>
|
||||
</TR></TABLE>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,84 +0,0 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta name="GENERATOR" content="mkd2html 2.2.0">
|
||||
<meta http-equiv="Content-Type"
|
||||
content="text/html; charset=utf-8"> <link rel="stylesheet"
|
||||
type="text/css"
|
||||
href="../hint.css">
|
||||
<title>Best small program</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Best small program</h1>
|
||||
|
||||
<p>Volker Diels-Grabsch<br>
|
||||
<a href="https://njh.eu">https://njh.eu</a></p>
|
||||
|
||||
<h2>Judges' comments:</h2>
|
||||
|
||||
<h3>To use:</h3>
|
||||
|
||||
<pre><code>make
|
||||
|
||||
./prog < file > file.Z
|
||||
</code></pre>
|
||||
|
||||
<h3>Try:</h3>
|
||||
|
||||
<pre><code>dd bs=1024 count=2048 < /dev/zero | compress > ref.Z
|
||||
dd bs=1024 count=2048 < /dev/zero | ./prog > dg1.Z
|
||||
cmp ref.Z dg1.Z
|
||||
|
||||
compress < ../mills/Shakespeare.txt > ref.Z
|
||||
./prog < ../mills/Shakespeare.txt > dg1.Z
|
||||
cmp ref.Z dg1.Z # Oops, the files differ
|
||||
zcat < dg1.Z | cmp - ../mills/Shakespeare.txt # Decompresses correctly. What is going on?
|
||||
</code></pre>
|
||||
|
||||
<h3>Selected Judges Remarks:</h3>
|
||||
|
||||
<p>Finally, IOCCC now has its own text compression program, and it is short
|
||||
enough to qualify for the Best Short Program award! That said, can you
|
||||
compress it down to a one-liner, still producing files in the UNIX compress format,
|
||||
albeit sacrificing the compression ratio?</p>
|
||||
|
||||
<p>Can you explain why the zero file compresses identically by the entry and the
|
||||
standard compress tool, but the text file compresses differently? The judges know. :)</p>
|
||||
|
||||
<h2>Author’s comments:</h2>
|
||||
|
||||
<h3>A tiny “compress” tool</h3>
|
||||
|
||||
<p><code>Usage:</code></p>
|
||||
|
||||
<pre><code>./prog < guidelines.txt > guidelines.txt.Z
|
||||
</code></pre>
|
||||
|
||||
<p><code>Check for correctness:</code></p>
|
||||
|
||||
<pre><code>zcat guidelines.txt.Z > guidelines2.txt
|
||||
diff -s guidelines.txt guidelines2.txt
|
||||
</code></pre>
|
||||
|
||||
<p><code>Of course you can</code><br>
|
||||
<code>also use 2015/mills2 instead of zcat to verify the output</code><br>
|
||||
<code>file. The achieved compression ratio roughly matches that</code><br>
|
||||
<code>of the classic Unix compress tool. And the source code is</code><br>
|
||||
<code>very compressed, too: It has exactly the same size as the</code><br>
|
||||
<code>paragraph you are reading right now. And exactly the same</code><br>
|
||||
<code>shape. Nevertheless, it is portable C99 code that runs on</code><br>
|
||||
<code>32-bit and 64-bit platforms. It compiles without warnings</code><br>
|
||||
<code>on both GCC and Clang even with -Wextra and -Weverything.</code></p>
|
||||
|
||||
<hr />
|
||||
|
||||
<TABLE><TR>
|
||||
<TD><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/"><img alt="Creative Commons License" style="border-width:0" src="/png/by-sa-3.0-88x31.png" /></a></TD>
|
||||
<TD><P>© Copyright 1984-2019,
|
||||
<A HREF="/judges.html">Leo Broukhis, Simon Cooper, Landon Curt Noll</A>
|
||||
- All rights reserved<BR>
|
||||
This work is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.</P></TD>
|
||||
<TD> <!--<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a>--></TD>
|
||||
</TR></TABLE>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,210 +0,0 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta name="GENERATOR" content="mkd2html 2.2.0">
|
||||
<meta http-equiv="Content-Type"
|
||||
content="text/html; charset=utf-8"> <link rel="stylesheet"
|
||||
type="text/css"
|
||||
href="../hint.css">
|
||||
<title>Most self-aware</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Most self-aware</h1>
|
||||
|
||||
<p>Volker Diels-Grabsch<br>
|
||||
<a href="https://njh.eu">https://njh.eu</a></p>
|
||||
|
||||
<h2>Judges' comments:</h2>
|
||||
|
||||
<h3>To use:</h3>
|
||||
|
||||
<pre><code>make
|
||||
</code></pre>
|
||||
|
||||
<h3>Try:</h3>
|
||||
|
||||
<pre><code>./prog
|
||||
sha512sum prog.c || shasum -a 512 prog.c
|
||||
</code></pre>
|
||||
|
||||
<h3>Selected Judges Remarks:</h3>
|
||||
|
||||
<p>Before running this one, stay calm. Chill out, have a cup of tea. Is this bad?</p>
|
||||
|
||||
<p>To understand what is going on here, please check if you are dreaming, or are you dreaming
|
||||
that you are dreaming? Or is it, wake up from the simulation?</p>
|
||||
|
||||
<h2>Author’s comments:</h2>
|
||||
|
||||
<h3>Almost a hash collision</h3>
|
||||
|
||||
<p>This program prints its own SHA-512 hash. To verify, run:</p>
|
||||
|
||||
<pre><code>sha512sum prog.c; ./prog
|
||||
</code></pre>
|
||||
|
||||
<p>Output:</p>
|
||||
|
||||
<pre><code>43bd0c4381a5078d2850fca9ae5e0647596bcc03cd67d8d973ad02ff35c316c728e7b347ca70abe6c74e745e63646cc7643cb0cffcd3d9a969cbf31a7ce5bf68 prog.c
|
||||
43bd0c4381a5078d2850fca9ae5e0647596bcc03cd67d8d973ad02ff35c316c728e7b347ca70abe6c74e745e63646cc7643cb0cffcd3d9a969cbf31a7ce5bf68
|
||||
</code></pre>
|
||||
|
||||
<p>It started as a complicated “hello world” program that prints some
|
||||
random hex digits instead of “hello world”. Then, the digits were
|
||||
adjusted until the resulting program matched its own SHA-512 hash,
|
||||
via brute force through all 2<sup>512</sup> possibilities. Just kidding.</p>
|
||||
|
||||
<p>This wasn’t brute force, but involved quite some math, using a variant
|
||||
of Jane Doe’s algorithm for generating SHA-512 collisions efficiently.
|
||||
If you still don’t believe me, you are right.</p>
|
||||
|
||||
<p>No crypto was broken in the making of this program!</p>
|
||||
|
||||
<p>The trick is that under the hood, this program is a quine! That is,
|
||||
it is aware of its own source code in string representation. But
|
||||
instead of printing that string, its SHA-512 hash is computed and
|
||||
printed. So two major steps were needed to obfuscate this program:</p>
|
||||
|
||||
<ul>
|
||||
<li>Step 1: Hide the quine</li>
|
||||
<li>Step 2: Hide the SHA-512 computation</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>Note that both steps were achieved mostly algorithmically. The code
|
||||
is otherwise as short and as clean as possible. The implementation is
|
||||
portable C99 code for 32-bit and 64-bit systems that compiles without
|
||||
any warnings on current GCC and Clang versions even with -Wextra and
|
||||
-Weverything. The program is composed of mostly pure functions, and
|
||||
the “#define"s are only used to shorten redundant parts of the code,
|
||||
they are not meant to obfuscate much. Having that out of the way, how
|
||||
are step 1 and step 2 achieved then?</p>
|
||||
|
||||
<p>Regarding step 1, note that the large string literal “`x{bh}ndbq…”
|
||||
in the last line doesn’t look like C code. Moreover, it is too short
|
||||
to contain even a compressed representation of all preceding and
|
||||
subsequent code. Instead, the preceding code including the ‘“’
|
||||
character has already been run through the SHA-512 computation by an
|
||||
external script. The internal state of the SHA-512 computation at
|
||||
that point has been extracted and encoded into the string. This works
|
||||
because the preceding code is exactly 1280 bytes, which is a multiple
|
||||
of 1024 bits, the block size of SHA-512. The program itself then just
|
||||
continues the SHA-512 computation from that point, adding the last
|
||||
1024-bit block, finalizing the hash and printing it. The last block
|
||||
is taken directly from the string literal at runtime, plus the
|
||||
subsequent code which consists of just ‘”’, ‘;’ and ‘\n’. Those 3
|
||||
characters are short enough to be hidden in … well, let’s leave this
|
||||
as an exercise for the reader.</p>
|
||||
|
||||
<p>To summarize, the main trick here is to move the string constant to
|
||||
the end, so the preceding code can be precalculated and the subsequent
|
||||
code is short enough to be hidden anywhere.</p>
|
||||
|
||||
<p>Regarding step 2, almost all macros, functions and variables have
|
||||
meaningless single-letter names. Moreover, the SHA-512 computation is
|
||||
reduced to the special case of adding just one block and immediately
|
||||
finalizing it, where the total size is known in advance.</p>
|
||||
|
||||
<p>But this is not enough! For SHA-512 we need 8 initial hash values and
|
||||
80 round constants, which are very characteristic. If any of them
|
||||
appeared in the source, a quick web search (e.g. for 0x428a2f98d728ae22)
|
||||
would tell the reader immediately what’s going on. Because of the
|
||||
precomputation, the 8 initial hash values are not needed, but the
|
||||
80 round constants are.</p>
|
||||
|
||||
<p>It is almost impossible to hide those 640 bytes in the source.
|
||||
Luckily, the round constants where not chosen by fair dice rolls.
|
||||
They are the fractional parts of certain irrational numbers, namely,
|
||||
the cube roots of the first 80 primes. So the round constrants are
|
||||
calculated at runtime, which makes the program shorter and adds
|
||||
another layer of obfuscation.</p>
|
||||
|
||||
<p>While this sounds like just iterating primes and executing pow(), this
|
||||
turned out to be harder than expected. If we were computing SHA-256,
|
||||
it would have been as simple as that, because the round constants
|
||||
would have been 32-bit values. However, SHA-512 needs the cube roots
|
||||
at 64-bit precision, while double has only 53-bit mantissa precision.</p>
|
||||
|
||||
<p>Using higher-precision floating point is no option here, because the
|
||||
program is meant to be portable C99. And implementing
|
||||
arbitrary-precision arithmetic would have been overkill and would have
|
||||
increased the program size dramatically.</p>
|
||||
|
||||
<p>Instead, the program just multiplies. That is, it computes cubes
|
||||
instead of cube roots. It steps through cube root candidates via
|
||||
bisection, and calculates their cubes until those match the given
|
||||
prime. Thus, we don’t need floating point and can use 64-bit integers
|
||||
instead. But that still doesn’t solve the problem as it poses the
|
||||
following new challenges:</p>
|
||||
|
||||
<ul>
|
||||
<li>Challenge 1: We can multiply at most two 32-bit numbers at once.</li>
|
||||
<li>Challenge 2: We need to calculate with at least 67-bit precision.</li>
|
||||
<li>Challenge 3: We need to truncate the cube root result.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>The reason for challenge 1 is that in portable C99, the result must
|
||||
fit into the same data type. To multiply two 64-bit integers we would
|
||||
need a 128-bit integer for the result. So we can at most multiply two
|
||||
32-bit integers to get the 64-bit product at full precision. While
|
||||
GCC and Clang both support 128-bit arithmetics, that isn’t standard
|
||||
and wouldn’t have been portable C99.</p>
|
||||
|
||||
<p>The reason for challenge 2 is that cube root of the 80th prime is
|
||||
around 7, so we need to compute with 3 bits for the integral part and
|
||||
64 bits for the fractional part.</p>
|
||||
|
||||
<p>The reason for challenge 3 is that the cube of a 67-bit number is
|
||||
3*67=201 bits, but fortunately we only need to check if it is close
|
||||
enough to the actual prime. It won’t be exactly e.g. “409.0000…”
|
||||
anyway, but must be accurate enough to distinguish the correct cube
|
||||
root from its neighbours, e.g. the same number with the 64th
|
||||
fractional digit increased.</p>
|
||||
|
||||
<p>The solution here is to split the 67-bit input into 3 integers of
|
||||
24-bit each. Those are cubed using the multiplication formulas
|
||||
numbers with 3 “digits” of base 2<sup>24</sup> each. During that calculation,
|
||||
at most two values are multiplied at once, and the accuracy of
|
||||
intermediate values is reduced using right-shifts as far as allowable
|
||||
for the computation. Finding woking right-shift values was mostly
|
||||
trial-and-error, producing correct results for the first 80 primes,
|
||||
but not much more. Moreover, for some terms it was possible to leave
|
||||
them out entirely, especially higher-order terms of the last digits.
|
||||
Luckily, in the end, all intermediate results did fit into 64-bit
|
||||
integer addition and multiplication, but that was not obvious at the
|
||||
beginning.</p>
|
||||
|
||||
<p>That’s it.</p>
|
||||
|
||||
<p>Well, I could continue with details about the encoding of the internal
|
||||
state values into the string, how to avoid escaping issues, how to
|
||||
find short notations for various loops, how to get rid of if(), where
|
||||
the final “\”;\n" is stored, how to eliminate intermediate values and
|
||||
arrays by reordering of operations, and so on. But those were all
|
||||
minor issues compared to the challenges described above. What’s
|
||||
documented here is all you need to know to write your own version of
|
||||
that program.</p>
|
||||
|
||||
<p>I’d like to finish with an open problem:</p>
|
||||
|
||||
<p>Is it possible for the cubing and bisection to be implemented with
|
||||
just two parts instead of three? That would simplify the formulas a
|
||||
lot. For example, one could split the 67-value into two 34-bit parts,
|
||||
but whenever one multiplies two of them, one needs to right-shift one
|
||||
of them by at least 4 bits, or both by 2. Or, one splits the 67 bits
|
||||
asymmetrically? I wasn’t able to make it work with 2 parts, but that
|
||||
doesn’t have to mean anything.</p>
|
||||
|
||||
<hr />
|
||||
|
||||
<TABLE><TR>
|
||||
<TD><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/"><img alt="Creative Commons License" style="border-width:0" src="/png/by-sa-3.0-88x31.png" /></a></TD>
|
||||
<TD><P>© Copyright 1984-2019,
|
||||
<A HREF="/judges.html">Leo Broukhis, Simon Cooper, Landon Curt Noll</A>
|
||||
- All rights reserved<BR>
|
||||
This work is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.</P></TD>
|
||||
<TD> <!--<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a>--></TD>
|
||||
</TR></TABLE>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,169 +0,0 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta name="GENERATOR" content="mkd2html 2.2.0">
|
||||
<meta http-equiv="Content-Type"
|
||||
content="text/html; charset=utf-8"> <link rel="stylesheet"
|
||||
type="text/css"
|
||||
href="../hint.css">
|
||||
<title>Best use of space and time</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Best use of space and time</h1>
|
||||
|
||||
<p>Gil Dogon</p>
|
||||
|
||||
<h2>Judges' comments:</h2>
|
||||
|
||||
<h3>To use:</h3>
|
||||
|
||||
<pre><code>make
|
||||
|
||||
./prog < pattern.mc
|
||||
</code></pre>
|
||||
|
||||
<h3>Try:</h3>
|
||||
|
||||
<pre><code>./prog < glider.mc
|
||||
|
||||
./prog < infinite_5x5.mc
|
||||
|
||||
./prog < mosquito5.mc
|
||||
# Press the minus key a few times until you see white specks,
|
||||
# then press space, and the ] key a few times.
|
||||
|
||||
./prog < message.mc
|
||||
</code></pre>
|
||||
|
||||
<h3>Selected Judges Remarks:</h3>
|
||||
|
||||
<p><a href="http://ioccc.org/years-spoiler.html#1991_davidguy">1991/davidguy</a>,
|
||||
<a href="http://ioccc.org/years-spoiler.html#2011_blakely">2011/blakely</a>,…</p>
|
||||
|
||||
<p>…This entry likely concludes the IOCCC category “Cellular automata simulators”,
|
||||
it would be very hard to beat it on its field.</p>
|
||||
|
||||
<p>FYI: The .mc extension stands for Golly macrocell format.</p>
|
||||
|
||||
<p>We truly have not much more to add, except two things:</p>
|
||||
|
||||
<ul>
|
||||
<li><p>If a large sparse pattern is given, at the start of the program
|
||||
you are much more likely to see a blank board rather than the loaded configuration,
|
||||
and to see it, you may need to zoom out a few times.</p></li>
|
||||
<li><p>if you’re using a virtual machine with less than 4 Gb of available
|
||||
model RAM, you’d want to reduce the value of the Z define in the Makefile
|
||||
to 0x10000000, otherwise the entry will promptly crash.</p></li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>In the interest of saving lives, this entry was not fuzzed.</p>
|
||||
|
||||
<h2>Author’s comments:</h2>
|
||||
|
||||
<h3>What is it?</h3>
|
||||
|
||||
<p>Just run with no arguments on an X11-compatible system using one of the supplied input .mc files as standard input. Press ‘-’ several time to zoom out if neccessary, then press space. For acceleration press ‘]’ several times. Thats rather quick dont you think?</p>
|
||||
|
||||
<h3>Historical Timeline.</h3>
|
||||
|
||||
<ul>
|
||||
<li>3.5 Billion years ago: Microrganism life appears on earth</li>
|
||||
<li>3000 BC: Ankh symbol is used in Egyptian hyroglyphs to represent life.</li>
|
||||
<li>1940s: John Von Neumann and Stanislav Ulam are interested in artificial life eventually kick starting research on Cellular Automata.</li>
|
||||
<li>1970: J.H. Conways research discovers a rather minimal set of rules that seems quite interesting. Conways game of life gets published in Gardners Scientific American column.</li>
|
||||
<li>1970’s: According to Knuth, a significant portion of computer time in the world is spent simulating GOL, mostly without the awareness of the people footing the bill.</li>
|
||||
<li>1984: Bill Gosper invents the Hashlife algorithm. His implementation runs on Symbolics Lisp Machines. Unrelated to that, but in a cosmic coincidence the first IOCCC is run!</li>
|
||||
<li>December 2000: Tom Rokicki has a breakthrough implementing Hashlife in C, efficient, stable and production ready for modern machines.</li>
|
||||
<li>2006: First version of Golly by Andrew Trevorrow and Tomas Rokicki. Brings the delights of hashlife to the masses.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>Which brings us to the present moment in time.</p>
|
||||
|
||||
<h3>Full documentation and keys UI.</h3>
|
||||
|
||||
<p>Yes, this program is Conway’s game of life graphic simulator implementing Bill Gosper’s hash life algorithm. The entry’s UI is rather terse and somewhat obfuscated, as it should be. Also, when you accidentally, or not, press any of the keys not specified bellow, some stuff will happen or not, there is no guarantee whatsoever.</p>
|
||||
|
||||
<p>The program should be used as follows:</p>
|
||||
|
||||
<p>The standard input should contain a Golly macrocell format file for a two state automaton. See: http://golly.sourceforge.net/Help/formats.html. Of course only Conways GOL rules are supported, so rules in the files are ignored. The supplied input examples should be quite interesting. Except one, all are taken from Golly’s standard distribution. Information and credits to their designer are in the header comment of each.</p>
|
||||
|
||||
<ul>
|
||||
<li>For image navigation use the arrow keys, and zoom in and out by the +/- keys (without pressing the shift on the +)</li>
|
||||
<li>Space : Toggle start/stop.</li>
|
||||
<li>0(zero): Toggle single step/free run mode.</li>
|
||||
<li>Backspace: Toggle direction for the arrow of time.</li>
|
||||
<li>]: Increase speed (time step) by a factor of 2.</li>
|
||||
<li>[: Decrease speed (time step) by a factor of 2.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>Notice that when going backward in time we ignore current timestep setting and just jump by the amount that was specified before. However going forward is always using the current time step.</p>
|
||||
|
||||
<p>There is a minimal text printout by the program, so you can know what Generation your are in(G:), Internal memory allocated till now (M:), and then which Level is the current macrocell (L) and which zoom Scale you are watching (S).</p>
|
||||
|
||||
<h3>Portability issues.</h3>
|
||||
|
||||
<p>This entry is quite portable hopefully on most modern platforms. It assumes X Window support of course. More than 2GBytes memory is recommended. Configuration through makefile is provided for the display windows size and for the memory usage. The makefile provided configuration (Z=0x20000000) will need a system with 4GB DRAM to run nicely. There are several other salient assumptions though:</p>
|
||||
|
||||
<ul>
|
||||
<li>The entry assumes sizeof(long)=8. However it actually almost avoids using any pointers which are 64 bits this days, in order to save space. On machines with sizeof(long)=4 try adding #define long long long …</li>
|
||||
<li>The entry assumes the X11 displays default visual has 24 bit depth (RGB).</li>
|
||||
<li>The entry assumes little endianity (in only one place though! if you have powerpc you can try and find that place to make it work), but more than this it assumes unaligned loads of 64 bits longs are supported, i.e. it assumes that a long can be loaded from any address that could hold an int otherwise. On non antiquated X86 you’ll be fine.</li>
|
||||
<li>The entry heavily relies on bit complement arithmetic (you dont say!).</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<h3>Limitations.</h3>
|
||||
|
||||
<p>Foremost the entry is very strict, unsafe and picky regarding its input. Anything straying from the standard Golly macrocell legal format will result in undefined behavior, meaning mostly a very defined core dump. Since aforementioned core is rather large by default (about 4Gb) it is highly recommended to ‘set limit coredumpsize 0’ in the shell before playing with fire …</p>
|
||||
|
||||
<p>This entry needs quite a lot of memory to be effective, but by modern standards that is not too much. It does not do any garbage collection, so sooner or later memory will run out, resulting, surprise, surprise, in a core dump. On the plus side, this enables the nice time reversal feature… Anyway the amount of memory used can be controlled by the Makefile -DZ=… argument. With the default Z, the program will support in total about 240 million 64bit longs for the internal memory, which should be sufficient for about 100 million nodes+leafs, as each node is 3 longs and each leaf is 2.
|
||||
<em>Important:</em> Makefile Z define must be apower of two!</p>
|
||||
|
||||
<p>Zooming out past about S58 scale will enter a buggy twilight zone as only long integers are used in the drawing code. Reverting to doubles could have remedied that, but then I’d need to use a lot of ldexps which would take the program over the size limit. Also the program will crash once the universe goes past level 64, though this is simply fixed by modifying a single constant in the program, can you find it ? Anyway 2<sup>58</sup> squared is quite a big enough universe for such a tiny progam to simulate, no ?</p>
|
||||
|
||||
<p>In case cell memory did not run out, or your pattern did not grow too big, and you have had patience enough and did not increase speed too much, you can run out of the history buffer after 33333 steps.</p>
|
||||
|
||||
<p>If all of the above did not happen and result in the expected core dump, you will encounter the final limitation, which is the absence of an escape key. Just close the window or use ^C to exit ….</p>
|
||||
|
||||
<h3>Warnings.</h3>
|
||||
|
||||
<p>Oh don’t get me started on the torrent of litanies gcc emits when used with -Wall ! Does it think I have not memorized by heart C’s precious operator precedence table (or at least have a cheatsheet nearby) that it dares suggest extra parentheses that would take the program over the limit ? And what if I do not use a variable here or there, do I have to pay extra tax for every orphaned variable?</p>
|
||||
|
||||
<p>There is the reasonable warning about some longs printed as ints but its OK too, trust me.</p>
|
||||
|
||||
<p>Then there’s the warning about ‘gets’ being unsafe, which is indeed the only one that appears by default in some versions of gcc linker. Yes Virginia, I know it is unsafe, and I would not recommend anyone using this entry as part of a web daemon, or a spaceship, no, Seriously Dont do that!</p>
|
||||
|
||||
<h3>Obfuscations.</h3>
|
||||
|
||||
<p>Well almost none of the obfuscations are done on purpose, but mostly out of necessity, so all identifiers possible to be one character, are, and they are mercilessly reused. In the main There’s even a nice little identifier first used as a global and then reused (shadowed) in the <em>same</em> scope, and this did not produce even a lile squeak of a warning from the compiler! Macro compression is raher sparingly used, however Magic numbers abound everywhere, and bit twiddling is magical too. Do you know for example why we have unsigned F=-1 ? (No warning there of course :) Hint:Later in the program you have F/=17. Did that make any sense? Further hint: print the resulting F in hexadecimal…</p>
|
||||
|
||||
<p>And yes, the layout hint involves a mix of the most trendy typographical character juxtaposed against a very ancient one, Not to mention the program includes some famous name dropping and documents itself as usual.</p>
|
||||
|
||||
<h3>Further spoilers, engineering, and obfuscation galore.</h3>
|
||||
|
||||
<p>Those wait for you, curious and interpid reader, in the attached apohlife.c file, Which is a partially obfuscated version of the program I wrote with a lot of helpful annotations and comments for myself and posterity, revealing some of the magic under the hood. This program is also a somewhat improved version which uses ldexps in the draw routine and has bigger constants, so it can zoom up to level 1024 if memory allows, and it tries its best to avoid dumping core on the poor user when its memory runs out, rather it enters freeze mode where you can still travel in calculated frozen space-time. Well, its no nice and tame, it even has an escape key, usage message, and to top it all it even avoids gets!</p>
|
||||
|
||||
<h3>Special thanks.</h3>
|
||||
|
||||
<p>To Tom Rockiki, the original co-writer of Golly, a personal friend and hashlife master, who reviewed some earlier versions of the entry and freely shared some of his vast knowledge regarding GOL programming and hashlife. As a free bonus, Tom has allowed me to show his code which is probably the most efficient yet most obfuscated way of calculating the GOL function on a 64 bits 8*8 leaf:</p>
|
||||
|
||||
<p><code>c
|
||||
unsigned long life8x8(long a) { unsigned long aw = a << 1, ae = a >> 1, s0 = aw ^ ae, s1 = aw & ae, hs0 = s0 ^ a, hs1 = (s0 & a) | s1, hs0w8 = hs0 >> 8, hs0e8 = hs0 << 8, hs1w8 = hs1 >> 8, hs1e8 = hs1 << 8, ts0 = hs0w8 ^ hs0e8, ts1 = (hs0w8 & hs0e8) | (ts0 & s0); return (hs1w8 ^ hs1e8 ^ ts1 ^ s1) & ((hs1w8 | hs1e8) ^ (ts1 | s1)) & ((ts0 ^ s0) | a); }
|
||||
</code>
|
||||
This uses only 19 bitwise operations and six shifts to calculate the inner 6x6 next generation bits of the input 8x8 !</p>
|
||||
|
||||
<hr />
|
||||
|
||||
<TABLE><TR>
|
||||
<TD><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/"><img alt="Creative Commons License" style="border-width:0" src="/png/by-sa-3.0-88x31.png" /></a></TD>
|
||||
<TD><P>© Copyright 1984-2019,
|
||||
<A HREF="/judges.html">Leo Broukhis, Simon Cooper, Landon Curt Noll</A>
|
||||
- All rights reserved<BR>
|
||||
This work is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.</P></TD>
|
||||
<TD> <!--<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a>--></TD>
|
||||
</TR></TABLE>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,196 +0,0 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta name="GENERATOR" content="mkd2html 2.2.0">
|
||||
<meta http-equiv="Content-Type"
|
||||
content="text/html; charset=utf-8"> <link rel="stylesheet"
|
||||
type="text/css"
|
||||
href="../hint.css">
|
||||
<title>Best collaborative graphics</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Best collaborative graphics</h1>
|
||||
|
||||
<p>Etienne Duble<br>
|
||||
<a href="https://lig-membres.imag.fr/duble">https://lig-membres.imag.fr/duble</a><br>
|
||||
Twitter: @etienne_duble</p>
|
||||
|
||||
<h2>Judges' comments:</h2>
|
||||
|
||||
<h3>To use:</h3>
|
||||
|
||||
<pre><code>make
|
||||
</code></pre>
|
||||
|
||||
<h3>Try:</h3>
|
||||
|
||||
<pre><code>./prog /tmp/drawing
|
||||
|
||||
(open another window / terminal)
|
||||
|
||||
./prog /tmp/drawing
|
||||
|
||||
(open more terminals....)
|
||||
|
||||
./prog /tmp/drawing
|
||||
</code></pre>
|
||||
|
||||
<h3>Selected Judges Remarks:</h3>
|
||||
|
||||
<p>After starting the program, use the cursor keys, then try some modes, like “p”
|
||||
or “l” (they toggle).</p>
|
||||
|
||||
<h2>Author’s comments:</h2>
|
||||
|
||||
<h3>Introduction</h3>
|
||||
|
||||
<p>This program is a <strong>graphics editor</strong>, running on the terminal.</p>
|
||||
|
||||
<p>It provides <strong>collaborative</strong> features: one can join the drawing session
|
||||
of someone else by just opening the same file!</p>
|
||||
|
||||
<h3>Getting started</h3>
|
||||
|
||||
<p>The program can run on Linux, FreeBSD, Mac OS X, and in most terminals.
|
||||
See section ‘Limits, Portability’ for more info.</p>
|
||||
|
||||
<p>To build, type <code>make</code> (assuming gcc) or <code>make CC=clang</code>.</p>
|
||||
|
||||
<p>Then you can start the program. It expects a file path as its first argument:</p>
|
||||
|
||||
<pre><code>$ ./prog /tmp/drawing
|
||||
</code></pre>
|
||||
|
||||
<p>(If not started this way, <code>prog</code> will refuse to start.)</p>
|
||||
|
||||
<p>If the file does not exist, you will start with a blank drawing.</p>
|
||||
|
||||
<p>If someone else (or another <em>instance of yourself</em>, maybe??) is already
|
||||
editing this file, you will join the session!</p>
|
||||
|
||||
<h3>Edition features</h3>
|
||||
|
||||
<ul>
|
||||
<li><strong>arrow keys</strong>: move cursor.</li>
|
||||
<li><strong>p</strong> key: enter <em>pen</em> mode (type <strong>p</strong> again to return to the default <em>move</em> mode).</li>
|
||||
<li><strong>l</strong> key: enter <em>line</em> mode (type <strong>l</strong> again to validate the line and return to the <em>move</em> mode).</li>
|
||||
<li><strong>c</strong> key: cycle through a small set of different <em>colors</em>.</li>
|
||||
<li><strong>q</strong> key: quit the editor. The drawing is automatically saved.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>The mode and current color is indicated at the bottom left corner of
|
||||
the editor.</p>
|
||||
|
||||
<h3>Bonus features</h3>
|
||||
|
||||
<ul>
|
||||
<li>If you just want to display the content of a drawing file, you do not
|
||||
need to start the editor. Just <strong>execute the drawing file</strong>: <code>./<the-file></code>.
|
||||
(This will fail if someone is still editing the file, though.)</li>
|
||||
<li><code>ls --color <drawing-files...></code> will probably select different colors
|
||||
to indicate whether a drawing file is being edited or not. :-)</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<h3>Obfuscation</h3>
|
||||
|
||||
<p>You will probably have a hard time analyzing this entry, because:</p>
|
||||
|
||||
<ul>
|
||||
<li>Analyzing a system made of several communicating programs is known to
|
||||
be a complex task. (And if you think that observing a single-user drawing
|
||||
session will be a good starting point, read on.)</li>
|
||||
<li>Code coverage of single vs multi-user sessions is more or less the same. :-)</li>
|
||||
<li>Code coverage when using <em>move</em>, <em>pen</em> or <em>line</em> modes is almost the same.
|
||||
(so, is there a Bresenham algorithm somewhere??)</li>
|
||||
<li>Some OS resource values and stdlib call results are inferred.</li>
|
||||
<li>Whenever possible, variable names are reused.</li>
|
||||
<li>You might start analyzing escape code sequences right now, but warming up
|
||||
your brain is actually recommended. Start with a simple question such as:
|
||||
<em>in inverse video mode, will a space char show the foreground color or
|
||||
background color?</em></li>
|
||||
</ul>
|
||||
|
||||
|
||||
<h3>A few more things</h3>
|
||||
|
||||
<p>Did you notice the <code>enum</code> declaration? Could it be just an <code>int</code> or <code>const int</code>
|
||||
variable, instead? (spoiler: http://c-faq.com/ansi/constasconst.html)</p>
|
||||
|
||||
<p>If your keyboard has no arrow keys, you can probably find alternate keys by
|
||||
reading the source code.</p>
|
||||
|
||||
<h3>Limits, Portability</h3>
|
||||
|
||||
<p>I tested the program on several Linux systems, on FreeBSD 12, and on
|
||||
Mac OS X (mojave).</p>
|
||||
|
||||
<p>On Linux, the program uses an OS-specific detail to hide communication
|
||||
artefacts, and avoids the need to remove them on exit.
|
||||
Since this is Linux-specific, it was not possible on FreeBSD or Mac OS.
|
||||
Its behavior is a little more <em>dirty</em> there.</p>
|
||||
|
||||
<p>The program should work on most terminals that support 8-bit colors,
|
||||
inverse video mode, and movement escape codes, e.g. <code>xterm</code>, <code>gnome-terminal</code>,
|
||||
<code>terminator</code>, etc.</p>
|
||||
|
||||
<p>Note: the drawing files <code>prog</code> generates are probably even more portable than
|
||||
<code>prog</code> itself!</p>
|
||||
|
||||
<h3>Drawing area size</h3>
|
||||
|
||||
<p><em>The drawing area size is set at compilation time</em>.</p>
|
||||
|
||||
<p>The default size is the one of my xterm (see top of Makefile).
|
||||
You may set it to the full size of your terminal window by typing:</p>
|
||||
|
||||
<pre><code>$ make fullscreen
|
||||
</code></pre>
|
||||
|
||||
<p>IMPORTANT NOTES:</p>
|
||||
|
||||
<ul>
|
||||
<li><strong>The program compiled with a given size will fail to reload a drawing file
|
||||
with a different size!</strong></li>
|
||||
<li>No scrolling is implemented (sorry but that would be too much for rule2!),
|
||||
so any peer reaching the session should have a terminal with the same
|
||||
size or larger.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<h3>prog.c vs prog.orig.c</h3>
|
||||
|
||||
<p>The file <code>prog.orig.c</code> is the one I submitted.</p>
|
||||
|
||||
<p>Judges proposed a small update: the program was using macro <code>FD_SET</code> inside an
|
||||
expression, which breaks compilation on Mac OS X. Wrapping this macro into a
|
||||
function was enough to fix this compilation issue.</p>
|
||||
|
||||
<p>However, that was not enough to make the program work on Mac OS X. If you analyse
|
||||
the program you will see that it heavily relies on OS resources. And, for this
|
||||
first version of the program, you even needed to increase default <code>sysctl</code>
|
||||
parameters to make it work on FreeBSD (this was the purpose of file <code>check-os.sh</code>).
|
||||
I tried hard to tune Mac OS X the same, but failed.</p>
|
||||
|
||||
<p>The simple fact Judges proposed this update meant they wish it could work on Mac.
|
||||
And if judges wish something, it <strong>has to</strong> be done. ;)
|
||||
So I refactored a little more the program to reduce OS resources consumption.
|
||||
Or maybe not reduce consumption, but consume them differently…
|
||||
And I obtained <code>prog.c</code>. With this version, no need to touch <code>sysctl</code> parameters,
|
||||
and it works on Mac OS X too!
|
||||
I must confess it was challenging to remain below the size limit with this little
|
||||
change.</p>
|
||||
|
||||
<hr />
|
||||
|
||||
<TABLE><TR>
|
||||
<TD><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/"><img alt="Creative Commons License" style="border-width:0" src="/png/by-sa-3.0-88x31.png" /></a></TD>
|
||||
<TD><P>© Copyright 1984-2019,
|
||||
<A HREF="/judges.html">Leo Broukhis, Simon Cooper, Landon Curt Noll</A>
|
||||
- All rights reserved<BR>
|
||||
This work is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.</P></TD>
|
||||
<TD> <!--<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a>--></TD>
|
||||
</TR></TABLE>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,124 +0,0 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta name="GENERATOR" content="mkd2html 2.2.0">
|
||||
<meta http-equiv="Content-Type"
|
||||
content="text/html; charset=utf-8"> <link rel="stylesheet"
|
||||
type="text/css"
|
||||
href="../hint.css">
|
||||
<title>Most in need of debugging</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Most in need of debugging</h1>
|
||||
|
||||
<p>Yusuke Endoh<br>
|
||||
Twitter: @mametter</p>
|
||||
|
||||
<h2>Judges' comments:</h2>
|
||||
|
||||
<h3>To use:</h3>
|
||||
|
||||
<pre><code>make
|
||||
</code></pre>
|
||||
|
||||
<h3>Try:</h3>
|
||||
|
||||
<pre><code>./prog
|
||||
|
||||
gdb ./prog || llvm ./prog
|
||||
</code></pre>
|
||||
|
||||
<h3>Selected Judges Remarks:</h3>
|
||||
|
||||
<p>The purpose of this program is to crash. You’ll want to have memorized “man
|
||||
ascii” when debugging it to reveal its purpose.</p>
|
||||
|
||||
<h2>Author’s comments:</h2>
|
||||
|
||||
<h3>backtrace quine</h3>
|
||||
|
||||
<p>Compile prog.c with no optimization.</p>
|
||||
|
||||
<pre><code>$ gcc -g -O0 -o prog prog.c
|
||||
</code></pre>
|
||||
|
||||
<p>Then, run it. You will see a segmentation fault.</p>
|
||||
|
||||
<pre><code>$ ./prog
|
||||
Segmentation fault (core dumped)
|
||||
</code></pre>
|
||||
|
||||
<p>Let’s debug. As usual, run it under gdb.</p>
|
||||
|
||||
<pre><code>$ gdb ./prog
|
||||
(gdb) run
|
||||
Starting program: /home/.../prog
|
||||
|
||||
Program received signal SIGSEGV, Segmentation fault.
|
||||
0x0000000000000000 in ?? ()
|
||||
</code></pre>
|
||||
|
||||
<p>Okay, check the backtrace.</p>
|
||||
|
||||
<pre><code>(gdb) backtrace
|
||||
#0 0x0000000000000000 in ?? ()
|
||||
#1 0x0000555555555201 in x23 () at prog.c:35
|
||||
#2 0x000055555555571a in x64 () at prog.c:100
|
||||
#3 0x0000555555555747 in x65 () at prog.c:101
|
||||
#4 0x0000555555555774 in x66 () at prog.c:102
|
||||
#5 0x00005555555557ce in x69 () at prog.c:105
|
||||
#6 0x0000555555555828 in x6e () at prog.c:110
|
||||
#7 0x0000555555555747 in x65 () at prog.c:101
|
||||
...
|
||||
</code></pre>
|
||||
|
||||
<p>See the line numbers and lookup the ASCII table.</p>
|
||||
|
||||
<pre><code> 35 = '#'
|
||||
100 = 'd'
|
||||
101 = 'e'
|
||||
102 = 'f'
|
||||
105 = 'i'
|
||||
110 = 'n'
|
||||
101 = 'e'
|
||||
...
|
||||
</code></pre>
|
||||
|
||||
<h4>One more thing</h4>
|
||||
|
||||
<p>The original program can be used as a GDB command file.</p>
|
||||
|
||||
<pre><code>$ gdb -q -x prog.c ./prog | cat
|
||||
Reading symbols from ./prog...done.
|
||||
|
||||
Program received signal SIGSEGV, Segmentation fault.
|
||||
0x0000000000000000 in ?? ()
|
||||
#0 0x0000000000000000 in ?? ()
|
||||
#1 0x0000555555555201 in x23 () at prog.c:35
|
||||
#2 0x000055555555571a in x64 () at prog.c:100
|
||||
#3 0x0000555555555747 in x65 () at prog.c:101
|
||||
#4 0x0000555555555774 in x66 () at prog.c:102
|
||||
#5 0x00005555555557ce in x69 () at prog.c:105
|
||||
#6 0x0000555555555828 in x6e () at prog.c:110
|
||||
#7 0x0000555555555747 in x65 () at prog.c:101
|
||||
...
|
||||
</code></pre>
|
||||
|
||||
<p>By using this, you can confirm that it is actually a quine.</p>
|
||||
|
||||
<pre><code>$ gdb -q -x prog.c ./prog | sed -n -r 's/#.*:([0-9]+)/\1/p' | awk '{printf "%c",$1}' > prog2.c
|
||||
$ diff prog.c prog2.c
|
||||
</code></pre>
|
||||
|
||||
<hr />
|
||||
|
||||
<TABLE><TR>
|
||||
<TD><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/"><img alt="Creative Commons License" style="border-width:0" src="/png/by-sa-3.0-88x31.png" /></a></TD>
|
||||
<TD><P>© Copyright 1984-2019,
|
||||
<A HREF="/judges.html">Leo Broukhis, Simon Cooper, Landon Curt Noll</A>
|
||||
- All rights reserved<BR>
|
||||
This work is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.</P></TD>
|
||||
<TD> <!--<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a>--></TD>
|
||||
</TR></TABLE>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,192 +0,0 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta name="GENERATOR" content="mkd2html 2.2.0">
|
||||
<meta http-equiv="Content-Type"
|
||||
content="text/html; charset=utf-8"> <link rel="stylesheet"
|
||||
type="text/css"
|
||||
href="../hint.css">
|
||||
<title>Most in need of wide space</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Most in need of wide space</h1>
|
||||
|
||||
<p>Edward Giles</p>
|
||||
|
||||
<h2>Judges' comments:</h2>
|
||||
|
||||
<h3>To use:</h3>
|
||||
|
||||
<pre><code>make prog
|
||||
./prog infile.wav outfile.wav
|
||||
</code></pre>
|
||||
|
||||
<h3>Try:</h3>
|
||||
|
||||
<pre><code>play MonodyVocals.wav
|
||||
./prog MonodyVocals.wav
|
||||
play out.wav
|
||||
</code></pre>
|
||||
|
||||
<h3>Selected Judges Remarks:</h3>
|
||||
|
||||
<p>Do you need to pretend enjoying a more spacious living environment than you actually are in,
|
||||
or the great outdoors with mountains and canyons, when recording your outgoing voicemail message?
|
||||
Then don’t waste another minute! Record yourself, send the audio file through this entry,
|
||||
and let your callers be surprised.</p>
|
||||
|
||||
<h2>Author’s comments:</h2>
|
||||
|
||||
<h2>Description</h2>
|
||||
|
||||
<p>This program takes a WAV audio file as input, and applies an artificial reverberation effect to it. It works by feeding the audio through a number of delay lines of various lengths, with various amounts of feedback, to create a smooth echo that fades away over time. A number of parameters of the effect can be altered, to change its sonic characteristics.</p>
|
||||
|
||||
<p>BlockDiagram.png is a block-diagram of the internal structure of the reverb algorithm, which is most likely easier to understand than the source code.</p>
|
||||
|
||||
<p>MonodyVocals.wav is for testing the program: It is an extract of the unprocessed vocals from <a href="https://www.youtube.com/watch?v=B7xai5u_tnk">Monody by TheFatRat, featuring Laura Brehm</a>. The track was published under a free license.</p>
|
||||
|
||||
<p>This program has also been successfully compiled on Windows, with both the <a href="https://bellard.org/tcc/">Tiny C Compiler</a> and Microsoft’s C/C++ compiler. However, the latter reports warnings about loss of information when converting floats, and that <code>void (*)()</code> and <code>void (*)(void)</code> are not the same thing.</p>
|
||||
|
||||
<p>Interesting fact: The size of the program from <code>./iocccsize -i</code> is 2019.</p>
|
||||
|
||||
<p><strong>How to use</strong>:</p>
|
||||
|
||||
<pre><code>./prog input [output [size [dry [wet [damping]]]]]
|
||||
prog.exe input [output [size [dry [wet [damping]]]]]
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li><code>input</code>: The path to the input file. Required.</li>
|
||||
<li><code>output</code>: The path to the output file. Optional, defaults to <code>out.wav</code>.</li>
|
||||
<li><code>size</code>: The perceived size of the room, i.e. the approximate amount of time taken for the echo to fade away. Optional, default 1.6 seconds.</li>
|
||||
<li><code>dry</code>: The amount of unprocessed signal (“dry” in audio terminology) to add to the output. Optional, default 0.3 or 30%.</li>
|
||||
<li><code>wet</code>: The amount of processed (“wet”) signal to add to the output. Optional, default 0.5 or 50%.</li>
|
||||
<li><code>damping</code>: The amount by which high frequencies are damped out as the echo continues. This models the tendency for higher frequencies to be absorbed more than lower ones. Increasing this value causes the echo to be less “harsh” and more “muffled”. Optional, default 0.06 or 6%.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>The values <code>size</code>, <code>dry</code>, <code>wet</code>, and <code>damping</code> can be specified either as decimal numbers (e.g. 0.35) or as percentages (e.g. 35%).</p>
|
||||
|
||||
<p>The default settings are designed to make the effect obvious rather than subtle, so it may be a good idea to adjust the ratio of “dry” to “wet”, to give a more pleasant result:</p>
|
||||
|
||||
<pre><code>./prog in.wav out.wav 1.8 80% 15%
|
||||
</code></pre>
|
||||
|
||||
<p>The program only supports WAV files, and only mostly. Other audio formats, such as MP3, are not supported because they are difficult to decode, but there are free programs such as <code>ffmpeg</code> that can perform the conversion. Some of the advanced features of the WAV file format are also not supported, such as metadata, 3 or more channels, or a bit depth other than 16 bits. Fortunately, WAV files that don’t use these features are very common. The program validates the input file, so passing it a file it can’t process will cause the program to print <code>Bad input file</code> or <code>Wave file must be 16 bits,1-2 channels.</code> depending on what it thinks the problem is. If the output file cannot be opened, the program will print <code>Can't open output file</code>.</p>
|
||||
|
||||
<h2>Assumptions</h2>
|
||||
|
||||
<ol>
|
||||
<li>The code is compiled using the C99 or C11 standard</li>
|
||||
<li>Values are stored in memory in little-endian order</li>
|
||||
<li><code>float</code> is a floating-point type of any size, such that all zero bits represents the number zero</li>
|
||||
<li><code>int16_t</code> and <code>int32_t</code> are 2’s-complement, <code>sizeof(int16_t) == 2</code> and <code>sizeof(int32_t) == 4</code>.</li>
|
||||
</ol>
|
||||
|
||||
|
||||
<h2>Obfuscations</h2>
|
||||
|
||||
<p>The main obfuscation is that digital signal processing algorithms are inherently opaque. They just consist of iterating through a list of floating-point numbers and performing a series of arithmetic operations on each value. Even if this entry were entirely cleaned of all other obfuscation, it will appear to the average programmer to do the following:</p>
|
||||
|
||||
<ul>
|
||||
<li>Read and validate command-line arguments.</li>
|
||||
<li>Read and validate the header portion of the input file, and print some errors if that fails.</li>
|
||||
<li>Write a modified form of that header into the output file.</li>
|
||||
<li>Print the parameters read from the command-line arguments, in a human-readable format.</li>
|
||||
<li>Dynamically allocate some arrays of floating-point numbers, to be used later, with lengths based on the <code>size</code> parameter</li>
|
||||
<li>Read 16-bit audio samples from the input file, converting them to floating-point.</li>
|
||||
<li>For each audio sample, do a weird combination of addition, subtraction, and multiplication by constants, using the previously-allocated arrays to store results between iterations. Write the resulting audio samples to the output file, converting them back to 16-bit integers.</li>
|
||||
<li>Close all of the file handles and free the memory.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>Note that no part of the above description of the code’s function had anything to do with echoes in a room. The purpose of the code is therefore hidden.</p>
|
||||
|
||||
<p>The source code, on cursory inspection, appears to be a jumbled series of characters arranged into three arcs, representing sound waves. The overall layout distracts people from the fact that it is source code rather than only ASCII art, and the random appearance of the code discourages people from trying to decipher it.</p>
|
||||
|
||||
<p><code>#define</code>s are used to abbreviate the code, which has a side effect of making it very slightly harder to read.</p>
|
||||
|
||||
<ul>
|
||||
<li><code>k</code> is just a general-purpose replacement for 16.</li>
|
||||
<li><code>i</code> is the number of delay-lines to allocate and use (currently 24).</li>
|
||||
<li><code>q</code> is an abbreviation for multiplying two elements of <code>w[]</code> together.</li>
|
||||
<li><code>u()</code> is used to display percentages with proper rounding.</li>
|
||||
<li><code>j()</code> imposes an upper limit on a floating-point value, and then negates it. This operation is used eight times.</li>
|
||||
<li><code>D()</code> abbreviates a call to <code>setvbuf</code>, used to speed up file I/O.</li>
|
||||
<li><code>l()</code> is used to clip audio samples to the acceptable range, and convert them to 16-bit integers for output.</li>
|
||||
<li><code>g()</code> handles errors.</li>
|
||||
<li><code>C()</code> contains the bulk of the signal-processing code, as well as a preprocessor obfuscation trick: If you were to read the body of the macro out of context, it will appear to have multiple mismatched square brackets, since <code>[</code> occurs more than <code>]</code>. However, this macro is always invoked with arguments such as <code>C(v],+10)</code> which contain an unmatched <code>]</code> character. When the macro is used in this manner, the extra <code>[</code>s in the macro’s body are matched with the <code>]</code>s added through the first argument.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>Every variable name is one letter long. This allows the code to fit into the IOCCC size limit, and it also makes the purpose of a given variable much less clear. For example, it is not evident from the name of the variable <code>t</code> that it contains pointers to the delay-line arrays.</p>
|
||||
|
||||
<p>As many values as possible have been crammed into arrays. <code>float w[]</code> contains:</p>
|
||||
|
||||
<ul>
|
||||
<li>the input parameters (<code>w[0]</code> to <code>w[3]</code>)</li>
|
||||
<li>various constants needed for the reverb algorithm (<code>w[4]</code> to <code>w[9]</code>)</li>
|
||||
<li>temporary storage of floating-point audio samples (<code>w[10]</code> to <code>w[20]</code>)</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p><code>int32_t f[]</code> is even more multi-purpose. It contains:</p>
|
||||
|
||||
<ul>
|
||||
<li>the header of the input file (<code>f[0]</code> to <code>f[11]</code>)</li>
|
||||
<li>the number of channels in the input file (<code>f[12]</code>)</li>
|
||||
<li>the number of samples of output to generate (<code>f[13]</code>)</li>
|
||||
<li>an end-of-file flag (<code>f[14]</code>)</li>
|
||||
<li>a variable that I forgot the exact purpose of (<code>f[15]</code>)</li>
|
||||
<li>indices into the circular buffers where audio samples should be inserted (<code>f[16]</code> to <code>f[39]</code>)</li>
|
||||
<li>indices where audio samples should be read out (<code>f[40]</code> to <code>f[63]</code>)</li>
|
||||
<li>lengths of the buffers (<code>f[64]</code> to <code>f[87]</code>)</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>All of the strings are concatenated together and “encrypted” by breaking the string into 4-byte chunks (by casting it to an <code>int32_t*</code>), and then XORing the sequence of ints with a known numeric sequence. The “decryption” is not done in-place, to avoid writing over the string constant.</p>
|
||||
|
||||
<p><code>int32_t *c</code> is a pointer that points into <code>f[]</code>, except that it is often incremented or decremented when it is accessed. In order to determine which indices of <code>f[]</code> are being accessed through <code>c</code> at any given point in the code, you would have to locate the places where <code>c</code> is modified.</p>
|
||||
|
||||
<p>In order for this program to read and write WAV files, it needs to parse the header. The necessary information, such as the sample rate and channel count, is defined by the standard to be in somewhat arbitrary locations within the file. Therefore, even if you know which indices of <code>f[]</code> are being accessed, it takes extra effort to look up the WAV file format and determine which field that corresponds to.</p>
|
||||
|
||||
<p>Some array accesses (which make up most of the program) have been replaced with harder-to-interpret equivalents, for example:</p>
|
||||
|
||||
<ul>
|
||||
<li><code>X[1]</code> -> <code>1[X]</code></li>
|
||||
<li><code>f[14]</code> -> <code>10[f+4]</code></li>
|
||||
<li><code>X[0]</code> -> <code>*X</code></li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>Assignments return the assigned value, allowing them to be included into later expressions: <code>10[f]=f[13]<<2; f[1]=20+10[f]+k;</code> is replaced with <code>f[1]=20+(10[f]=f[13]<<2)+k;</code></p>
|
||||
|
||||
<p>This technique is used in conjuction with the comma operator to further obscure where the data ends up. For example,</p>
|
||||
|
||||
<ul>
|
||||
<li><code>12[f]=4[++c]>>k;</code> was incorporated into <code>w[0]*=6[f];</code> to yield <code>w[0]*=(12[f]=4[++c]>>k,6[f]);</code></li>
|
||||
<li><code>w[15]=w[k]=e[12[f]]; w[11]=w[10]=e[1];</code> becomes <code>w[11]=(w[15]=w[k]=e[12[f]],w[10]=e[1]);</code></li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>A few other miscellaneous obfuscations have also been done:</p>
|
||||
|
||||
<ul>
|
||||
<li><code>while(++f[15]<12);</code> is used instead of <code>f[15]=12;</code></li>
|
||||
<li><code>for(s=14[f]=0;s<f[13];s++)</code> is used instead of <code>f[14]=0; for(s=0;s<f[13];s++)</code></li>
|
||||
<li>The line that contains <code>printf("Hello, world!");</code> is mostly commented out.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<hr />
|
||||
|
||||
<TABLE><TR>
|
||||
<TD><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/"><img alt="Creative Commons License" style="border-width:0" src="/png/by-sa-3.0-88x31.png" /></a></TD>
|
||||
<TD><P>© Copyright 1984-2019,
|
||||
<A HREF="/judges.html">Leo Broukhis, Simon Cooper, Landon Curt Noll</A>
|
||||
- All rights reserved<BR>
|
||||
This work is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.</P></TD>
|
||||
<TD> <!--<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a>--></TD>
|
||||
</TR></TABLE>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,105 +0,0 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta name="GENERATOR" content="mkd2html 2.2.0">
|
||||
<meta http-equiv="Content-Type"
|
||||
content="text/html; charset=utf-8"> <link rel="stylesheet"
|
||||
type="text/css"
|
||||
href="../hint.css">
|
||||
<title>Most in need of whitespace</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Most in need of whitespace</h1>
|
||||
|
||||
<p>Joshua Karns<br>
|
||||
Twitter: @jkarnss</p>
|
||||
|
||||
<h2>Judges' comments:</h2>
|
||||
|
||||
<h3>To use:</h3>
|
||||
|
||||
<pre><code>make
|
||||
|
||||
./prog < textfile_that_fits_on_the_screen
|
||||
</code></pre>
|
||||
|
||||
<h3>Try:</h3>
|
||||
|
||||
<pre><code>make test
|
||||
|
||||
./prog < hint.text
|
||||
</code></pre>
|
||||
|
||||
<h3>Selected Judges Remarks:</h3>
|
||||
|
||||
<p>If you typically get lost in mazes, watching this program run might – or might not, YMMV –
|
||||
teach you how to find your way in a maze. The algorithm is well-known but the drawing is amazing!</p>
|
||||
|
||||
<p>Before running, make sure that your terminal can accommodate the whole file. Before running “make test”,
|
||||
make it about 120x40 to be safe, ensuring that you see the @ sign as well as the exclamation mark.</p>
|
||||
|
||||
<p>…Oh, and the first M in YMMV stands for “mazing”.</p>
|
||||
|
||||
<p>A puzzle for the reader: Can you change the program to consider a diagonal movement as one step?</p>
|
||||
|
||||
<h2>Author’s comments:</h2>
|
||||
|
||||
<h2>The Program</h2>
|
||||
|
||||
<p>This program is pretty simple! It performs a breadth first search on the
|
||||
specified graph. The graph can be any ascii text file that has an ‘at’
|
||||
character, which is going to be the starting location, and a ‘!’ character
|
||||
which will be the destination.</p>
|
||||
|
||||
<p>The nodes on this graph that are spaces are connected with any directly
|
||||
adjacent nodes that are also spaces. Nodes that aren’t spaces are not
|
||||
connected with anything.</p>
|
||||
|
||||
<h2>Compiling and Running</h2>
|
||||
|
||||
<p>This program usually compiles under both GCC and clang. Build with:</p>
|
||||
|
||||
<pre><code>$(CC) -std=c99 -o tbfs prog.c
|
||||
</code></pre>
|
||||
|
||||
<p>where $(CC) is cc, gcc, clang, or somet other c compiler.</p>
|
||||
|
||||
<p>You can then run it with:
|
||||
cat <input file> | ./tbfs</p>
|
||||
|
||||
<p>Examples:
|
||||
./tbfs < maze
|
||||
./tbfs < prog.c</p>
|
||||
|
||||
<h2>Issues</h2>
|
||||
|
||||
<ul>
|
||||
<li>The program must be ran in a terminal that supports ANSI escape codes for
|
||||
moving the cursor and changing colors.</li>
|
||||
<li>Segfaults happen sometimes.</li>
|
||||
<li>I don’t know what memergy management there is, if any.</li>
|
||||
<li>Ths program will not compile by an ANSI C compiler: it uses for loops, and
|
||||
it uses C++ style comments. It should compile cleanly using a C99 standard.</li>
|
||||
<li>Breadth first search is slow (and very slow on certain maps), but my A star
|
||||
version of this program is too big.</li>
|
||||
<li>Lots of extra whitespace may be needed if you want to feed in an arbitrary
|
||||
file.</li>
|
||||
<li>It fails to compile on GCC sometimes (according to a friend of mine, I’ve
|
||||
not encountered this myself).</li>
|
||||
<li>The program could be obfuscated much further.</li>
|
||||
<li>The program contains some unused code and data.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<hr />
|
||||
|
||||
<TABLE><TR>
|
||||
<TD><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/"><img alt="Creative Commons License" style="border-width:0" src="/png/by-sa-3.0-88x31.png" /></a></TD>
|
||||
<TD><P>© Copyright 1984-2019,
|
||||
<A HREF="/judges.html">Leo Broukhis, Simon Cooper, Landon Curt Noll</A>
|
||||
- All rights reserved<BR>
|
||||
This work is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.</P></TD>
|
||||
<TD> <!--<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a>--></TD>
|
||||
</TR></TABLE>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,298 +0,0 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta name="GENERATOR" content="mkd2html 2.2.0">
|
||||
<meta http-equiv="Content-Type"
|
||||
content="text/html; charset=utf-8"> <link rel="stylesheet"
|
||||
type="text/css"
|
||||
href="../hint.css">
|
||||
<title>Most functional compiler</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Most functional compiler</h1>
|
||||
|
||||
<p>Ben Lynn<br>
|
||||
<a href="https://crypto.stanford.edu/~blynn/">https://crypto.stanford.edu/~blynn/</a><br>
|
||||
Twitter: @bmxlynn</p>
|
||||
|
||||
<h2>Judges' comments:</h2>
|
||||
|
||||
<h3>To use:</h3>
|
||||
|
||||
<pre><code>make
|
||||
</code></pre>
|
||||
|
||||
<h3>Try:</h3>
|
||||
|
||||
<pre><code>(./prog < fib.hs; cat prog.c) > fib.c
|
||||
cc fib.c -o fib
|
||||
./fib
|
||||
</code></pre>
|
||||
|
||||
<h3>Selected Judges Remarks:</h3>
|
||||
|
||||
<p>A fully functional compiler. The example prints out the 30th Fibonacci number.</p>
|
||||
|
||||
<h2>Author’s comments:</h2>
|
||||
|
||||
<h1>Remarks</h1>
|
||||
|
||||
<p>A Haskell compiler. Supports a subset of Haskell more than large enough to
|
||||
self-host. Like GHC with custom language extensions:</p>
|
||||
|
||||
<ul>
|
||||
<li><p><code>WeDontServeYourType</code>: Compilation failing because of inscrutable type
|
||||
checking rules? Confused by the ever-growing mire of extensions to the type
|
||||
system? The solution is simple: no type checking. It also means no
|
||||
typeclasses. Just pass dictionaries explicitly.</p></li>
|
||||
<li><p><code>ZeroPauseGarbageCollection</code>: Instead of disruptive stop-the-world garbage
|
||||
collection, we only tidy up when the world stops of its own accord, that is,
|
||||
on program termination.</p></li>
|
||||
<li><p><code>OneHundredPercentPure</code>: Gone is the catch-all lawless IO monad. And no
|
||||
trace of those scary <code>unsafeThisAndThat</code> functions. All functions must be pure.</p></li>
|
||||
<li><p><code>SyntaxForTheMasses</code>: See below.</p></li>
|
||||
</ul>
|
||||
|
||||
|
||||
<h2>Demos</h2>
|
||||
|
||||
<p>Build the compiler:</p>
|
||||
|
||||
<pre><code>$ $(CC) -o prog prog.c
|
||||
</code></pre>
|
||||
|
||||
<h3>Fibonacci numbers</h3>
|
||||
|
||||
<p>Test the compiler on <code>fib.hs</code>:</p>
|
||||
|
||||
<pre><code>$ (./prog < fib.hs ; cat prog.c) > fib.c
|
||||
</code></pre>
|
||||
|
||||
<p>Compiling the output produces a binary that prints the 30th Fibonacci number.</p>
|
||||
|
||||
<p>The file <code>ghcfib.hs</code> includes <code>fib.hs</code> with some glue code, and shows GHC
|
||||
also accepts our subset of Haskell:</p>
|
||||
|
||||
<pre><code>$ ghc ghcfib.hs
|
||||
</code></pre>
|
||||
|
||||
<h3>Self-hosting compiler</h3>
|
||||
|
||||
<p>To avoid spoiling this entry by revealing the original Haskell source, we
|
||||
instead provide <code>hint.hs</code>, the output of a certain stage of the compiler when
|
||||
run on itself. This intermediate output is hopefully difficult to understand,
|
||||
yet is accepted by our compiler:</p>
|
||||
|
||||
<pre><code>$ (./prog < hint.hs ; cat prog.c) > hint.c
|
||||
</code></pre>
|
||||
|
||||
<p>The output program behaves like the compiler itself.</p>
|
||||
|
||||
<p>Unlike the original source from which it is derived, GHC fails to compile
|
||||
<code>hint.hs</code>. This is because values have been replaced with their Scott encodings
|
||||
by this stage, which messes up type-checks; we’d need equirecursive types for
|
||||
it to work.</p>
|
||||
|
||||
<h3>Regexes</h3>
|
||||
|
||||
<p>The file <code>lol.hs</code> contains an adaptation of <a href="https://www.cs.dartmouth.edu/~doug/nfa.pdf">Doug McIlroy’s elegant code from
|
||||
“Enumerating the strings of regular
|
||||
languages”</a>. We exercise it by
|
||||
showing the first entries of the length-ordered list of all strings consisting
|
||||
of the characters a and b that contain an even number of a’s.</p>
|
||||
|
||||
<pre><code>$ (./prog < lol.hs ; cat prog.c) > lol.c
|
||||
</code></pre>
|
||||
|
||||
<p>A GHC wrapper is provided:</p>
|
||||
|
||||
<pre><code>$ ghc ghclol.hs
|
||||
</code></pre>
|
||||
|
||||
<h3>Strongly-connected components</h3>
|
||||
|
||||
<p>See <code>scc.hs</code> (and its GHC wrapper <code>ghcscc.hs</code>) for an elegant way to print the
|
||||
strongly-connected components of a graph in reverse topological order.</p>
|
||||
|
||||
<pre><code>$ (./prog < scc.hs ; cat prog.c) > scc.c
|
||||
</code></pre>
|
||||
|
||||
<p>It expects the input to be in a similar format as a previous entry
|
||||
(2018 vokes). Indeed, obtain the 2018 winners, and run:</p>
|
||||
|
||||
<pre><code>$ ./scc < 2018/vokes/example-1.txt
|
||||
$ ./scc < 2018/vokes/example-2.txt
|
||||
</code></pre>
|
||||
|
||||
<p>The output should agree, though our program omits line numbers and does not
|
||||
sort entries within a line. (Also, our program only treats spaces as
|
||||
whitespace, and supports any nonspace character in a vertex name.)</p>
|
||||
|
||||
<h2>Syntax</h2>
|
||||
|
||||
<p>Some claim Haskell syntax is frightening because braces and semicolons are
|
||||
optional. Some complain about a zoo of twisty little operators, all alike. Our
|
||||
compiler addresses such concerns by making braces and semicolons compulsory,
|
||||
disabling layout-based rules, making every operator left-associative with same
|
||||
precedence, and only defining 6 primitive functions:</p>
|
||||
|
||||
<pre><code>: + - * / <=
|
||||
</code></pre>
|
||||
|
||||
<p>The arithmetic operations behave as they do in C on unsigned ints, and <code>(:)</code> is
|
||||
Haskell’s cons function. Like Haskell and unlike C, <code>(<=)</code> returns a Bool
|
||||
(which are Church booleans behind the scenes) and not an Int.</p>
|
||||
|
||||
<p>No ifs, ands, or buts. Define them yourself. No tuples. Who needs them when
|
||||
algebraic data types delivers a better product? There is no do. No where. No
|
||||
list comprehensions. No unary operators.</p>
|
||||
|
||||
<p>Let expressions and operator sections are supported.</p>
|
||||
|
||||
<p>Within global scope or a let expression, each definition can only refer to
|
||||
itself or previous definitions. This implies we can only achieve mutual
|
||||
recursion by having one function pass itself to others.</p>
|
||||
|
||||
<h2>Caveats</h2>
|
||||
|
||||
<p>The only primitive type is <code>Int</code>. In particular, they represent characters. To
|
||||
interoperate with GHC, our compiler treats any undefined functions as the
|
||||
identity function, so that we may freely use <code>ord</code> and <code>chr</code> (or <code>fromEnum</code> and
|
||||
<code>toEnum</code>) to make our code acceptable to both compilers.</p>
|
||||
|
||||
<p>The main function is the last function to successfully parse. It should have
|
||||
type <code>[Int] -> [Int]</code>, and our compiler treats this like the function passed
|
||||
into <code>Prelude.interact</code>, that is, the entire standard input is passed to this
|
||||
function, and the result is printed to standard output.</p>
|
||||
|
||||
<p>Bool must be defined as:</p>
|
||||
|
||||
<pre><code>data Bool = True | False;
|
||||
</code></pre>
|
||||
|
||||
<p>so that it matches the Scott-encoded booleans internally used by the primitive
|
||||
function <code>(<=)</code>.</p>
|
||||
|
||||
<p>The alternatives in a case expressions must list every data constructor in
|
||||
the order they are defined. For example, if we have:</p>
|
||||
|
||||
<pre><code>data Foo a b = Bar a | Baz | Qux Int [b]
|
||||
</code></pre>
|
||||
|
||||
<p>then a case expression that examines a term of this type must have the form:</p>
|
||||
|
||||
<pre><code>case x of
|
||||
{ Bar a -> ...
|
||||
; Baz -> ...
|
||||
; Qux n bs -> ...
|
||||
}
|
||||
</code></pre>
|
||||
|
||||
<p>We stress braces and semicolons are required. Our fussy parser treats
|
||||
semicolons as separators, not terminators.</p>
|
||||
|
||||
<p>The effect of erroneous input is undefined. It may be best to develop with GHC,
|
||||
but even then, be mindful of changes needed because of issues caused by the
|
||||
uniform operator precedence and the touchy format of case alternatives.</p>
|
||||
|
||||
<h2>Why?</h2>
|
||||
|
||||
<p>One-letter variable names abound in IOCCC entries, and for good reason. These
|
||||
tiny pieces of confetti are hard to read, and leave room for more code. Then
|
||||
why not go further and use zero-letter variable names? That is, tacit
|
||||
programming or point-free style.</p>
|
||||
|
||||
<p>I had been playing with an algorithm devised by Oleg Kiselyov that effortlessly
|
||||
and efficiently eliminates those pesky variables, leaving behind terms composed
|
||||
from a small set of combinators. No need for lambda lifting or supercombinators.</p>
|
||||
|
||||
<p>By adding a handful of lines of mostly parsing code, we get a Haskell compiler,
|
||||
or rather, a compiler that accepts a subset of Haskell sufficiently large to
|
||||
self-host. You might say I wrote a tool for this contest, then ran it on itself
|
||||
to make an entry for it.</p>
|
||||
|
||||
<h2>Obfuscation techniques</h2>
|
||||
|
||||
<p>Even with Kiselyov’s algorithm and some term rewriting, the compiler only fit
|
||||
after compression, which naturally obfuscates the code. More tricks were needed
|
||||
to fit within the size limits.</p>
|
||||
|
||||
<ul>
|
||||
<li>One-letter variables, until enough is done to banish variables completely.</li>
|
||||
<li>Ipse dixit. Near the end, the code declares itself to be an obfuscated
|
||||
program. (There is also an exhortation intended for the judges in a similar
|
||||
format earlier in the source.)</li>
|
||||
<li>Typical C mischief: pre- and post-increment, commutative array indexing,
|
||||
ternary operators, and so on.</li>
|
||||
<li>Huffman coding.</li>
|
||||
<li>Base-85 because high bits are frowned upon.</li>
|
||||
<li>Mixed radix encoding to game iocccsize. From a past winner (2018 bellard),
|
||||
it seems 9 11 12 32 are the only whitespace octets that may appear verbatim in
|
||||
string literals.</li>
|
||||
<li>Choosing what to encode in Huffman/base-85 and what to encode in mixed radix
|
||||
was a delicate balancing act. In the end, I only had a few bytes to spare,
|
||||
which I spent on gratuitous confusion.</li>
|
||||
<li>The effects of some functions depend on the order their arguments are
|
||||
evaluated, yet the program works either way. Why?</li>
|
||||
<li>Ugly macros for the runtime system’s jump table for lazy reduction. A
|
||||
previous winner (2013 endoh1) has a cuter solution, which I avoid because of
|
||||
originality concerns and also because my combinators are compressed.</li>
|
||||
<li>Relies on the inability of C comments to nest.</li>
|
||||
<li>Cheaper and more complicated to print comma-separated ints (and header and
|
||||
footer) in C.</li>
|
||||
<li>Primitive functions use a trick described in depth by <a href="https://www.cs.york.ac.uk/fp/reduceron/jfp-reduceron.pdf">Naylor and Runciman,
|
||||
“The Reduceron reconfigured and
|
||||
re-evaluated”</a>. We
|
||||
represent the integer n with a term equivalent to <code>Y(BT)n</code>; it works because
|
||||
<code>Y(BT)ne = e(Y(BT)n)</code>.</li>
|
||||
<li>A sentinel in the heap often confused me, so ought to confuse others.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>Other obfuscation techniques are better appreciated after decoding the
|
||||
compiler. See <code>hint.hs</code>.</p>
|
||||
|
||||
<ul>
|
||||
<li>Mercilessly point-free. Everything is a combinator.</li>
|
||||
<li>Scott encoding. Everything is a combinator.</li>
|
||||
<li>Sum types are oddly ordered and may even contain unused data constructors
|
||||
to reduce code duplication. (In C terms, unions may have extra fields, and
|
||||
they’re ordered in such a way so we can reuse code to access certain fields.)</li>
|
||||
<li>The <code>undefined</code> function compiles to the <code>(.)</code> function.</li>
|
||||
<li>Replaced the only <code>(&&)</code> with <code>(||)</code> and <code>not</code> using De Morgan’s law, which
|
||||
makes some comparisons less comprehensible.</li>
|
||||
<li><a href="http://okmij.org/ftp/tagless-final/ski.pdf">Kiselyov only published “λ to SKI, Semantically” last
|
||||
year</a>.</li>
|
||||
<li>Recursion via the fixpoint (Y) combinator, which is represented by the
|
||||
variable “” during one stage of compilation.</li>
|
||||
<li>Involves theory that may be less familiar to C programmers: maps, folds,
|
||||
parser combinators, lambda calculus, bracket abstraction, denotational
|
||||
semantics, etc.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<h2>Warnings</h2>
|
||||
|
||||
<p>On my system, it compiles cleanly with <code>-Wall</code> with older standards, e.g.
|
||||
<code>-std=c89</code>, but less cleanly if <code>-pedantic</code> is also supplied.</p>
|
||||
|
||||
<p>Compiling the compiler output with <code>-Wall</code> triggers a warning about a
|
||||
strange-looking comment.</p>
|
||||
|
||||
<h2>Behind-the-scenes commentary</h2>
|
||||
|
||||
<p><a href="https://crypto.stanford.edu/~blynn/compiler/ioccc.html">My website reveals how this compiler
|
||||
works.</a></p>
|
||||
|
||||
<hr />
|
||||
|
||||
<TABLE><TR>
|
||||
<TD><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/"><img alt="Creative Commons License" style="border-width:0" src="/png/by-sa-3.0-88x31.png" /></a></TD>
|
||||
<TD><P>© Copyright 1984-2019,
|
||||
<A HREF="/judges.html">Leo Broukhis, Simon Cooper, Landon Curt Noll</A>
|
||||
- All rights reserved<BR>
|
||||
This work is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.</P></TD>
|
||||
<TD> <!--<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a>--></TD>
|
||||
</TR></TABLE>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,651 +0,0 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta name="GENERATOR" content="mkd2html 2.2.0">
|
||||
<meta http-equiv="Content-Type"
|
||||
content="text/html; charset=utf-8"> <link rel="stylesheet"
|
||||
type="text/css"
|
||||
href="../hint.css">
|
||||
<title>Most in need to be tweeted</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Most in need to be tweeted</h1>
|
||||
|
||||
<p>Christopher Mills<br>
|
||||
Twitter: @MisterXopher</p>
|
||||
|
||||
<h2>Judges' comments:</h2>
|
||||
|
||||
<h3>To use:</h3>
|
||||
|
||||
<pre><code>make
|
||||
|
||||
make cpclean
|
||||
# Let this run for about about an hour and then kill it
|
||||
./prog Shakespeare.txt
|
||||
./prog < $(< ls -1tr cp* | tail -1) | head -100
|
||||
</code></pre>
|
||||
|
||||
<h3>Try:</h3>
|
||||
|
||||
<pre><code># If you are on a 64-bit system:
|
||||
|
||||
make test-64bit
|
||||
|
||||
less IOCCC-Rules-Guidelines.output.txt
|
||||
|
||||
less IOCCC-hints.output.txt
|
||||
|
||||
less Eugene_Onegin.output.txt
|
||||
|
||||
# However, as the binary model files used to produce the output are in an implementation-specific format,
|
||||
# your mileage may vary.
|
||||
</code></pre>
|
||||
|
||||
<h3>Selected Judges Remarks:</h3>
|
||||
|
||||
<p>Can a machine learn?</p>
|
||||
|
||||
<p>Some say so.</p>
|
||||
|
||||
<p>But can a machine learn to write like Shakespeare,</p>
|
||||
|
||||
<p>or tweet like <a href="http://trumptwitterarchive.com">Trump</a>?</p>
|
||||
|
||||
<p>Can it write rules and guidelines for the IOCCC?</p>
|
||||
|
||||
<p>You decide. :-)</p>
|
||||
|
||||
<h2>Author’s comments:</h2>
|
||||
|
||||
<h1>Welcome to OMLET! 🍳</h1>
|
||||
|
||||
<p>OMLET is the <em>Obfuscated Machine Learning Environment Toolkit</em>, a
|
||||
micro-framework for experimenting with <a href="https://en.wikipedia.org/wiki/Recurrent_neural_network">recurrent neural networks</a>. OMLET
|
||||
lets you build, train and evaluate <a href="https://en.wikipedia.org/wiki/Deep_learning">deep neural networks</a>. Why invest
|
||||
hours reading documentation and megabytes of disk space on a full-featured DNN
|
||||
framework like <a href="https://www.tensorflow.org">TensorFlow</a> or <a href="http://torch.ch">Torch</a> when you can have full RNN
|
||||
functionality in less than 4 KB!</p>
|
||||
|
||||
<p>OMLET has the following features:</p>
|
||||
|
||||
<ul>
|
||||
<li>User-programmable network configurations and hyperparameters</li>
|
||||
<li>Support for various types of recurrent and feed-forward neural networks
|
||||
including vanilla RNNs, LSTMs and GRUs of depths of up to 99 layers</li>
|
||||
<li>No limit on parameter size (except for those imposed by the system)</li>
|
||||
<li>Training and inference modes, with periodic checkpointing</li>
|
||||
<li>Advanced <a href="https://arxiv.org/abs/1412.6980">Adam</a> optimizer with <a href="https://www.fast.ai/2018/07/02/adam-weight-decay/">weight decay</a> for simplified
|
||||
training</li>
|
||||
<li>Hyperparamer support for batch sizing, learning rate schedule, weight
|
||||
decay and gradient clipping</li>
|
||||
<li>Easily extensible (requires some expertise in the C programming language)</li>
|
||||
<li>Friendly markdown documentation</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>OMLET is based on <a href="https://cs.stanford.edu/people/karpathy/">Andrej Karpathy’s</a> <a href="https://github.com/karpathy/char-rnn">character-level language model</a>
|
||||
as described in his blog post <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/"><em>The Unreasonable Effectiveness of Recurrent
|
||||
Neural Networks</em></a>. I’ve included a small sample dataset to use for
|
||||
training, but you can have even more fun by downloading some larger datasets:</p>
|
||||
|
||||
<ul>
|
||||
<li><a href="https://cs.stanford.edu/people/karpathy/char-rnn/linux_input.txt">Linux kernel source</a></li>
|
||||
<li><a href="https://cs.stanford.edu/people/karpathy/char-rnn/warpeace_input.txt"><em>War and Peace</em></a></li>
|
||||
<li><a href="https://github.com/ryanmcdermott/trump-speeches">Donald Trump’s 2016 campaign speeches</a></li>
|
||||
</ul>
|
||||
|
||||
|
||||
<h2>Getting started with OMLET</h2>
|
||||
|
||||
<p>OMLET has three operating modes:</p>
|
||||
|
||||
<ul>
|
||||
<li><em>Training</em>, which trains a network from scratch</li>
|
||||
<li><em>Continuation</em>, which trains starting from an existing checkpoint</li>
|
||||
<li><em>Inference</em>, which uses an existing trained network to make predictions</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>For your first OMLET experiment, we will try training a simple single-level
|
||||
RNN to write some <a href="https://en.wikipedia.org/wiki/William_Shakespeare">Shakespere</a> plays. Start by typing</p>
|
||||
|
||||
<pre><code>make
|
||||
</code></pre>
|
||||
|
||||
<p>After it builds, train it using</p>
|
||||
|
||||
<pre><code>./prog shakespere.txt
|
||||
</code></pre>
|
||||
|
||||
<p>This will immediately start outputting gibberish to the output, e.g.</p>
|
||||
|
||||
<pre><code>./prog shakespere.txt
|
||||
sins ohennAu
|
||||
T-teooclelp tiThoWy
|
||||
|
||||
g
|
||||
nlakuafy
|
||||
e
|
||||
sselW usnsofueB Aoee pasfUsuslhe ooM ot Wou moy
|
||||
me neltAl -no IoyI mhuyakse inT-l chu ghenn ffo? fnsoe yhyye
|
||||
ue nnfrlass heUthole saounlcesyee pee
|
||||
t,
|
||||
T0:0% 3.210888
|
||||
o',,vU
|
||||
An ,hTf lnm Far rur:s moilt WoEgrv wonds mith Aog thernw
|
||||
Rni So
|
||||
co Nnd :
|
||||
For an bImy pgafoun:
|
||||
Wf'r hom wortiverita
|
||||
int fod mous Eheledet,
|
||||
Tho he theket nonS wnu-ang dorlaMSp
|
||||
nrocWiSe tflg 'o.
|
||||
T0:0% 2.995950
|
||||
d whecedhencrysesil yr bn,
|
||||
we hh y thiwt
|
||||
|
||||
|
||||
hut fithlot,
|
||||
Fmdy s he alt
|
||||
Vh th no dh foud bobt werw:s Aotnf Fhwi't whe, eusu
|
||||
|
||||
lhh thele wewcond ary soupfy wind tDont couc ths:
|
||||
er fucwald oncli hen bos, f
|
||||
T0:1% 2.878945
|
||||
</code></pre>
|
||||
|
||||
<p>The gibberish is the networks attempt to write a play. So far, it’s not very
|
||||
successful! Between the chunks of gibberish are training progress reports
|
||||
that look like</p>
|
||||
|
||||
<pre><code>T0:1% 2.878945
|
||||
</code></pre>
|
||||
|
||||
<p>This tells you that we are 1% through training epoch 0, and that the training
|
||||
<a href="https://en.wikipedia.org/wiki/Loss_function">loss</a> is about 2.88. As the training continues, the training loss will
|
||||
reduce and the generated snippets will improve quickly:</p>
|
||||
|
||||
<pre><code>ses, kuth
|
||||
LAs of the wish,
|
||||
As, I nos you,
|
||||
Yov not to nalll,
|
||||
Tr tot wonds.
|
||||
|
||||
First Sondy, llt lrte, our, tw.
|
||||
|
||||
First.
|
||||
|
||||
BRUTUS:
|
||||
Helsting the kith gops of hoch Whay, fars surd what to,
|
||||
The cownens golt te.
|
||||
T0:12% 2.259114
|
||||
eplerrotrur tandans one wiok thy or thach and cullice ded yourssting
|
||||
And wours:
|
||||
Whed ur surt.
|
||||
|
||||
SINENIUS:
|
||||
On we lain bith
|
||||
rerytund: tich lon hyivetetgor.
|
||||
|
||||
VOLUONIA:
|
||||
He brich nom dove worthan then wise,
|
||||
T0:13% 2.254926
|
||||
</code></pre>
|
||||
|
||||
<p>It’s already started to figure out things about Shakespere plays – how to
|
||||
spell short English words, how long lines tend to be, and that characters take
|
||||
turns speaking with their names capitalized. The training loss has dropped to
|
||||
2.25 and the improvement is noticeable.</p>
|
||||
|
||||
<p>Eventually, we will finish with the training data set and move on to a
|
||||
validation cycle:</p>
|
||||
|
||||
<pre><code>y fath onother,
|
||||
I sucess,
|
||||
For I me the west crare.
|
||||
|
||||
TRANIO:
|
||||
Whow and'd have not to had you you one in my lapteny she very ame come me a gut and shourd aghir you as ignested; shend to make I strem
|
||||
To h
|
||||
T0:99% 1.924063
|
||||
notnce gaud and is nicked thou day, ha the dusing you disaid: in thim, you things in ere thee thus erile Iht that tare theme my hast thesp thou shay: thou not eaten-or-ho-bess resing: I the but had d
|
||||
T0:99% 1.923128
|
||||
V0:0% 1.678907
|
||||
V0:9% 1.700527
|
||||
V0:18% 1.733179
|
||||
V0:27% 1.714891
|
||||
V0:36% 1.716672
|
||||
V0:45% 1.782946
|
||||
V0:54% 1.835629
|
||||
V0:63% 1.876108
|
||||
V0:72% 1.906814
|
||||
V0:81% 1.924096
|
||||
V0:90% 1.954492
|
||||
V0:99% 1.969287
|
||||
serfs you'll alliencseard:
|
||||
We
|
||||
got you? before
|
||||
I say.
|
||||
|
||||
Farstred dentlentecaly, sir, I it one bosticield
|
||||
All me the backnour mino,
|
||||
Whith capitaned mid! but stell the ifvemion
|
||||
Willerity.
|
||||
|
||||
First Cumfol of
|
||||
T1:0% 1.885619
|
||||
</code></pre>
|
||||
|
||||
<p>Validation cycles are used to test the network to see if it has learned to
|
||||
generalize – how well performs on data it hasn’t seen before (as opposed to
|
||||
the training data that the network will see many times as it trains). Progress
|
||||
on the validation set is also displayed with a validation progress report that
|
||||
looks like</p>
|
||||
|
||||
<pre><code>V0:36% 1.716672
|
||||
</code></pre>
|
||||
|
||||
<p>which means we are 36% of the way through the validation for epoch 0, and the
|
||||
validation loss is about 1.72. Comparing the validation loss and the
|
||||
training loss will give you an idea of how well the network is learning and
|
||||
can let you know if the network is <a href="https://en.wikipedia.org/wiki/Overfitting">overfitting or underfitting</a>.</p>
|
||||
|
||||
<p>As part of the training process, the data set (which for OMLET is the
|
||||
<code>shakespere.txt</code> file you gave on the command line) is divided into
|
||||
<a href="https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets">training and validation sets</a> (by default, 95% of the data is used for
|
||||
training, but like most OMLET parameters, you can change this at compile-time).</p>
|
||||
|
||||
<p>At the end of the validation run, OMLET writes out a checkpoint file with a
|
||||
name like <code>cp01_1.970</code>. This saves the state of the run at the start of the
|
||||
epoch 1, after computing a validation loss of 1.970. The checkpoint is
|
||||
helpful if you need to stop and restart training. You can stop training by
|
||||
doing <code>Control-C</code>.</p>
|
||||
|
||||
<p>You can continue training from a previous checkpoint by providing the
|
||||
checkpoint file name as the second parameter, for example:</p>
|
||||
|
||||
<pre><code>./prog shakespere.txt cp01_1.970
|
||||
</code></pre>
|
||||
|
||||
<p>After the validation cycle finishes, OMLET begins the next epoch by restarting
|
||||
training at the beginning of the training set. Training continues forever,
|
||||
until you quit it with <code>Control-C</code>. You should monitor the checkpoints to
|
||||
see that the validation loss continues to drop. If it rises, the network has
|
||||
probably started to overfit on the training data.</p>
|
||||
|
||||
<p>Once you’ve trained the validation loss as low as it will go, you can use
|
||||
OMLET to run the network in inference mode which uses the frozen checkpoint
|
||||
parameters to generate data. Inference mode takes the checkpoint file as
|
||||
<em>standard input</em> (not on the command line) and hence must be run with a
|
||||
command like</p>
|
||||
|
||||
<pre><code>./prog < cp55_1.807
|
||||
</code></pre>
|
||||
|
||||
<p>Running it produces an infinite amount of generated output, until you hit
|
||||
<code>Control-C</code> to stop it.</p>
|
||||
|
||||
<p>Note that if you decide to change networks or use a different input file, you
|
||||
will want to delete all the checkpoint files because the format depends on
|
||||
both the network and the input – using the wrong checkpoint is likely to
|
||||
cause a crash.</p>
|
||||
|
||||
<h2>Experimenting with different networks</h2>
|
||||
|
||||
<p>The default network for OMLET (the one you get if you type <code>make</code> with the
|
||||
executable name <code>prog</code>) is the simplest recurrent neural network. It looks
|
||||
like</p>
|
||||
|
||||
<pre><code>h = tanh(Wxh * x + Whh * h' + Bh)
|
||||
y = Why * h + By
|
||||
</code></pre>
|
||||
|
||||
<p>where</p>
|
||||
|
||||
<ul>
|
||||
<li><code>x</code> is the <em>input vector</em></li>
|
||||
<li><code>y</code> is the <em>output vector</em></li>
|
||||
<li><code>h</code> is the <em>hidden state vector</em></li>
|
||||
<li><code>h'</code> is the previous value of <code>h</code></li>
|
||||
<li><code>Wxh</code>, <code>Whh</code>, and <code>Why</code> are <em>weight matrices</em></li>
|
||||
<li><code>Bh</code> and <code>By</code> are <em>bias vectors</em>.</li>
|
||||
<li><code>tanh</code> is the <a href="https://en.wikipedia.org/wiki/Hyperbolic_function">hyperbolic tangent function</a></li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>The <code>W</code>’s and <code>B</code>’s are the trainable parameters of the network, and the
|
||||
process of training is optimizing the values of these parameters to minimize
|
||||
the loss of the network across the training set.</p>
|
||||
|
||||
<p>It is the presence of the hidden state vector that allows the RNN to
|
||||
“remember” the past. We can see what would happen if we removed this hidden
|
||||
state. If you type</p>
|
||||
|
||||
<pre><code>make lin1
|
||||
</code></pre>
|
||||
|
||||
<p>OMLET will create an <a href="https://en.wikipedia.org/wiki/ADALINE">ADALINE</a> network that does</p>
|
||||
|
||||
<pre><code>y = Wxy * x + By
|
||||
</code></pre>
|
||||
|
||||
<p>This simple <a href="https://en.wikipedia.org/wiki/Linear_map">linear</a> <a href="https://en.wikipedia.org/wiki/Feedforward_neural_network">feed-forward network</a>. You can run it with</p>
|
||||
|
||||
<pre><code>./lin1 shakespeare.txt
|
||||
</code></pre>
|
||||
|
||||
<p>The linear network won’t be able to get past the gibberish stage, because it
|
||||
lacks history:</p>
|
||||
|
||||
<pre><code>./lin1 shakespeare.txt
|
||||
UERond w,
|
||||
|
||||
|
||||
Gir:
|
||||
KINof s, mesther s thouth.
|
||||
E:
|
||||
KINTret, at fu,
|
||||
GOMy t, as sth kesewit sooos atse ang k, ck,
|
||||
Sotheouserivesthecowhet been's, t he, h nre; t and, har wiread of pincer cedst sur has, ut:
|
||||
|
||||
T14:67% 2.465115
|
||||
UKESpan,
|
||||
NGaromy soreate e m esewfoure pamitherarjulthengeoly tl.
|
||||
NG s at e! w.
|
||||
|
||||
WAllinoully?
|
||||
Wamisw ofilem:
|
||||
I'delandinarrstath har aksubly s cath Whern t Is, weciss:
|
||||
|
||||
GLat s; llde.
|
||||
Y aterit dsthence
|
||||
T14:67% 2.465404
|
||||
</code></pre>
|
||||
|
||||
<p>It is able to guess at what character is likely to follow the current one
|
||||
(by doing a <a href="https://en.wikipedia.org/wiki/Linear_regression">linear regression</a>), but it lacks any history beyond that to
|
||||
guide it.</p>
|
||||
|
||||
<p>You might be wondering about the role of the <code>tanh</code> function in the RNN.
|
||||
<code>tanh</code> acts as an <a href="https://en.wikipedia.org/wiki/Activation_function">activation function</a> which adds <a href="https://en.wikipedia.org/wiki/Nonlinear_system">nonlinearity</a>
|
||||
to the network and allows it to solve complicated problems. Without
|
||||
nonlinearity, all of the linear functions would fold together into a single
|
||||
matrix-vector multiply and you’d effectively regress to the linear network
|
||||
above. Alas, even adding a nonlinearity to the feed-forward network
|
||||
(creating a <a href="https://en.wikipedia.org/wiki/Perceptron">perceptron</a>) does not improve the performance because we
|
||||
still lack the history provided by the hidden state vector (although if you
|
||||
want to try it yourself, you can do so with <code>make per1</code>).</p>
|
||||
|
||||
<h2>Going deeper</h2>
|
||||
|
||||
<p>We can try to improve the RNN’s performance by stacking RNN modules atop each
|
||||
other:</p>
|
||||
|
||||
<pre><code>h1 = RNN(h1', x)
|
||||
h2 = RNN(h2', h1)
|
||||
y = Why * h2 + By
|
||||
</code></pre>
|
||||
|
||||
<p>with <code>RNN(h, x)</code> defined as above. Each RNN module has its own set of
|
||||
parameters and its own hidden state vector. This will improve the network’s
|
||||
performance, at the cost of a much larger parameter space</p>
|
||||
|
||||
<blockquote><p><strong>IMPORTANT NOTE</strong>: Since OMLET uses the system stack for network storage,
|
||||
larger networks may cause OMLET to crash (typically with a message like
|
||||
<code>Segmentation fault</code>) unless the system stack size is first increased.
|
||||
The exact command for doing so depends on your shell and your system’s
|
||||
hard limits. On <code>sh</code>/<code>ksh</code>/<code>bash</code> shells, you can view the hard limit
|
||||
with <code>ulimit -Hs</code> and set it with <code>ulimit -s 65532</code> (replacing <code>65532</code>
|
||||
with the actual hard limit). On <code>csh</code>/<code>tcsh</code> shells, you can view
|
||||
the hard limit with <code>limit -h stacksize</code> and set it with
|
||||
<code>limit stacksize 65532</code> (replacing <code>65532</code> with the actual hard limit).</p></blockquote>
|
||||
|
||||
<p>You can try the deeper network by doing</p>
|
||||
|
||||
<pre><code>make rnn2
|
||||
</code></pre>
|
||||
|
||||
<p>(or even <code>make rnn3</code> if you want a three-layer RNN) and train it with</p>
|
||||
|
||||
<pre><code>./rnn2 shakespeare.txt
|
||||
</code></pre>
|
||||
|
||||
<p>The additional depth should allow the network to make better predictions (it
|
||||
can represent more complicated history), but it may take a long time to train
|
||||
– both because the network (being larger) now requires more time to train and
|
||||
because of the <a href="https://en.wikipedia.org/wiki/Vanishing_gradient_problem">vanishing and exploding gradient problem</a>, which might
|
||||
keep it from ever reaching its potential.</p>
|
||||
|
||||
<h2>LSTMs and GRUs</h2>
|
||||
|
||||
<p>RNNs are particularly hard to train because the they are trained using
|
||||
<a href="https://en.wikipedia.org/wiki/Backpropagation_through_time">bankpropagation through time</a>. The RNN is trained by effectively
|
||||
converting it into a non-recurrent network by making many copies of it
|
||||
and propagating the hidden state through the copies. During training,
|
||||
the backpropagation through many clones of the network amplifies the
|
||||
gradient, worsening the exploding and vanishing gradient problem.</p>
|
||||
|
||||
<p><a href="https://en.wikipedia.org/wiki/Long_short-term_memory">Long Short Term Memory</a> networks (also called LSTMs) were developed to
|
||||
solve this problem. Christopher Olah gives a good description of them
|
||||
at his <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">blog posting</a>. You can build a two-level LSTM by doing</p>
|
||||
|
||||
<pre><code>make lstm2
|
||||
</code></pre>
|
||||
|
||||
<p>and train with it with</p>
|
||||
|
||||
<pre><code>./lstm2 shakespeare.txt
|
||||
</code></pre>
|
||||
|
||||
<p>The LSTM is much easier to train because it explicitly decides how to update
|
||||
its hidden state via “gates”. These gates are called</p>
|
||||
|
||||
<ul>
|
||||
<li><em>input gate</em>, which decides what part of the input to pay attention to</li>
|
||||
<li><em>forget gate</em>, which decides what part of the hidden state to forget</li>
|
||||
<li><em>output gate</em>, which decides what part of the hidden state is used to
|
||||
produce the output</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>The basic LSTM equations are</p>
|
||||
|
||||
<pre><code>f = sigmoid(Wxf * x + Whf * h' + Bf)
|
||||
i = sigmoid(Wxi * x + Whi * h' + Bi)
|
||||
o = sigmoid(Wxo * x + Who * h' + Bo)
|
||||
c = f * c' + i * tanh(Wxc * x + Whc * h' + Bc)
|
||||
h = o * tanh(c)
|
||||
</code></pre>
|
||||
|
||||
<p>Where</p>
|
||||
|
||||
<ul>
|
||||
<li><code>x</code> is the input vector</li>
|
||||
<li><code>h</code> is the hidden state (and the output to the next layer)</li>
|
||||
<li><code>c</code> is the <em>cell state</em> which represents the “memory” of the LSTM</li>
|
||||
<li><code>h'</code> and <code>c</code>‘ are the previous values of <code>h</code> and <code>c</code> respectively</li>
|
||||
<li><code>f</code> is the <em>forget gate</em> that tells the LSTM what portion of the hidden
|
||||
state to forget</li>
|
||||
<li><code>i</code> is the <em>input gate</em> that tells the LSTM what portion of the input
|
||||
vector to pay attention to</li>
|
||||
<li><code>o</code> is the <em>output gate</em> that tells the LSTM what portion of the cell
|
||||
state to use to generate the hidden state</li>
|
||||
<li><code>Wxf</code>, <code>Whf</code>, <code>Wxi</code>, <code>Whi</code>, <code>Wxo</code>, <code>Who</code>, <code>Wxc</code>, and <code>Whc</code> are trainable
|
||||
parameter matrices</li>
|
||||
<li><code>Bf</code>, <code>Bi</code>, <code>Bo</code>, and <code>Bc</code> are trainable bias vectors</li>
|
||||
<li><code>tanh</code> is the hyperbolic tangent function</li>
|
||||
<li><code>sigmoid</code> is the <a href="https://en.wikipedia.org/wiki/Logistic_function">logistic function</a></li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>There are several LSTM variants (see <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">C. Olah’s blog post</a> for more
|
||||
examples). One important one is the <a href="https://en.wikipedia.org/wiki/Gated_recurrent_unit">gated recurrent unit</a>. GRUs are
|
||||
simplified versions of an LSTM which combine the gates together, meaning they
|
||||
require fewer learned parameters. This allows them to train faster than a
|
||||
generic LSTM. You can build a two-layer GRU with</p>
|
||||
|
||||
<pre><code>make gru2
|
||||
</code></pre>
|
||||
|
||||
<h2>Building your own networks</h2>
|
||||
|
||||
<p>The OMLET <code>Makefile</code> comes with one-, two- and three-layer RNNs, LSTMs and
|
||||
GRUs, along with simpler feed-forward networks like multi-layer perceptrons and
|
||||
a linear network. This isn’t the limit of OMLET’s power – you can create your
|
||||
own networks by modifying the <code>Makefile</code>. Networks are passed in on the
|
||||
compiler’s command-line by using <code>-D</code> directives. The network is defined by
|
||||
a <code>-DNW='...'</code> command which consists of a series of comma-separated
|
||||
assignments. For example, the simple one-layer RNN could be defined like</p>
|
||||
|
||||
<pre><code>-DNW=` x = I(n), hp = I(128), \
|
||||
h = C(hp, T(A(L(128, x), L(128, hp)))), \
|
||||
y = L(n, h)'
|
||||
</code></pre>
|
||||
|
||||
<p>The network declares <code>x</code> as an input vector (there must be a declaration for
|
||||
<code>x</code>). It is declared as <code>I(n)</code>, which is an input vector of size <code>n</code>, which
|
||||
is the number of characters of the input alphabet (OMLET computes this from
|
||||
the input file at the start of training). OMLET will arrange to present the
|
||||
input character as a <a href="https://en.wikipedia.org/wiki/One-hot">one-hot</a> vector based on the current input
|
||||
character.</p>
|
||||
|
||||
<p>The second declaration, <code>hp</code>, declares the previous hidden state vector (what
|
||||
we called <code>h'</code> above). We declare this to be of size 128 – an arbitrary
|
||||
choice. A larger state vector can (theoretically) carry more state, but at
|
||||
a cost of larger parameter matrices and longer training time. You can
|
||||
experiment with increasing the hidden vector size and see.</p>
|
||||
|
||||
<p>The third line is the core of the RNN. It sets <code>h</code>, the hidden vector output
|
||||
to be the sum of two linear elements specified by <code>L</code>. The <code>L</code> function
|
||||
takes two parameters – the output vector size (which must match the size of
|
||||
<code>h</code>) and the input vector. <code>L</code> will compute <code>y = W * x + B</code> where each <code>L</code>
|
||||
has its own <code>W</code> (weight) and <code>B</code> (bias) training parameters. Both <code>x</code> and
|
||||
<code>hp</code> are sent through <code>L</code> and the result passed through the <code>A</code> function,
|
||||
which does vector addition. That result is passed through <code>T</code> which does
|
||||
element-wise <code>tanh</code> activation.</p>
|
||||
|
||||
<p>Next, we wrap the whole thing with the <code>C</code> function. <code>C</code> connects <code>hp</code>
|
||||
with <code>h</code>, causing the new value of <code>h</code> to be passed to the <code>hp</code> vector on
|
||||
the next iteration of the algorithm (allowing the RNN to retain state in <code>h</code>).</p>
|
||||
|
||||
<p>Finally, the whole result is passed through another instance of <code>L</code>, this time
|
||||
producing a vector of size <code>n</code>, which will have the negative log
|
||||
<a href="https://en.wikipedia.org/wiki/Likelihood_function">likelihood</a> function. This is assigned to <code>y</code>, which is the output of
|
||||
the network (and hence must also be declared).</p>
|
||||
|
||||
<p>OMLET will take the <code>y</code> result and pass it through the <a href="https://en.wikipedia.org/wiki/Softmax_function">softmax</a> function,
|
||||
which converts the log probabilities into a probability distribution. In
|
||||
inference mode, this is used to select the next character to emit. In
|
||||
training mode, this is used to generate the loss which is backpropagated.</p>
|
||||
|
||||
<p>As an example of a more complicated network, we can look at a two-layer
|
||||
GRU network:</p>
|
||||
|
||||
<pre><code>-DHS=128, \
|
||||
-DNW=' x = I(n), \
|
||||
y = L(n, MD(MD(x)))' \
|
||||
-DBK=' hp = I(HS), \
|
||||
z = S(A(L(HS, x), L(HS, hp))), \
|
||||
r = S(A(L(HS, x), L(HS, hp))), \
|
||||
c = T(A(L(HS, x), L(HS, hp))), \
|
||||
zc = OG(1, -1, z), \
|
||||
h = C(hp, A(M(zc, hp), M(z, c))), \
|
||||
y = h'
|
||||
</code></pre>
|
||||
|
||||
<p>We are using a few new tricks here – first, we are defining <code>HS</code> as the size
|
||||
of the hidden and cell vectors. There’s nothing special about this name, its
|
||||
just convenient to specify it so we don’t have a bunch of constants in the
|
||||
code. Second, the network itself is very simple – it declares <code>x</code> and has
|
||||
the matrix that converts the <code>HS</code>-sized hidden vector back to the <code>n</code>-sized
|
||||
alphabet vector… but it now calls <code>MD</code>, which is the user-definable
|
||||
module (here we are using it twice, to have two cascaded GRU blocks). The
|
||||
<code>MD</code> function performs the sub-network defined by the <code>BK</code> compile-time
|
||||
parameter (specified in the <code>-DBK='...'</code> setting). This sub-module again
|
||||
takes an <code>x</code> parameter and produces a <code>y</code> output. Inside it, we declare
|
||||
<code>hp</code> and <code>h</code>, the previous and current state vector, plus equations for the
|
||||
various GRU gates (these use <code>S</code> for the sigmoid activation function).
|
||||
One final new call is <code>OG</code> which does offset and gain, performing
|
||||
<code>y = offset + gain * x</code> where <code>offset</code> (the first parameter) and <code>gain</code> (the
|
||||
second) are constants. We are using this here to compute <code>(1 - z)</code> for the
|
||||
GRU’s linear interpolator.</p>
|
||||
|
||||
<p>The full set of available function blocks follows:</p>
|
||||
|
||||
<ul>
|
||||
<li><code>I(s)</code>: declares a vector (input or state) of size <code>s</code></li>
|
||||
<li><code>L(s, x)</code>: learnable linear function <code>y = W * x + B</code> with an output vector
|
||||
size of <code>s</code></li>
|
||||
<li><code>CM(x)</code>: learnable element-wise gain function <code>y = W * x</code></li>
|
||||
<li><code>A(a, b)</code>: element-wise add: <code>y = a + b</code></li>
|
||||
<li><code>M(a, b)</code>: element-wise multiply: <code>y = a * b</code></li>
|
||||
<li><code>S(x)</code>: sigmoid activation function <code>y = sigmoid(x)</code></li>
|
||||
<li><code>T(x)</code>: hyperbolic tangent activation function <code>y = tanh(x)</code></li>
|
||||
<li> <code>C(xp, x)</code>: copy <code>x</code> to <code>xp</code> in the next time step (propagate through
|
||||
time)</li>
|
||||
<li> <code>OG(o, g, x)</code>: apply a constant offset and gain: <code>y = o + g * x</code></li>
|
||||
<li> <code>MD(x)</code>: apply the sub-network specified by <code>BK</code></li>
|
||||
</ul>
|
||||
|
||||
|
||||
<p>Note: even if you don’t use <code>MD</code> in your network, you should still define
|
||||
<code>BK</code> by adding <code>-DBK='y=x'</code> to the command line, otherwise you will get a
|
||||
compile-time error.</p>
|
||||
|
||||
<h2>Hyperparameters</h2>
|
||||
|
||||
<p>OMLET has a large number of training and inference parameters which can be
|
||||
changed by the user. All of these are set by <code>-D</code> on the compile command line.
|
||||
The list of hyperparameters follows:</p>
|
||||
|
||||
<ul>
|
||||
<li><code>TP</code>: Temperature parameter for use in inference mode. This divides the
|
||||
log probabilities before softmax. A low temperature makes the model choose
|
||||
safer but more boring choices. A high temperature takes more risks but
|
||||
makes more mistakes. Default is 1.0, which uses the computed
|
||||
probabilities.</li>
|
||||
<li>‘N’: Batch size. This is the number of times the RNN is unrolled, so it
|
||||
controls how far back in the past the RNN can see. The default is 50.
|
||||
Larger batch sizes update the weights less frequently and allow the RNN to
|
||||
see farther back in time, but at a cost of proportionally more memory.</li>
|
||||
<li><code>TR</code>: The percentage of batches in the input data set that will be used
|
||||
for training. The default value of 0.95 sets this as 95%.</li>
|
||||
<li><code>LR</code>: The initial learning rate, The default is 0.002.</li>
|
||||
<li><code>LE</code>: The epoch where the learning rate will start decaying. Defaults to
|
||||
epoch 10.</li>
|
||||
<li><code>LD</code>: Learning rate decay, per epoch (after <code>LE</code> epochs). The learning
|
||||
rate is scaled by this number. Default is 0.97</li>
|
||||
<li><code>WD</code>: Weight-decay parameter, to promote <a href="https://en.wikipedia.org/wiki/Regularization_(mathematics)">regularizaiton</a>. The default
|
||||
is 0.00008.</li>
|
||||
<li><code>RS</code>: The random scale for weight initialization. Weight parameters will
|
||||
be initialized to be between <code>-RS</code> to <code>+RS</code>. The default is 0.15.</li>
|
||||
<li><code>CL</code>: Clamp value for gradients. Gradients will be limited to the range
|
||||
<code>-CL</code> to <code>+CL</code>. Default is 5.</li>
|
||||
<li><code>B1</code>: Momentum mean parameter for Adam optimizer. Set to 0.9.</li>
|
||||
<li><code>B2</code>: Momentum variance parameter for Adam optimizer, Set to 0.999.</li>
|
||||
<li><code>EP</code>: Epsilon parameter for Adam, to provide numerical stability. Set to
|
||||
0.00000001.</li>
|
||||
<li><code>DI</code>: How often to print a training or validation progress message and
|
||||
inference snippet. The default prints every 100 training batches.</li>
|
||||
<li><code>SL</code>: The number of characters to print when doing an inference snippet.
|
||||
Default is 200.</li>
|
||||
<li><code>PF</code>: Format string for the checkpoint filename. The default is
|
||||
<code>"cp%02d_%.3f"</code> which includes the epoch and validation loss. You may
|
||||
wish to add a subdirectory to the name to keep checkpoint files out of
|
||||
the current directory.</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<hr />
|
||||
|
||||
<TABLE><TR>
|
||||
<TD><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/"><img alt="Creative Commons License" style="border-width:0" src="/png/by-sa-3.0-88x31.png" /></a></TD>
|
||||
<TD><P>© Copyright 1984-2019,
|
||||
<A HREF="/judges.html">Leo Broukhis, Simon Cooper, Landon Curt Noll</A>
|
||||
- All rights reserved<BR>
|
||||
This work is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.</P></TD>
|
||||
<TD> <!--<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a>--></TD>
|
||||
</TR></TABLE>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,185 +0,0 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta name="GENERATOR" content="mkd2html 2.2.0">
|
||||
<meta http-equiv="Content-Type"
|
||||
content="text/html; charset=utf-8"> <link rel="stylesheet"
|
||||
type="text/css"
|
||||
href="../hint.css">
|
||||
<title>Most calendrical</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Most calendrical</h1>
|
||||
|
||||
<p>Timo Poikola<br>
|
||||
<a href="https://ig.fi/other/ioccc">https://ig.fi/other/ioccc</a><br>
|
||||
Twitter: @timpoiko</p>
|
||||
|
||||
<h2>Judges' comments:</h2>
|
||||
|
||||
<h3>To use:</h3>
|
||||
|
||||
<pre><code>make prog
|
||||
./prog 512 ./prog
|
||||
|
||||
sleep 86400; make clobber prog
|
||||
./prog 512 ./prog
|
||||
|
||||
sleep 86400; make clobber prog
|
||||
./prog 512 ./prog
|
||||
</code></pre>
|
||||
|
||||
<h3>Try:</h3>
|
||||
|
||||
<pre><code>If you don't have a few days, try:
|
||||
|
||||
TZ=UTC24 make clobber prog
|
||||
./prog 512 ./prog
|
||||
|
||||
TZ=UTC48 make clobber prog
|
||||
./prog 512 ./prog
|
||||
</code></pre>
|
||||
|
||||
<h3>Selected Judges Remarks:</h3>
|
||||
|
||||
<p>Do you have the time to see what this program does? Think again, come back and try again.</p>
|
||||
|
||||
<p>Note that 2 can be said to be the least odd prime because it is the greatest even prime.
|
||||
Moreover 2 can be also said to be the greatest odd prime because it is the least even prime.</p>
|
||||
|
||||
<p>The source code layout and some of the variable names honors the ski jumping accomplishments of
|
||||
<a href="https://en.wikipedia.org/wiki/Matti_Nyk%C3%A4nen">Matti Nykänen</a>.</p>
|
||||
|
||||
<h2>Author’s comments:</h2>
|
||||
|
||||
<h3>How to build</h3>
|
||||
|
||||
<pre><code>gcc -O0 -o prog prog.c
|
||||
</code></pre>
|
||||
|
||||
<p>or</p>
|
||||
|
||||
<pre><code>clang -O[0123s] -o prog prog.c
|
||||
</code></pre>
|
||||
|
||||
<h3>Poster</h3>
|
||||
|
||||
<p>You can generate an A3 sized poster by <em>make docs</em>. This command creates a pdf file <em>poikola.pdf</em>.</p>
|
||||
|
||||
<h3>What this entry does</h3>
|
||||
|
||||
<p>An old wise man mumbled to me:</p>
|
||||
|
||||
<blockquote><p>Implement a program, which can calculate SHA-3 checksum for a file</p></blockquote>
|
||||
|
||||
<p>I think it is easy, but the wizard didn’t stop yet:</p>
|
||||
|
||||
<blockquote><p>and primes and Fibonacci numbers.</p></blockquote>
|
||||
|
||||
<p>Still easy. But the Gandalf the Grey impersonator is not ready:</p>
|
||||
|
||||
<blockquote><p>But use only <em>main()</em>, and do not use <em>math.h</em> nor predefined stuff at all. But wait, there are some more rules: This
|
||||
work should reflect the extraordinary mind of Matti Nykänen,
|
||||
the all-time best <a href="https://en.wikipedia.org/wiki/Matti_Nyk%C3%A4nen">ski jumper</a> and
|
||||
<a href="http://telefinn.blogspot.com/2011/11/matti-nykanen-quotes.html">amateur philosopher</a>, who passed away on fourth of
|
||||
February 2019.</p></blockquote>
|
||||
|
||||
<h4>Oh boys.</h4>
|
||||
|
||||
<p>So, I followed the rules given by Gandalf the White and the Judges of IOCCC. Output of <code>iocccsize</code>,
|
||||
using the <a href="/2019/iocccsize.c">2019 version of iocccsize</a>, is carefully selected.
|
||||
For the sake of clarity, I used single letter variables in the code. I also avoided unnecessary use of functions.
|
||||
Like a tripundra, this program has three levels. In order to reveal all of them, you have to compile this program
|
||||
on three consecutive days.</p>
|
||||
|
||||
<p>As this is a contest for obfuscated code, the program does not perform unnecessary checks, but either fails or gets
|
||||
stuck if invoked erroneously. The program is invoked in this way: ./prog <code><integer></code> <code><file></code> in where <code><integer></code> is
|
||||
224, 256, 384, 512 or even 1024. Yes, I know 1024 is not a supported length of output, but as I said, the program does
|
||||
not validate its input in any way. These same parameters should be given in calculating a Fibonacci sequence or prime
|
||||
numbers. <em>Please note that maximum file size is one gigabyte.</em></p>
|
||||
|
||||
<h3>Obfuscation</h3>
|
||||
|
||||
<p><em>Matti Nykänen</em>: Every time I jump, and I get that feeling of <em>bon voyage</em>, the feeling that I’ve been here
|
||||
before.</p>
|
||||
|
||||
<p><em>Matti Nykänen</em>: When you are about to jump, you are all alone and have to make your own decisions. Up there,
|
||||
it’s all <em>up yours</em>.</p>
|
||||
|
||||
<p>This code has some jumping too. I used some <code>goto</code>s instead of <code>longjmp()</code>. Some Finnish ski jumping sites are
|
||||
used as labels, however, there is <code>lahti</code> instead of <tt style="font-family: Monaco, Courier New, monospace;font-size:
|
||||
12px;">salpausselkä</tt>.</p>
|
||||
|
||||
<h3>SHA-3-512 Compatibility chart</h3>
|
||||
|
||||
<p>Tested against <a href="https://www.di-mgt.com.au/sha_testvectors.html">test vectors</a>.
|
||||
Clang-4.0: For optimization levels [0123s], the program compiles and produces correct output for every tested vector using
|
||||
c11, c89, c90, c99, gnu11, gnu1x, gnu89, gnu90, gnu99, iso9899:1990, iso9899:199409, iso9899:1999 and
|
||||
iso9899:2011 as argument of <code>-std=</code></p>
|
||||
|
||||
<p>Gcc-6: For optimization level 0, correct output using
|
||||
c11, c99, gnu11, gnu1x, gnu89, gnu90, gnu99, iso9899:1999 and iso9899:2011</p>
|
||||
|
||||
<h3>Missing a prime</h3>
|
||||
|
||||
<p>For obvious reason, the oddest prime is missing from output.</p>
|
||||
|
||||
<h3>Observations</h3>
|
||||
|
||||
<p>Gcc and clang are very fragile compilers. I spent numerous hours trying to achieve exactly the same output from
|
||||
different optimization levels and compilers. Small change in code can produce really unexpected results, i.e. the compiler
|
||||
can skip a few expressions or statements without obvious reason.</p>
|
||||
|
||||
<h3>Major spoilers</h3>
|
||||
|
||||
<p>Please scroll down if you want to see spoilers.</p>
|
||||
|
||||
<div style="margin-bottom:61em;"> </div>
|
||||
|
||||
|
||||
<p>It is commonly believed that <a href="http://www.cadaeic.net/calendar.htm">Mike Keith’s algorithm</a> published in <em>Journal of
|
||||
Recreational Mathematics</em>, Vol. 22, No. 4, 1990, p. 280, is the shortest way to calculate weekday for given date.
|
||||
Keith’s algorithm:<br>
|
||||
<code>(d+=m<3?y--:y-2,23*m/9+d+4+y/4-y/100+y/400)%7</code><br>
|
||||
My algorithm: I noticed that<br>
|
||||
<code>(23*m/9+(m<3?y--:y-2)+d+4+y/4-y/100+y/400)%7</code><br>
|
||||
is one character shorter. In my solution, <code>m<3?y--:y-2</code> is used directly, instead of assign value for <code>d</code>.</p>
|
||||
|
||||
<p>Q is pronounced in Finnish exactly like “kuu”, a word for month or the Moon. So, <code>Q = k</code> is intended.</p>
|
||||
|
||||
<p>Last term of outputted Fibonacci sequence is the last one smaller than 2<sup>64</sup>. It is plain coincidence (but a funny)
|
||||
that value of ‘^’ is 94 in ASCII table.</p>
|
||||
|
||||
<p>For prime calculations: I recycled code from my high school C-programming course. Originally, I wrote it over 20 years
|
||||
ago. My teacher said that it is (too) complicated to use bits stored in arrays, so I did it.</p>
|
||||
|
||||
<h3>Other stuff</h3>
|
||||
|
||||
<p>Cody: I really like that you enjoyed my previous entry. I hope that you will like this entry too.
|
||||
Because I did not want to give too many clues to Judges, I tried to write this entry in a different way
|
||||
than Most Stellar, but in this code there is at least one recycled thing from it. By the way,
|
||||
answers to questions: 1. 255. 2. Try to compare binary representations of those floats and binary representation
|
||||
of the string “25th IOCCC!”. 3. I don’t know. I used standard trigonometric functions from <code>math.h</code> for rotating the Big Dipper
|
||||
and later replaced those functions with my own implementations. But there was a bug in my code and the effect was more
|
||||
beautiful than intended. I did not even debug this bug and now it works as a feature. 4. It is not possible. If you
|
||||
change a single bit, the Fletcher 16 checksum does not match anymore and <code>goto</code> jumps to end of the code.</p>
|
||||
|
||||
<p>I see that <em>Volker Diels-Grabsch</em>’s winner is awarded in category <strong>Most self-aware</strong>. In last year, my entry incorporates
|
||||
the Fletcher 16 algorithm to ensure integrity of code. One of my first ideas for this year was to do the same thing,
|
||||
but with SHA-3. Using Fletcher 16, calculating a checksum which matches with <code>sleeping time factor</code> took about one minute.
|
||||
I realized that SHA-3 even with 224-bit output could take too much CPU time.</p>
|
||||
|
||||
<p>I would like to say <em>thanks</em> to PSP, because there were not too much interrupts during the coding phase.</p>
|
||||
|
||||
<hr />
|
||||
|
||||
<TABLE><TR>
|
||||
<TD><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/"><img alt="Creative Commons License" style="border-width:0" src="/png/by-sa-3.0-88x31.png" /></a></TD>
|
||||
<TD><P>© Copyright 1984-2019,
|
||||
<A HREF="/judges.html">Leo Broukhis, Simon Cooper, Landon Curt Noll</A>
|
||||
- All rights reserved<BR>
|
||||
This work is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.</P></TD>
|
||||
<TD> <!--<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a>--></TD>
|
||||
</TR></TABLE>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,229 +0,0 @@
|
||||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
||||
<html>
|
||||
<head>
|
||||
<meta name="GENERATOR" content="mkd2html 2.2.0">
|
||||
<meta http-equiv="Content-Type"
|
||||
content="text/html; charset=utf-8"> <link rel="stylesheet"
|
||||
type="text/css"
|
||||
href="../hint.css">
|
||||
<title>Most in need of transparency</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Most in need of transparency</h1>
|
||||
|
||||
<p>Don Yang<br>
|
||||
<a href="http://uguu.org/">http://uguu.org/</a></p>
|
||||
|
||||
<h2>Judges' comments:</h2>
|
||||
|
||||
<h3>To use:</h3>
|
||||
|
||||
<pre><code>make
|
||||
|
||||
./prog sample_input.txt a b c d
|
||||
|
||||
cat a
|
||||
|
||||
cat b
|
||||
|
||||
cat c
|
||||
|
||||
cat d
|
||||
|
||||
cat a b c d | ./prog
|
||||
</code></pre>
|
||||
|
||||
<h3>Try:</h3>
|
||||
|
||||
<pre><code>./prog prog.c a b c d e
|
||||
|
||||
cat a
|
||||
|
||||
cat b
|
||||
|
||||
cat c
|
||||
|
||||
cat d
|
||||
|
||||
cat e
|
||||
|
||||
cat a b c d e | ./prog
|
||||
|
||||
cat a b c d e | ./prog | diff - prog.c
|
||||
</code></pre>
|
||||
|
||||
<h3>Selected Judges Remarks:</h3>
|
||||
|
||||
<p>Do you understand how this program works?
|
||||
You may have to go over and over the output again.</p>
|
||||
|
||||
<h2>Author’s comments:</h2>
|
||||
|
||||
<h1>Summary</h1>
|
||||
|
||||
<p>Scramble/unscramble to single file:</p>
|
||||
|
||||
<pre><code>./prog original.txt > scrambled.txt
|
||||
|
||||
./prog < scrambled.txt > unscrambled.txt
|
||||
</code></pre>
|
||||
|
||||
<p>Scramble/unscramble to multiple files:</p>
|
||||
|
||||
<pre><code>./prog input.txt output1.txt output2.txt ...
|
||||
|
||||
cat output1.txt output2.txt ... | ./prog > unscrambled.txt
|
||||
</code></pre>
|
||||
|
||||
<p>Re-scramble:</p>
|
||||
|
||||
<pre><code>./prog original.txt | ./prog - > scrambled1.txt
|
||||
./prog original.txt | ./prog - | ./prog - > scrambled2.txt
|
||||
./prog original.txt | ./prog - | ./prog - | ... > scrambled3.txt
|
||||
</code></pre>
|
||||
|
||||
<p>Scramble slowly:</p>
|
||||
|
||||
<pre><code>./prog input.txt | ruby prog.c
|
||||
</code></pre>
|
||||
|
||||
<p>Input must be in UTF-8 encoding.</p>
|
||||
|
||||
<h1>Details</h1>
|
||||
|
||||
<p>This is a text scrambling utility. To scramble text, specify input
|
||||
file name as the first argument:</p>
|
||||
|
||||
<pre><code>./prog input.txt > output.txt
|
||||
</code></pre>
|
||||
|
||||
<p>If original input file was small enough to fit in your terminal, you
|
||||
can simply use <code>cat</code> to unscramble the output text.</p>
|
||||
|
||||
<pre><code>cat output.txt
|
||||
</code></pre>
|
||||
|
||||
<p>It might seem like nothing was scrambled, but you can try passing the
|
||||
output through <code>strings(1)</code> and see that most of the original strings
|
||||
are gone. What happened is that the original text has been shuffled
|
||||
with ANSI cursor positioning codes inserted such that characters are
|
||||
printed in random order. This effect is more visible if there is a
|
||||
delay after each character is printed. For convenience, an utility is
|
||||
included in the source to output characters slowly:</p>
|
||||
|
||||
<pre><code>./prog input.txt | ruby prog.c
|
||||
</code></pre>
|
||||
|
||||
<p>Scrambling order is deterministic, but you can reshuffle the output by
|
||||
piping it through the same utility again, by specifying <code>-</code> as the
|
||||
first argument:</p>
|
||||
|
||||
<pre><code>./prog input.txt | ./prog - | ./prog - | ...
|
||||
</code></pre>
|
||||
|
||||
<p>You can also split the input to multiple files, each getting a subset
|
||||
of the original input characters:</p>
|
||||
|
||||
<pre><code>./prog input.txt output1.txt output2.txt ...
|
||||
cat output1.txt output2.txt ... > unscrambled.txt
|
||||
</code></pre>
|
||||
|
||||
<p>You can only unscramble files with <code>cat</code> if the input fits in a single
|
||||
terminal screen. If you don’t know what will be the terminal size of
|
||||
the recipient, note that <code>prog.c</code> will fit in a 80x25 terminal, so a
|
||||
good guideline is to pick files that have fewer rows and columns than
|
||||
<code>prog.c</code>.</p>
|
||||
|
||||
<p>For files that are larger than your terminal window, <code>cat</code> will not be
|
||||
sufficient for unscrambling because any scrolling causes the character
|
||||
positions to be misaligned. But fear not, this utility also includes
|
||||
unscrambling functionality. If no argument is specified, input will
|
||||
be read through stdin and unscrambled to stdout:</p>
|
||||
|
||||
<pre><code>./prog < scrambled.txt > unscrambled.txt
|
||||
</code></pre>
|
||||
|
||||
<p>Where this is useful is that if you scramble the input to multiple
|
||||
files and unscramble each output files individually, those output
|
||||
files can be printed on transparencies and distributed separately.</p>
|
||||
|
||||
<pre><code>./prog input.txt output1.txt output2.txt output3.txt
|
||||
./prog < output1.txt > layer1.txt
|
||||
./prog < output2.txt > layer2.txt
|
||||
./prog < output3.txt > layer3.txt
|
||||
</code></pre>
|
||||
|
||||
<p>In other words, this is a utility for facilitating
|
||||
<a href="https://en.wikipedia.org/wiki/Visual_cryptography">visual cryptography</a>.</p>
|
||||
|
||||
<p>This utility can also be used just for removing trailing whitespaces
|
||||
and expanding tabs to 8 spaces.</p>
|
||||
|
||||
<pre><code>./prog input.txt | ./prog
|
||||
</code></pre>
|
||||
|
||||
<h1>Features</h1>
|
||||
|
||||
<h2>UTF-8 support</h2>
|
||||
|
||||
<p>A simple text scrambling utility might not have required a screenful
|
||||
of code, but this particular one natively supports UTF-8.
|
||||
Specifically: individual UTF-8 byte sequences will not be split into
|
||||
broken ones, and the utility is aware of full-width versus half-width
|
||||
differences such that cursor is positioned correctly for most CJK
|
||||
input. <code>sample_input.txt</code> demonstrates this feature with a mix of
|
||||
Basic Latin and Japanese characters.</p>
|
||||
|
||||
<p>Output cursor positions will most definitely be wrong for inputs
|
||||
containing combining characters and right-to-left text, but it should
|
||||
be possible to unscramble those text back to the original using this
|
||||
same utility, since the utility is self-consistent in how cursor
|
||||
positions are handled.</p>
|
||||
|
||||
<h2>Error handling</h2>
|
||||
|
||||
<p>If any file can not be opened for reading or writing, an error is
|
||||
printed to stderr, and the utility will exit with nonzero status.</p>
|
||||
|
||||
<p>If there isn’t enough memory, the utility will likely crash.</p>
|
||||
|
||||
<p>If input contains CR-LF end of line sequences, those CRs are silently
|
||||
dropped. In fact, most control codes are silently ignored except line
|
||||
feeds (preserved) and tabs (expanded to 8 spaces).</p>
|
||||
|
||||
<h2>Compatibility</h2>
|
||||
|
||||
<p>Verified to work and compiles without warnings on these environments:</p>
|
||||
|
||||
<ul>
|
||||
<li> gcc 7.3.0 on Linux</li>
|
||||
<li> gcc 7.4.0 on Cygwin</li>
|
||||
<li> clang 4.0.1-10 on Linux</li>
|
||||
<li> clang 5.0.1 on Cygwin</li>
|
||||
<li> gcc 4.8.4 on Linux</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<h2>Full version</h2>
|
||||
|
||||
<p>I tried really hard to make <code>prog.c</code> fit in a 80x25 terminal without
|
||||
wrapping. If I were not constrained by space, I would have made an
|
||||
ASCII art out of it. <code>violet.c</code> is that program, and also comes with
|
||||
the minor benefit of supporting lines longer than a million columns
|
||||
depending on <code>sizeof(int)</code>.</p>
|
||||
|
||||
<p>Violet is named after “Violet Evergarden”, the character who is highly
|
||||
skilled in writing letters.</p>
|
||||
|
||||
<hr />
|
||||
|
||||
<TABLE><TR>
|
||||
<TD><a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/"><img alt="Creative Commons License" style="border-width:0" src="/png/by-sa-3.0-88x31.png" /></a></TD>
|
||||
<TD><P>© Copyright 1984-2019,
|
||||
<A HREF="/judges.html">Leo Broukhis, Simon Cooper, Landon Curt Noll</A>
|
||||
- All rights reserved<BR>
|
||||
This work is licensed under a <a rel="license" href="https://creativecommons.org/licenses/by-sa/3.0/">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.</P></TD>
|
||||
<TD> <!--<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a>--></TD>
|
||||
</TR></TABLE>
|
||||
</body>
|
||||
</html>
|
||||
Reference in new issue
Block a user