Move 1988-2004 entries into subdirs
We we move 1988-2004 entries from all of a year's winning entries in a single directory to each winner being in their own sub-directory under the given year directory.
This commit is contained in:
Landon Curt Noll committed
1 parent
1e82e8aa3e
commit
d2a42f42e8
478 files changed
No files matched your search
@@ -0,0 +1,116 @@
|
||||
.\"
|
||||
.\" To format this file into a text file say
|
||||
.\"
|
||||
.\" nroff -man samefile.1 or
|
||||
.\" groff -man -Tascii samefile.1
|
||||
.\"
|
||||
.TH SAMEFILE 1 "OCTOBER 1997" IOCCC "User Manuals"
|
||||
.SH NAME
|
||||
samefile \- find identical files
|
||||
.SH SYNOPSIS
|
||||
.B samefile
|
||||
.SH DESCRIPTION
|
||||
.B samefile
|
||||
reads a list of file names (one file name per line) from stdin.
|
||||
For each file name pair with identical contents, a
|
||||
line consisting of six tab separated fields is output:
|
||||
The size in bytes, two file names, the character ``='' if
|
||||
the two files are on the same device, ``X'' otherwise,
|
||||
and the link counts of the two files.
|
||||
The output is sorted in reverse order by size.
|
||||
|
||||
.B samefile
|
||||
uses two stages to give optimum performance.
|
||||
|
||||
In the first stage, all non-plain files are silently
|
||||
ignored (directories, devices, FIFOs, sockets,
|
||||
symbolic links) as well as files for which
|
||||
.BR stat (2)
|
||||
or
|
||||
.BR open (2)
|
||||
fails.
|
||||
The result of the first stage (the file names) is written into
|
||||
a binary tree with one node for every file size. Each node
|
||||
is in turn a linked list of file names along with inode and
|
||||
device information.
|
||||
It is also at this first stage where checks for hard links are done.
|
||||
For any inode only one filename will be added to the binary tree.
|
||||
|
||||
In the second stage all files having the same
|
||||
size are compared against each other. The rules
|
||||
of mathematical logic are applied to reduce work
|
||||
and output noise: if files A, B, and C have the same size and
|
||||
.B samefile
|
||||
finds that A = B and A = C then it will not
|
||||
compare B against C (and will not
|
||||
output a line for B and C) but only
|
||||
for A = B and A = C. The algorithm will detect
|
||||
equality across arbitrarily long chains.
|
||||
Note however, that because
|
||||
only the first filename per inode gets into the
|
||||
first stage, the output for a group of identical files
|
||||
with different inode numbers is also minimized. Suppose
|
||||
you have six identical files of size 100
|
||||
in an inode group consisting of the three
|
||||
inodes with numbers 10, 20 and 30:
|
||||
|
||||
.nf
|
||||
$ ls -i # output edited for readabilty:
|
||||
10 file1 20 file4 30 file6
|
||||
10 file2 20 file5
|
||||
10 file3
|
||||
$ ls | samefile
|
||||
100 file1 file4 = 3 2
|
||||
100 file1 file6 = 3 1
|
||||
|
||||
.fi
|
||||
The sum of the sizes in the first column is the amount of disk space
|
||||
you could gain by making all 6 files links to only one
|
||||
file or remove all but one of the files. To be precise,
|
||||
disk space is allocated in blocks - you will probably gain
|
||||
two blocks here, rather than 200 bytes.
|
||||
Note that it is not enough to just remove file4 and
|
||||
file6 (you would gain only 100 bytes because file5
|
||||
still exists.)
|
||||
|
||||
.SH LIMITATIONS
|
||||
.B Samefile
|
||||
was written with no limits in mind. The number of input lines is
|
||||
unlimited. The size of the actual files is only limited by
|
||||
available virtual memory needed to compare one pair of files.
|
||||
The only hard limit is the constraint that there should not
|
||||
be more than about 8192 files having the same size. Experience
|
||||
has shown that there are rarely more than a couple dozen
|
||||
files of the same size.
|
||||
.SH EXAMPLES
|
||||
.nf
|
||||
|
||||
For everybody:
|
||||
What are the duplicates under my home directory?
|
||||
|
||||
$ find $HOME | samefile
|
||||
|
||||
For the sysadmin folks:
|
||||
Report all duplicate files under /usr larger than 16k:
|
||||
|
||||
$ find /usr -size +16384c -a -type f | samefile
|
||||
|
||||
For the ftp and WWW admins:
|
||||
How much space is wasted below our site's /pub directory?
|
||||
|
||||
$ find /pub -type f | samefile | awk '{sum += $1} END {print sum}'
|
||||
|
||||
.fi
|
||||
|
||||
.SH "EXIT STATUS"
|
||||
If
|
||||
.B samefile
|
||||
runs out of memory the exit status is 1,
|
||||
zero otherwise.
|
||||
.fi
|
||||
.SH "SEE ALSO"
|
||||
.BR find (1),
|
||||
.BR ln (1),
|
||||
.BR rm (1)
|
||||
.\"
|
||||
.\" End Of Man Page
|
||||
@@ -0,0 +1,61 @@
|
||||
%:define _POSIX_SOURCE
|
||||
#include<fcntl.h>
|
||||
#include<stdio.h>
|
||||
#include<unistd.h>
|
||||
#include<stdlib.h>
|
||||
#include<string.h>
|
||||
#include<sys/types.h>
|
||||
#include<sys/stat.h>
|
||||
#define D(N,t)Z t*N V<%t*z U M0*z); H z)u z; X}
|
||||
#define k(x,y)x<0||fstat(x,&y)||
|
||||
#define h(x)=open(x,O_RDONLY)
|
||||
#define b(x),(int)x.st_nlink
|
||||
#define B ;typedef g
|
||||
#define X exit (1);
|
||||
#define O .st_size
|
||||
#define U =malloc(
|
||||
#define Y S.st_ino
|
||||
#define v ;%>else
|
||||
#define W .st_dev
|
||||
#define o ||read(
|
||||
#define Z static
|
||||
#define g struct
|
||||
#define u return
|
||||
#define I char*
|
||||
#define V (M2)
|
||||
#define H if(
|
||||
#define _ ->
|
||||
|
||||
/* HE WHO SAYS */
|
||||
|
||||
Z I A<:32767/ M0(I )]; Z g stat S,T; Z size_t y B f{
|
||||
I n ; g f * x ; dev_t d ; ino_t i; } f B t{ M1 s,c; f
|
||||
*l; g t*L,*R; } t; D(a,t)D(E ,f)Z t*J(t*p,I n){ H! p){ p=
|
||||
a(); p _ s =S O; p _ c=1; p _ L=p _ R=0;
|
||||
p _ l= E(); p _ l _ n=n; p _ l _ x=0; p
|
||||
_ l _ d=S W; p _ l _ i= Y v H S O==p _
|
||||
s){ f* e; for (e=p _ l; e; e=e _ x) { H S W==e _
|
||||
d&&Y== e _ i) { u p; } } e=E(); e _ x=p _ l; e _
|
||||
n=n; e _ d =S W; e _ i=Y; p _ l=e; ++p _
|
||||
c v H S O< p _ s) { p _ L= J( p _ L,n)v{
|
||||
p _ R= J (p _ R,n ); } u p ; } Z int Q(
|
||||
I G,I F){ int d h(G),D h(F); I m,*M; H k(d,S )k(D,T )(y =S
|
||||
O)-T O){ y= 0; goto d; } H! (m U y))||!(M U y))o d ,m,y)-
|
||||
y o D,M,y)-y) X y=!memcmp( m,M,y); M5(m) ; M5(M ); d:V
|
||||
|
||||
close (d );V close(D); u y; } Z M2 C(M1 z,M1 N){ M1 i=N*(N-1)/2,
|
||||
j=1,s; I q,*e,* p,*w,*l; e=q= M4((size_t)i,1); H!e) X p=q+i; for(i=0; e-p
|
||||
; ++e){ H!*e&&Q (A[i:>,A[j])){ V printf("%""l""d" "\t""%" "s""\t" "%""s"
|
||||
"??/t" "%""c" "\11" "%""d" "??/t" "%""d" "\n", z,A[i]
|
||||
,A[j], S W -T W?'X' :'='b( S)b(T) ); H j -i-1) { s=N-
|
||||
i-3; w =e+s+1; l=q+N* (j-1)- j*(j-1 )/ 2 ; do{ *w
|
||||
=1; H w==l) break; w+= s; } while( s-->0); } } H++j ==N){ j=i+++
|
||||
2; } } M5(q); } Z M2 P(t*p ){ H p){ P(p _ R); H p _ c>1){ M1 i=0
|
||||
; f*l= p _ l; for (; i< p _ c; ++i ){ A[i
|
||||
]= l _ n; l=l _ x; } C (p _ s, p _ c); } P (p _
|
||||
L) ; } } int main V { t*r= 0; I F ; for (; ; )
|
||||
{ H!(F U 1024)) )X H ! fgets( F,1024 ,stdin) )break; *(F+(y
|
||||
=strlen(F))-1)= 0; H!( F=M3(F, y)))X H stat(F,&S)==0 &&S_ISREG(S.
|
||||
st_mode)&&S O )r=J(r ,F ); } H r)P(r ); u 0; }/* Obfuscated C
|
||||
|
||||
IS FREE THINKS MONEY GROWS ON DIRECTORY TREE */
|
||||
@@ -0,0 +1,193 @@
|
||||
Most Space Efficient:
|
||||
|
||||
Jens Schweikhardt
|
||||
DFN Network Operation Center
|
||||
Schlartaeckerweg 3 (Home address)
|
||||
D-71384 Weinstadt
|
||||
Germany
|
||||
|
||||
http://www.shuttle.de/schweikh/
|
||||
|
||||
|
||||
Judges' comments:
|
||||
|
||||
To build:
|
||||
|
||||
make schweikh3
|
||||
|
||||
To use:
|
||||
|
||||
If "dir" is the directory (or a directory tree) where you keep
|
||||
all your favorite pictures off the Internet or from
|
||||
alt.binaries.pictures.*, do
|
||||
|
||||
find dir -type f -print | ./schweikh3
|
||||
|
||||
Maybe you will not need to buy another 10 Gb disk to store them. :-)
|
||||
|
||||
Also Try:
|
||||
|
||||
cp schweikh3 file1
|
||||
cp schweikh3 file2
|
||||
cp schweikh3.c file3
|
||||
ln file3 file4
|
||||
|
||||
ls -1 file? | ./schweikh3
|
||||
|
||||
NOTE: this program is not as smart as to detect identity of JPEG
|
||||
files with the same picture but with different compression levels.
|
||||
|
||||
NOTE: Some non-gcc compilers that are not fully ANSI standard do
|
||||
not compile this entry correctly.
|
||||
|
||||
|
||||
Selected author's comments:
|
||||
|
||||
The source is expected to conform to IEEE Std 1003.1-1990 ("POSIX").
|
||||
Thank God the IEEE does not standardize a coding style...
|
||||
|
||||
What this program does
|
||||
----------------------
|
||||
|
||||
This is best explained in the man page. The troff source for this
|
||||
man page can be found in the file samefile.1.
|
||||
|
||||
SAMEFILE(1) User Manuals SAMEFILE(1)
|
||||
|
||||
|
||||
NAME
|
||||
samefile - find identical files
|
||||
|
||||
SYNOPSIS
|
||||
samefile
|
||||
|
||||
DESCRIPTION
|
||||
samefile reads a list of file names (one file name per
|
||||
line) from stdin. For each file name pair with identical
|
||||
contents, a line consisting of six tab separated fields is
|
||||
output: The size in bytes, two file names, the character
|
||||
``='' if the two files are on the same device, ``X'' oth-
|
||||
erwise, and the link counts of the two files. The output
|
||||
is sorted in reverse order by size.
|
||||
|
||||
samefile uses two stages to give optimum performance.
|
||||
|
||||
In the first stage, all non-plain files are silently
|
||||
ignored (directories, devices, FIFOs, sockets, symbolic
|
||||
links) as well as files for which stat(2) or open(2)
|
||||
fails. The result of the first stage (the file names) is
|
||||
written into a binary tree with one node for every file
|
||||
size. Each node is in turn a linked list of file names
|
||||
along with inode and device information. It is also at
|
||||
this first stage where checks for hard links are done.
|
||||
For any inode only one filename will be added to the
|
||||
binary tree.
|
||||
|
||||
In the second stage all files having the same size are
|
||||
compared against each other. The rules of mathematical
|
||||
logic are applied to reduce work and output noise: if
|
||||
files A, B, and C have the same size and samefile finds
|
||||
that A = B and A = C then it will not compare B against C
|
||||
(and will not output a line for B and C) but only for A =
|
||||
B and A = C. The algorithm will detect equality across
|
||||
arbitrarily long chains. Note however, that because only
|
||||
the first filename per inode gets into the first stage,
|
||||
the output for a group of identical files with different
|
||||
inode numbers is also minimized. Suppose you have six
|
||||
identical files of size 100 in an inode group consisting
|
||||
of the three inodes with numbers 10, 20 and 30:
|
||||
|
||||
$ ls -i # output edited for readability:
|
||||
10 file1 20 file4 30 file6
|
||||
10 file2 20 file5
|
||||
10 file3
|
||||
$ ls | samefile
|
||||
100 file1 file4 = 3 2
|
||||
100 file1 file6 = 3 1
|
||||
|
||||
The sum of the sizes in the first column is the amount of
|
||||
disk space you could gain by making all 6 files links to
|
||||
only one file or remove all but one of the files. To be
|
||||
precise, disk space is allocated in blocks - you will
|
||||
probably gain two blocks here, rather than 200 bytes.
|
||||
Note that it is not enough to just remove file4 and file6
|
||||
(you would gain only 100 bytes because file5 still
|
||||
exists.)
|
||||
|
||||
|
||||
LIMITATIONS
|
||||
Samefile was written with no limits in mind. The number of
|
||||
input lines is unlimited. The size of the actual files is
|
||||
only limited by available virtual memory needed to compare
|
||||
one pair of files. The only hard limit is the constraint
|
||||
that there should not be more than about 8192 files having
|
||||
the same size. Experience has shown that there are rarely
|
||||
more than a couple dozen files of the same size.
|
||||
|
||||
EXAMPLES
|
||||
For everybody:
|
||||
What are the duplicates under my home directory?
|
||||
|
||||
$ find $HOME | samefile
|
||||
|
||||
For the sysadmin folks:
|
||||
Report all duplicate files under /usr larger than 16k:
|
||||
|
||||
$ find /usr -size +16384c -a -type f | samefile
|
||||
|
||||
For the ftp and WWW admins:
|
||||
How much space is wasted below our site's /pub directory?
|
||||
|
||||
$ find /pub -type f | samefile | awk '{sum += $1} END {print sum}'
|
||||
|
||||
|
||||
|
||||
EXIT STATUS
|
||||
If samefile runs out of memory the exit status is 1, zero
|
||||
otherwise.
|
||||
|
||||
SEE ALSO
|
||||
find(1), ln(1), rm(1)
|
||||
|
||||
|
||||
Why I think it is obfuscated
|
||||
----------------------------
|
||||
|
||||
- the code is commented but this gives only an *extremely* faint clue.
|
||||
- the layout does not reflect the program structure, or does it?
|
||||
- The data structure the program builds is an ordered binary tree
|
||||
where each node consists of a linked list. This is probably too
|
||||
heavy to comprehend for N percent of all programmers ;-)
|
||||
- The tree is built and traversed by recursive function calls.
|
||||
- Why does this program look at dev_t and ino_t? Heavy wizardry.
|
||||
- Can you find out how the algorithm works that detects identity
|
||||
of files across arbitrarily long chains? In my private unobfuscated
|
||||
source there are 17 lines describing just that. A classical case for
|
||||
the well known /* You're not supposed to understand this */ comment.
|
||||
- Would Dijkstra consider this goto harmful?
|
||||
- The only string literal is split in more than a dozen substrings
|
||||
- For the task at hand the code is pretty terse. Some dynamic memory
|
||||
is even freed as soon as it's no longer needed. I have tried to
|
||||
make the preprocessor directives as short as possible. This means
|
||||
no spaces after #include and no spaces after some of the macro
|
||||
names.
|
||||
|
||||
Are you an algorithms guru? If so you should know why the output
|
||||
appears sorted without any use of qsort() or system("sort").
|
||||
|
||||
Rewrite (hah!) the program to use memory mapping of files instead
|
||||
of read(). Does this affect performance noticeably?
|
||||
|
||||
It's easy to write a script that takes samefile's output and
|
||||
hardlinks (or symlinks) identical files. Writing such a script is
|
||||
left as an exercise for the deobfuscator.
|
||||
|
||||
When you run samefile on some directory tree, try to estimate
|
||||
how much you think it will find. Is the actual result anywhere
|
||||
near your estimate? How many copies of the GPL COPYING file
|
||||
are there on your filesystems? :-)
|
||||
|
||||
In the remote event that the input has more than 8192 files with
|
||||
the same size (on systems where sizeof (char *) == 4, or 4096 when
|
||||
sizeof (char *) == 8), increase the manifest constant 32767 on line
|
||||
31.
|
||||
Reference in new issue
Block a user