mirror/ioccc
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fixes and updates to 1998/schweikh3

Browse Source 
Fixed a command to try.

Fixed formatting.

Updated .gitignore.

Fixed URL.
This commit is contained in:
Cody Boone Ferguson committed 2023-04-18 09:18:35 -07:00
1 parent e45c0d1755
commit 591d8774ef
2 files changed
+163 -133
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

1998/schweikh3/.gitignore
+4
View File
@@ -1 +1,5 @@
schweikh3 schweikh3
file1
file2
file3
file4
1998/schweikh3/README.md
+159 -133
View File
@@ -1,203 +1,229 @@
# Most Space Efficient # Most Space Efficient
Jens Schweikhardt Jens Schweikhardt
DFN Network Operation Center DFN Network Operation Center
Schlartaeckerweg 3 (Home address) Schlartaeckerweg 3 (Home address)
D-71384 Weinstadt D-71384 Weinstadt
Germany Germany
<http://www.schweikhardt.net>
http://www.shuttle.de/schweikh/
## To build: ## To build:
make all ```sh
make all
```
## To run; ## To run:
If "dir" is the directory (or a directory tree) where you keep If "dir" is the directory (or a directory tree) where you keep
all your favorite pictures off the Internet or from all your favorite pictures off the Internet or from
`alt.binaries.pictures.*`, do `alt.binaries.pictures.*`, do
find dir -type f -print | ./schweikh3 ```sh
find dir -type f -print | ./schweikh3
```
Maybe you will not need to buy another 10 Gb disk to store them. :-)
## Try: ## Try:
If you're in this winning entry's directory: If you're in this winning entry's directory:
find . -type f -print | ./schweikh3 ```sh
find . -type f -print | ./schweikh3
```
Notice that the tool finds a duplicate file, that of the man page. Notice that the tool finds a duplicate file, that of the man page.
## Judges' remarks: ## Judges' remarks:
Maybe you will not need to buy another 10 Gb disk to store them. :-)
Also Try: Try this:
cp schweikh3 file1 ```sh
cp schweikh3 file2 cp schweikh3 file1
cp schweikh3.c file3 cp schweikh3 file2
ln file3 file4 cp schweikh3.c file3
ln file3 file4
ls -1 file? | ./schweikh3 find . -type f -name 'file?' | ./schweikh3
```
NOTE: this program is not as smart as to detect identity of JPEG This program is not as smart as to detect identity of JPEG files with the same
files with the same picture but with different compression levels. picture but with different compression levels. On the other hand compression can
make it thing look/sound/feel different so these probably aren't duplicates
anyway. :-)
NOTE: Some non-gcc compilers that are not fully ANSI standard do NOTE: Some non-gcc compilers that are not fully ANSI standard do not compile
not compile this entry correctly. this entry correctly.
## Author's remarks ## Author's remarks
The source is expected to conform to IEEE Std 1003.1-1990 ("POSIX"). The source is expected to conform to IEEE Std 1003.1-1990 ("POSIX").
Thank God the IEEE does not standardize a coding style... Thank God the IEEE does not standardize a coding style...
What this program does ### What this program does
----------------------
This is best explained in the man page. The troff source for this This is best explained in the man page. The troff source for this
man page can be found in the file samefile.1. man page can be found in the file samefile.1. To render try: `man
./samefile.1`. Otherwise:
SAMEFILE(1) User Manuals SAMEFILE(1)
NAME **SAMEFILE(1) User Manuals SAMEFILE(1)**
samefile - find identical files
SYNOPSIS
samefile
DESCRIPTION **NAME**
samefile reads a list of file names (one file name per
line) from stdin. For each file name pair with identical
contents, a line consisting of six tab separated fields is
output: The size in bytes, two file names, the character
``='' if the two files are on the same device, ``X'' oth-
erwise, and the link counts of the two files. The output
is sorted in reverse order by size.
samefile uses two stages to give optimum performance. **samefile** - find identical files
In the first stage, all non-plain files are silently **SYNOPSIS**
ignored (directories, devices, FIFOs, sockets, symbolic
links) as well as files for which stat(2) or open(2)
fails. The result of the first stage (the file names) is
written into a binary tree with one node for every file
size. Each node is in turn a linked list of file names
along with inode and device information. It is also at
this first stage where checks for hard links are done.
For any inode only one filename will be added to the
binary tree.
In the second stage all files having the same size are **./samefile**
compared against each other. The rules of mathematical
logic are applied to reduce work and output noise: if
files A, B, and C have the same size and samefile finds
that A = B and A = C then it will not compare B against C
(and will not output a line for B and C) but only for A =
B and A = C. The algorithm will detect equality across
arbitrarily long chains. Note however, that because only
the first filename per inode gets into the first stage,
the output for a group of identical files with different
inode numbers is also minimized. Suppose you have six
identical files of size 100 in an inode group consisting
of the three inodes with numbers 10, 20 and 30:
**DESCRIPTION**
**samefile** reads a list of file names (one file name per
line) from stdin. For each file name pair with identical
contents, a line consisting of six tab separated fields is
output: The size in bytes, two file names, the character
`=` if the two files are on the same device, `X` otherwise,
and the link counts of the two files. The output
is sorted in reverse order by size.
samefile uses two stages to give optimum performance.
In the first stage, all non-plain files are silently
ignored (directories, devices, FIFOs, sockets, symbolic
links) as well as files for which `stat(2)` or `open(2)`
fails. The result of the first stage (the file names) is
written into a binary tree with one node for every file
size. Each node is in turn a linked list of file names
along with inode and device information. It is also at
this first stage where checks for hard links are done.
For any inode only one filename will be added to the
binary tree.
In the second stage all files having the same size are
compared against each other. The rules of mathematical
logic are applied to reduce work and output noise: if
files A, B, and C have the same size and samefile finds
that A = B and A = C then it will not compare B against C
(and will not output a line for B and C) but only for A =
B and A = C. The algorithm will detect equality across
arbitrarily long chains. Note however, that because only
the first filename per inode gets into the first stage,
the output for a group of identical files with different
inode numbers is also minimized. Suppose you have six
identical files of size 100 in an inode group consisting
of the three inodes with numbers 10, 20 and 30:
```sh
$ ls -i # output edited for readability: $ ls -i # output edited for readability:
10 file1 20 file4 30 file6 10 file1 20 file4 30 file6
10 file2 20 file5 10 file2 20 file5
10 file3 10 file3
$ ls | samefile $ ls | ./samefile
100 file1 file4 = 3 2 100 file1 file4 = 3 2
100 file1 file6 = 3 1 100 file1 file6 = 3 1
```
The sum of the sizes in the first column is the amount of The sum of the sizes in the first column is the amount of
disk space you could gain by making all 6 files links to disk space you could gain by making all 6 files links to
only one file or remove all but one of the files. To be only one file or remove all but one of the files. To be
precise, disk space is allocated in blocks - you will precise, disk space is allocated in blocks - you will
probably gain two blocks here, rather than 200 bytes. probably gain two blocks here, rather than 200 bytes.
Note that it is not enough to just remove file4 and file6 Note that it is not enough to just remove `file4` and `file6`
(you would gain only 100 bytes because file5 still (you would gain only 100 bytes because `file5` still
exists.) exists.)
LIMITATIONS **LIMITATIONS**
Samefile was written with no limits in mind. The number of
input lines is unlimited. The size of the actual files is
only limited by available virtual memory needed to compare
one pair of files. The only hard limit is the constraint
that there should not be more than about 8192 files having
the same size. Experience has shown that there are rarely
more than a couple dozen files of the same size.
EXAMPLES `samefile` was written with no limits in mind. The number of
For everybody: input lines is unlimited. The size of the actual files is
What are the duplicates under my home directory? only limited by available virtual memory needed to compare
one pair of files. The only hard limit is the constraint
that there should not be more than about 8192 files having
the same size. Experience has shown that there are rarely
more than a couple dozen files of the same size.
$ find $HOME | samefile **EXAMPLES**
For the sysadmin folks: For everybody:
Report all duplicate files under /usr larger than 16k:
$ find /usr -size +16384c -a -type f | samefile What are the duplicates under my home directory?
For the ftp and WWW admins: ```sh
How much space is wasted below our site's /pub directory? $ find $HOME | ./samefile
```
$ find /pub -type f | samefile | awk '{sum += $1} END {print sum}' For the sysadmin folks:
Report all duplicate files under /usr larger than 16k:
```sh
$ find /usr -size +16384c -a -type f | ./samefile
```
For the ftp and WWW admins:
How much space is wasted below our site's /pub directory?
```sh
$ find /pub -type f | samefile | awk '{sum += $1} END {print sum}'
```
**EXIT STATUS**
EXIT STATUS If `samefile` runs out of memory the exit status is 1, zero
If samefile runs out of memory the exit status is 1, zero otherwise.
otherwise.
SEE ALSO **SEE ALSO**
find(1), ln(1), rm(1)
`find(1)`, `ln(1)`, `rm(1)`
Why I think it is obfuscated
---------------------------- ### Why I think it is obfuscated
- the code is commented but this gives only an *extremely* faint clue. - The code is commented but this gives only an *extremely* faint clue.
- the layout does not reflect the program structure, or does it? - The layout does not reflect the program structure, or does it?
- The data structure the program builds is an ordered binary tree - The data structure the program builds is an ordered binary tree
where each node consists of a linked list. This is probably too where each node consists of a linked list. This is probably too
heavy to comprehend for N percent of all programmers ;-) heavy to comprehend for N percent of all programmers ;-)
- The tree is built and traversed by recursive function calls. - The tree is built and traversed by recursive function calls.
- Why does this program look at dev_t and ino_t? Heavy wizardry. - Why does this program look at `dev_t` and `ino_t`? Heavy wizardry.
- Can you find out how the algorithm works that detects identity - Can you find out how the algorithm works that detects identity
of files across arbitrarily long chains? In my private unobfuscated of files across arbitrarily long chains? In my private unobfuscated
source there are 17 lines describing just that. A classical case for source there are 17 lines describing just that. A classical case for
the well known /* You're not supposed to understand this */ comment. the well known `/* You're not supposed to understand this */` comment.
- Would Dijkstra consider this goto harmful? - Would Dijkstra consider this goto harmful?
- The only string literal is split in more than a dozen substrings - The only string literal is split in more than a dozen substrings
- For the task at hand the code is pretty terse. Some dynamic memory - For the task at hand the code is pretty terse. Some dynamic memory
is even freed as soon as it's no longer needed. I have tried to is even freed as soon as it's no longer needed. I have tried to
make the preprocessor directives as short as possible. This means make the preprocessor directives as short as possible. This means
no spaces after #include and no spaces after some of the macro no spaces after #include and no spaces after some of the macro
names. names.
Are you an algorithms guru? If so you should know why the output Are you an algorithms guru? If so you should know why the output
appears sorted without any use of qsort() or system("sort"). appears sorted without any use of `qsort()` or `system("sort")`.
Rewrite (hah!) the program to use memory mapping of files instead Rewrite (hah!) the program to use memory mapping of files instead
of read(). Does this affect performance noticeably? of `read()`. Does this affect performance noticeably?
It's easy to write a script that takes samefile's output and It's easy to write a script that takes `samefile`'s output and
hardlinks (or symlinks) identical files. Writing such a script is hard links (or symlinks) identical files. Writing such a script is
left as an exercise for the deobfuscator. left as an exercise for the deobfuscator.
When you run samefile on some directory tree, try to estimate When you run `samefile` on some directory tree, try to estimate
how much you think it will find. Is the actual result anywhere how much you think it will find. Is the actual result anywhere
near your estimate? How many copies of the GPL COPYING file near your estimate? How many copies of the GPL COPYING file
are there on your filesystems? :-) are there on your filesystems? :-)
In the remote event that the input has more than 8192 files with In the remote event that the input has more than 8192 files with
the same size (on systems where sizeof (char *) == 4, or 4096 when the same size (on systems where `sizeof (char *) == 4`, or 4096 when
sizeof (char *) == 8), increase the manifest constant 32767 on line `sizeof (char *) == 8`), increase the manifest constant 32767 on line
31. 31.
## Copyright and CC BY-SA 4.0 License: ## Copyright and CC BY-SA 4.0 License: