From 591d8774ef58b313ab5c9dc4548759667ac0723e Mon Sep 17 00:00:00 2001 From: Cody Boone Ferguson <53008573+xexyl@users.noreply.github.com> Date: Tue, 18 Apr 2023 09:02:53 -0700 Subject: [PATCH] Fixes and updates to 1998/schweikh3 Fixed a command to try. Fixed formatting. Updated .gitignore. Fixed URL. --- 1998/schweikh3/.gitignore | 4 + 1998/schweikh3/README.md | 292 +++++++++++++++++++++----------------- 2 files changed, 163 insertions(+), 133 deletions(-) diff --git a/1998/schweikh3/.gitignore b/1998/schweikh3/.gitignore index fc8e75c24..30f4eadf5 100644 --- a/1998/schweikh3/.gitignore +++ b/1998/schweikh3/.gitignore @@ -1 +1,5 @@ schweikh3 +file1 +file2 +file3 +file4 diff --git a/1998/schweikh3/README.md b/1998/schweikh3/README.md index 976ff03d5..a2cc16e9b 100644 --- a/1998/schweikh3/README.md +++ b/1998/schweikh3/README.md @@ -1,203 +1,229 @@ # Most Space Efficient - Jens Schweikhardt - DFN Network Operation Center - Schlartaeckerweg 3 (Home address) - D-71384 Weinstadt - Germany - - http://www.shuttle.de/schweikh/ +Jens Schweikhardt +DFN Network Operation Center +Schlartaeckerweg 3 (Home address) +D-71384 Weinstadt +Germany + ## To build: - make all +```sh +make all +``` -## To run; +## To run: If "dir" is the directory (or a directory tree) where you keep all your favorite pictures off the Internet or from `alt.binaries.pictures.*`, do - find dir -type f -print | ./schweikh3 +```sh +find dir -type f -print | ./schweikh3 +``` + +Maybe you will not need to buy another 10 Gb disk to store them. :-) ## Try: If you're in this winning entry's directory: - find . -type f -print | ./schweikh3 +```sh +find . -type f -print | ./schweikh3 +``` Notice that the tool finds a duplicate file, that of the man page. ## Judges' remarks: - Maybe you will not need to buy another 10 Gb disk to store them. :-) - Also Try: +Try this: - cp schweikh3 file1 - cp schweikh3 file2 - cp schweikh3.c file3 - ln file3 file4 +```sh +cp schweikh3 file1 +cp schweikh3 file2 +cp schweikh3.c file3 +ln file3 file4 - ls -1 file? | ./schweikh3 +find . -type f -name 'file?' | ./schweikh3 +``` - NOTE: this program is not as smart as to detect identity of JPEG - files with the same picture but with different compression levels. +This program is not as smart as to detect identity of JPEG files with the same +picture but with different compression levels. On the other hand compression can +make it thing look/sound/feel different so these probably aren't duplicates +anyway. :-) - NOTE: Some non-gcc compilers that are not fully ANSI standard do - not compile this entry correctly. +NOTE: Some non-gcc compilers that are not fully ANSI standard do not compile +this entry correctly. ## Author's remarks - The source is expected to conform to IEEE Std 1003.1-1990 ("POSIX"). - Thank God the IEEE does not standardize a coding style... +The source is expected to conform to IEEE Std 1003.1-1990 ("POSIX"). +Thank God the IEEE does not standardize a coding style... - What this program does - ---------------------- +### What this program does - This is best explained in the man page. The troff source for this - man page can be found in the file samefile.1. - - SAMEFILE(1) User Manuals SAMEFILE(1) +This is best explained in the man page. The troff source for this +man page can be found in the file samefile.1. To render try: `man +./samefile.1`. Otherwise: - NAME - samefile - find identical files +**SAMEFILE(1) User Manuals SAMEFILE(1)** - SYNOPSIS - samefile - DESCRIPTION - samefile reads a list of file names (one file name per - line) from stdin. For each file name pair with identical - contents, a line consisting of six tab separated fields is - output: The size in bytes, two file names, the character - ``='' if the two files are on the same device, ``X'' oth- - erwise, and the link counts of the two files. The output - is sorted in reverse order by size. +**NAME** - samefile uses two stages to give optimum performance. + **samefile** - find identical files - In the first stage, all non-plain files are silently - ignored (directories, devices, FIFOs, sockets, symbolic - links) as well as files for which stat(2) or open(2) - fails. The result of the first stage (the file names) is - written into a binary tree with one node for every file - size. Each node is in turn a linked list of file names - along with inode and device information. It is also at - this first stage where checks for hard links are done. - For any inode only one filename will be added to the - binary tree. +**SYNOPSIS** - In the second stage all files having the same size are - compared against each other. The rules of mathematical - logic are applied to reduce work and output noise: if - files A, B, and C have the same size and samefile finds - that A = B and A = C then it will not compare B against C - (and will not output a line for B and C) but only for A = - B and A = C. The algorithm will detect equality across - arbitrarily long chains. Note however, that because only - the first filename per inode gets into the first stage, - the output for a group of identical files with different - inode numbers is also minimized. Suppose you have six - identical files of size 100 in an inode group consisting - of the three inodes with numbers 10, 20 and 30: + **./samefile** +**DESCRIPTION** + +**samefile** reads a list of file names (one file name per +line) from stdin. For each file name pair with identical +contents, a line consisting of six tab separated fields is +output: The size in bytes, two file names, the character +`=` if the two files are on the same device, `X` otherwise, +and the link counts of the two files. The output +is sorted in reverse order by size. + +samefile uses two stages to give optimum performance. + +In the first stage, all non-plain files are silently +ignored (directories, devices, FIFOs, sockets, symbolic +links) as well as files for which `stat(2)` or `open(2)` +fails. The result of the first stage (the file names) is +written into a binary tree with one node for every file +size. Each node is in turn a linked list of file names +along with inode and device information. It is also at +this first stage where checks for hard links are done. +For any inode only one filename will be added to the +binary tree. + +In the second stage all files having the same size are +compared against each other. The rules of mathematical +logic are applied to reduce work and output noise: if +files A, B, and C have the same size and samefile finds +that A = B and A = C then it will not compare B against C +(and will not output a line for B and C) but only for A = +B and A = C. The algorithm will detect equality across +arbitrarily long chains. Note however, that because only +the first filename per inode gets into the first stage, +the output for a group of identical files with different +inode numbers is also minimized. Suppose you have six +identical files of size 100 in an inode group consisting +of the three inodes with numbers 10, 20 and 30: + +```sh $ ls -i # output edited for readability: 10 file1 20 file4 30 file6 10 file2 20 file5 10 file3 - $ ls | samefile + $ ls | ./samefile 100 file1 file4 = 3 2 100 file1 file6 = 3 1 +``` - The sum of the sizes in the first column is the amount of - disk space you could gain by making all 6 files links to - only one file or remove all but one of the files. To be - precise, disk space is allocated in blocks - you will - probably gain two blocks here, rather than 200 bytes. - Note that it is not enough to just remove file4 and file6 - (you would gain only 100 bytes because file5 still - exists.) +The sum of the sizes in the first column is the amount of +disk space you could gain by making all 6 files links to +only one file or remove all but one of the files. To be +precise, disk space is allocated in blocks - you will +probably gain two blocks here, rather than 200 bytes. +Note that it is not enough to just remove `file4` and `file6` +(you would gain only 100 bytes because `file5` still +exists.) - LIMITATIONS - Samefile was written with no limits in mind. The number of - input lines is unlimited. The size of the actual files is - only limited by available virtual memory needed to compare - one pair of files. The only hard limit is the constraint - that there should not be more than about 8192 files having - the same size. Experience has shown that there are rarely - more than a couple dozen files of the same size. +**LIMITATIONS** - EXAMPLES - For everybody: - What are the duplicates under my home directory? +`samefile` was written with no limits in mind. The number of +input lines is unlimited. The size of the actual files is +only limited by available virtual memory needed to compare +one pair of files. The only hard limit is the constraint +that there should not be more than about 8192 files having +the same size. Experience has shown that there are rarely +more than a couple dozen files of the same size. - $ find $HOME | samefile +**EXAMPLES** - For the sysadmin folks: - Report all duplicate files under /usr larger than 16k: +For everybody: - $ find /usr -size +16384c -a -type f | samefile +What are the duplicates under my home directory? - For the ftp and WWW admins: - How much space is wasted below our site's /pub directory? +```sh + $ find $HOME | ./samefile +``` - $ find /pub -type f | samefile | awk '{sum += $1} END {print sum}' +For the sysadmin folks: + +Report all duplicate files under /usr larger than 16k: + +```sh + $ find /usr -size +16384c -a -type f | ./samefile +``` + +For the ftp and WWW admins: +How much space is wasted below our site's /pub directory? + +```sh + $ find /pub -type f | samefile | awk '{sum += $1} END {print sum}' +``` +**EXIT STATUS** - EXIT STATUS - If samefile runs out of memory the exit status is 1, zero - otherwise. +If `samefile` runs out of memory the exit status is 1, zero +otherwise. - SEE ALSO - find(1), ln(1), rm(1) +**SEE ALSO** +`find(1)`, `ln(1)`, `rm(1)` - Why I think it is obfuscated - ---------------------------- + +### Why I think it is obfuscated - - the code is commented but this gives only an *extremely* faint clue. - - the layout does not reflect the program structure, or does it? - - The data structure the program builds is an ordered binary tree - where each node consists of a linked list. This is probably too - heavy to comprehend for N percent of all programmers ;-) - - The tree is built and traversed by recursive function calls. - - Why does this program look at dev_t and ino_t? Heavy wizardry. - - Can you find out how the algorithm works that detects identity - of files across arbitrarily long chains? In my private unobfuscated - source there are 17 lines describing just that. A classical case for - the well known /* You're not supposed to understand this */ comment. - - Would Dijkstra consider this goto harmful? - - The only string literal is split in more than a dozen substrings - - For the task at hand the code is pretty terse. Some dynamic memory - is even freed as soon as it's no longer needed. I have tried to - make the preprocessor directives as short as possible. This means - no spaces after #include and no spaces after some of the macro - names. +- The code is commented but this gives only an *extremely* faint clue. +- The layout does not reflect the program structure, or does it? +- The data structure the program builds is an ordered binary tree + where each node consists of a linked list. This is probably too + heavy to comprehend for N percent of all programmers ;-) +- The tree is built and traversed by recursive function calls. +- Why does this program look at `dev_t` and `ino_t`? Heavy wizardry. +- Can you find out how the algorithm works that detects identity + of files across arbitrarily long chains? In my private unobfuscated + source there are 17 lines describing just that. A classical case for + the well known `/* You're not supposed to understand this */` comment. +- Would Dijkstra consider this goto harmful? +- The only string literal is split in more than a dozen substrings +- For the task at hand the code is pretty terse. Some dynamic memory + is even freed as soon as it's no longer needed. I have tried to + make the preprocessor directives as short as possible. This means + no spaces after #include and no spaces after some of the macro + names. - Are you an algorithms guru? If so you should know why the output - appears sorted without any use of qsort() or system("sort"). +Are you an algorithms guru? If so you should know why the output +appears sorted without any use of `qsort()` or `system("sort")`. - Rewrite (hah!) the program to use memory mapping of files instead - of read(). Does this affect performance noticeably? +Rewrite (hah!) the program to use memory mapping of files instead +of `read()`. Does this affect performance noticeably? - It's easy to write a script that takes samefile's output and - hardlinks (or symlinks) identical files. Writing such a script is - left as an exercise for the deobfuscator. +It's easy to write a script that takes `samefile`'s output and +hard links (or symlinks) identical files. Writing such a script is +left as an exercise for the deobfuscator. - When you run samefile on some directory tree, try to estimate - how much you think it will find. Is the actual result anywhere - near your estimate? How many copies of the GPL COPYING file - are there on your filesystems? :-) +When you run `samefile` on some directory tree, try to estimate +how much you think it will find. Is the actual result anywhere +near your estimate? How many copies of the GPL COPYING file +are there on your filesystems? :-) - In the remote event that the input has more than 8192 files with - the same size (on systems where sizeof (char *) == 4, or 4096 when - sizeof (char *) == 8), increase the manifest constant 32767 on line - 31. +In the remote event that the input has more than 8192 files with +the same size (on systems where `sizeof (char *) == 4`, or 4096 when +`sizeof (char *) == 8`), increase the manifest constant 32767 on line +31. ## Copyright and CC BY-SA 4.0 License: