Smart File Duplicate Manager RS

Detection

• Multi-stage pipeline: size → prefix hash (4 KB) → full BLAKE3 hash → byte-by-byte confirmation
• Reads full file contents only for real candidates, not for every file
• Parallel hashing via rayon
• Optional sampling mode for very large files (opt-in, off by default)
• Symlink-aware (--follow-links)

Control

• Dry-run by default for every destructive action
• Reversible trash action (XDG-compliant)
• Interactive mode: confirm each file individually
• Validation of size and mtime before each action (in --from-report mode)
• Never touches anything without an explicit flag

Actions

• report — show groups, delete nothing (default)
• trash — move duplicates to XDG trash (reversible)
• move — move duplicates to a chosen folder
• delete — permanently remove duplicates
• hardlink — replace duplicates with hard links to the kept file

Strategies & Filters

• Keep first (alphabetical), newest, oldest, or shortest path
• Min/max file size (with human-readable suffixes)
• Extension allow-list
• Path exclusion (substring match, prunes subtrees)
• Hidden files (excluded by default)

Output and integration

• Plain text with ANSI colors (auto-detected)
• JSON output for scripts and pipelines
• Shell script output for manual review
• Reusable reports: --from-report <FILE> skips scanning
• Progress bars for long-running phases
• Quiet mode when piping JSON

Quality

• 68 unit and integration tests
• No unsafe
• No warnings on cargo build --release
• Tests run in isolated temporary directories

Duplicate detection runs in five phases. Each phase filters the candidate set further, so that expensive operations run only on files that survived all previous checks.

1. Scan — walk the directory tree, collect files (path, size, mtime)
2. Group by size — files of different sizes cannot be duplicates
3. Prefix hash — hash the first 4 KB of each candidate (fast filter)
4. Full hash — BLAKE3 over full content, parallelized with rayon
5. Confirm — byte-by-byte comparison to rule out hash collisions

With the optional --sample-chunk flag, phases 4 and 5 read only three chunks per file (start, middle, end) instead of the whole file. This is a probabilistic check.

Requires Rust 1.70 or newer.

git clone https://github.com/smartlegionlab/smart-file-duplicate-manager-rs
cd smart-file-duplicate-manager-rs
cargo build --release

Binary: target/release/smart_file_duplicate_manager

Install for system-wide use (Linux):

mkdir -p ~/.local/bin
ln -sf "$PWD/target/release/smart_file_duplicate_manager" ~/.local/bin/sfdm

Make sure ~/.local/bin is in your PATH. Add this to ~/.bashrc if needed:

export PATH="$HOME/.local/bin:$PATH"

Verify:

which sfdm
sfdm --version

Basic:

smart_file_duplicate_manager --path ~/Downloads
smart_file_duplicate_manager -p ~/Pictures

Options:

Size format: K/KB/KiB, M/MB/MiB, G/GB/GiB, T/TB/TiB. All suffixes are 1024-based (IEC).

Report only (safe):

smart_file_duplicate_manager --path ~/Photos

Only images, keep the newest copy, show top 10 groups:

smart_file_duplicate_manager --path ~/Photos \
    --ext jpg,jpeg,png,heic \
    --keep newest \
    --limit 10

Find duplicates larger than 10 MB, exclude .git and node_modules:

smart_file_duplicate_manager --path ~/Projects \
    --min-size 10M \
    --exclude .git,node_modules,target

Preview trash operation (dry-run):

smart_file_duplicate_manager --path ~/Downloads --action trash

Execute trash operation:

smart_file_duplicate_manager --path ~/Downloads --action trash --yes

Execute trash operation with per-file confirmation:

smart_file_duplicate_manager --path ~/Downloads --action trash --interactive

Export JSON report to file:

smart_file_duplicate_manager --path ~/Music --output json --output-file report.json

Scan and report

# Report duplicates in a directory (safe, nothing is deleted)
smart_file_duplicate_manager --path ~/Downloads

# Report duplicates in current directory
smart_file_duplicate_manager --path .

# Only files larger than 100 MB
smart_file_duplicate_manager --path ~/Videos --min-size 100M

# Only files between 10 MB and 1 GB
smart_file_duplicate_manager --path ~/Videos --min-size 10M --max-size 1G

# Only images
smart_file_duplicate_manager --path ~/Photos --ext jpg,jpeg,png,heic

# Exclude .git and node_modules
smart_file_duplicate_manager --path ~/Projects --exclude .git,node_modules,target

# Include hidden files
smart_file_duplicate_manager --path ~/Documents --hidden

# Show top 10 groups only
smart_file_duplicate_manager --path ~/Music --limit 10

# Show directories with the most duplicates
smart_file_duplicate_manager --path ~/Videos --group-by-dir

# No color (for logs or files)
smart_file_duplicate_manager --path ~/Downloads --color never

Keep strategies

# Keep the first by path (alphabetical)
smart_file_duplicate_manager --path ~/Photos --keep first

# Keep the most recently modified
smart_file_duplicate_manager --path ~/Photos --keep newest

# Keep the oldest by modification time
smart_file_duplicate_manager --path ~/Photos --keep oldest

# Keep the file with the shortest path
smart_file_duplicate_manager --path ~/Photos --keep shortest

Actions (dry-run by default)

# Dry-run (no changes, just the plan)
smart_file_duplicate_manager --path ~/Downloads --action trash
smart_file_duplicate_manager --path ~/Downloads --action delete
smart_file_duplicate_manager --path ~/Downloads --action move --action-dir /tmp/dupes_out
smart_file_duplicate_manager --path ~/Downloads --action hardlink

# Move duplicates to ~/.local/share/Trash (reversible)
smart_file_duplicate_manager --path ~/Downloads --action trash --yes

# Move duplicates to a custom folder
smart_file_duplicate_manager --path ~/Downloads --action move --action-dir /tmp/dupes_out --yes

# Replace duplicates with hard links to the kept file
smart_file_duplicate_manager --path ~/Downloads --action hardlink --yes

# Permanently delete duplicates (irreversible)
smart_file_duplicate_manager --path ~/Downloads --action delete --yes

Interactive mode

# Confirm each file individually (requires a TTY)
smart_file_duplicate_manager --path ~/Downloads --action trash --interactive

# Interactive with a saved report (no rescanning)
smart_file_duplicate_manager --from-report /tmp/dupes.json --action trash --interactive

Prompt keys: y — do it · n or Enter — skip · a — apply to all remaining · q — quit

Fast mode for large files

# Sample 3 chunks per large file (fast, not guaranteed)
smart_file_duplicate_manager --path /media/data --sample-chunk 4M

# Sample only files >= 500 MB, with 4 MB chunks
smart_file_duplicate_manager --path /media/data \
    --min-size 500M \
    --sample-chunk 4M \
    --sample-threshold 500M

Reuse a saved report (no rescanning)

# Step 1: scan once and save
smart_file_duplicate_manager --path /media/data \
    --min-size 100M \
    --output json --output-file /tmp/dupes.json

# Step 2: inspect the plan (instant)
smart_file_duplicate_manager --from-report /tmp/dupes.json

# Step 3: apply an action (still dry-run unless --yes is given)
smart_file_duplicate_manager --from-report /tmp/dupes.json --action trash

# Step 4: execute
smart_file_duplicate_manager --from-report /tmp/dupes.json --action trash --yes

Shell script output

# Generate a delete script (does NOT execute it)
smart_file_duplicate_manager --path /media/data \
    --action delete --output shell --output-file /tmp/cleanup.sh

less /tmp/cleanup.sh     # inspect
bash -n /tmp/cleanup.sh  # syntax check
bash /tmp/cleanup.sh     # execute when ready

JSON output:

{
  "path": "/home/user/Downloads",
  "keep_strategy": "First",
  "action": "Report",
  "groups": [
    {
      "index": 1,
      "size": 5242880,
      "wasted": 5242880,
      "keep": {
        "path": "/home/user/Downloads/file.bin",
        "size": 5242880,
        "mtime": 1700000000
      },
      "delete": [
        {
          "path": "/home/user/Downloads/file copy.bin",
          "size": 5242880,
          "mtime": 1700000000
        }
      ]
    }
  ],
  "total_groups": 1,
  "total_files": 2,
  "total_wasted": 5242880
}

When --output json is combined with --output-file, the file contains pure JSON. Progress bars, phase logs, and informational messages go to stderr.

Shell script output:

Not supported: --action report (use --output text) · --action trash (XDG trash requires metadata; use directly).

Exit codes:

• 0 — success
• 1 — invalid arguments, path does not exist, or report cannot be loaded

Testing:

cargo test

Runs unit and integration tests in isolated temporary directories. Never touches real files.

• Language: Rust 1.70+
• Hashing: BLAKE3 (prefix + full)
• Parallelism: rayon
• Platform: Linux / macOS / Windows
• License: BSD 3-Clause

By using this software, you agree to the full disclaimer terms.

Software provided "AS IS" without warranty. You assume all risks.

Full legal disclaimer: See DISCLAIMER.md

License: BSD 3-Clause License