AI Agent Hub
Back to skills
Duplicate File Cleaner icon

Duplicate File Cleaner

IT Ops & Security Updated 2026.08.30

Paste the following prompt into your AI chat to install this skill:

To install this skill in your AI assistant, follow the guide at https://skillhub.cn/install/skillhub.md and use the specification: @user_b3d06a4a/duplicate-file-cleaner.

About this skill

Problem: The Dilemma of Duplicate Files on Disk

A common yet challenging issue in daily development or operations is disk space being cluttered with duplicate files (e.g., temporary build artifacts, redundantly downloaded dependency packages, multiple backups of codebases). Manually finding these files is time-consuming and carries a high risk of accidental deletion, especially when filenames differ. We need a tool that can accurately identify and safely handle these duplicates.

Solution: An Intelligent Scanning and Safe Cleanup Workflow

This skill (@user_b3d06a4a/duplicate-file-cleaner) provides a structured workflow to address this problem by executing a series of scripts, forming a closed loop from identification to cleanup.

1. Multi-Dimensional Duplicate File Identification

The core tool is scripts/duplicate_scanner.py, which supports three scanning strategies:
* comprehensive: A combined strategy using both content and metadata, offering the highest accuracy. Recommended.
* content: Identifies files based solely on content (e.g., hash values), ensuring byte-for-byte consistency.
* metadata: A fast filter based on filename, size, and modification time, which may include false positives.

You can specify the target directory with the --directory parameter and filter specific file types (e.g., jpg,png,pdf) using --extensions.

2. Risk Assessment and Intelligent Decision-Making

After scanning, scripts/risk_assessor.py evaluates the operational risk, outputting a risk level from low to high along with specific contributing factors. This prevents blind deletion. Subsequently, the skill's "agent" interprets the report, recommends strategies based on the risk level, identifies high-value files (e.g., the earliest created version, critical system files), and generates a deletion list annotated with high-risk items.

3. Safe File Organization and Deletion

For non-duplicate files that need categorization, scripts/file_organizer.py can be invoked to organize them by type (type), date (date), or size (size) with a preview. For the final deletion, scripts/safety_manager.py provides four layers of protection: automatic backup, operation confirmation, log recording, and reversal within 30 days, minimizing the risk of data loss.

Boundaries and Important Considerations

  • Applicable Scenarios: This skill is best suited for handling duplicate files in user directories (e.g., ~/Downloads, ~/Documents) or project build directories. It is not recommended for critical system directories (e.g., C:\Windows\System32 on Windows or /usr on Linux).
  • Risk Control: When handling high-risk directories or if uncertain, always use the --dry-run parameter for a preview that does not execute actual deletion.
  • Strategy Selection: The metadata strategy is fast but may include non-duplicate files, so results should be reviewed carefully. For critical cleanup tasks, the comprehensive strategy is recommended.
  • Backup Management: The default backup function occupies space. Please regularly manage old backups via the cleanup mechanisms in scripts/safety_manager.py.

Use Cases

  • After a project concludes, the need arises to clean up multiple duplicate build artifact directories and backup files to free up server disk space, while ensuring no important codebases or configuration files are mistakenly deleted.
  • A photographer or designer has accumulated tens of thousands of images and asset files, requiring quick identification and safe cleanup of identical or similar multiple versions through content hashing and metadata analysis to free up storage for new projects.
  • A system administrator conducting periodic reviews of `/home` user directories needs to identify duplicate documents and log files generated from downloads and copies, organize them according to strategy (e.g., archive by date), and generate a risk report for team review.
  • A developer needs to quickly scan an entire workspace to find duplicate dependency packages (e.g., node_modules, .jar files) and test data, reducing repository size through safe deletion.

Best For

  • An operations engineer responsible for managing server storage space, needing to regularly perform duplicate file cleanup and archiving tasks with risk controls.
  • A software developer who needs to archive or clean up large project directories after project completion, seeking to safely delete redundant build caches and test data.
  • A creative worker managing a personal or work media library (e.g., photos, videos, design assets) who needs to organize and free up storage space.
  • An IT manager responsible for maintaining a team file server or cloud storage, needing to develop and implement file cleanup policies.