smart-pip-collector ^v0.1.0

• Recursive search — finds all requirements.txt in nested folders
• Version awareness — downloads specific package versions, creating package/version/ structure
• No duplicates — groups same packages with different versions
• Multithreading — uses all CPU cores for maximum speed
• Detailed reports — JSON reports for each download + global report
• Graceful shutdown — proper Ctrl+C handling (1 press — stop, 2 — force)
• Project tracking — stores information about which projects use which dependency
• Resume mode — retry failed packages

git clone https://github.com/smartlegionlab/smart-pip-collector.git
cd smart-pip-collector

# Basic run
python app.py /path/to/your/projects

# With custom output directory
python app.py /path/to/your/projects --output ~/my_pip_backup

# With custom thread count
python app.py /path/to/your/projects --workers 16

# Resume failed downloads
python app.py /path/to/your/projects --resume

~/pip_packages/
├── requests/
│   └── 2.28.1/
│       ├── requests-2.28.1-py3-none-any.whl
│       └── .download_report.json
├── django/
│   └── 4.2.1/
│       ├── Django-4.2.1-py3-none-any.whl
│       └── .download_report.json
├── global_download_report.json
└── all_requirements.txt

Local report (in each package folder):

{
  "package": "requests",
  "version": "2.28.1",
  "download_time": "2024-01-15T10:30:00",
  "source_projects": ["project_a", "project_b"],
  "status": "success",
  "files": ["requests-2.28.1-py3-none-any.whl"]
}

Global report (global_download_report.json):

• Overall statistics
• Detailed information for each package
• Rate limiting and cancellation markers

• Packages requiring compilation (with C extensions) may fail without system dependencies
• Non-standard links (git+, http) are ignored
• PyPI rate limiting is possible with very large number of requests

By using this software, you agree to the full disclaimer terms.

Software provided "AS IS" without warranty. You assume all risks.

Full legal disclaimer: See DISCLAIMER.md

License: BSD 3-Clause License