Smart Pip Collector
Utility for collecting all dependencies from requirements.txt files
Download archives for local installation. Create a local pip package repository, work in isolated environments without internet access, and backup dependencies.
Smart Pip Collector — a utility for collecting all dependencies from requirements.txt files in a specified directory and downloading their archives for local installation.
Why Use It
- Create a local pip package repository
- Work in isolated environments without internet access
- Backup dependencies of all your projects
- Migrate projects between machines
- Protection against PyPI blocks
- Recursive search — finds all
requirements.txtin nested folders - Version awareness — downloads specific package versions, creating
package/version/structure - No duplicates — groups same packages with different versions
- Multithreading — uses all CPU cores for maximum speed
- Detailed reports — JSON reports for each download + global report
- Graceful shutdown — proper Ctrl+C handling (1 press — stop, 2 — force)
- Project tracking — stores information about which projects use which dependency
- Resume mode — retry failed packages
git clone https://github.com/smartlegionlab/smart-pip-collector.git cd smart-pip-collector
# Basic run python app.py /path/to/your/projects # With custom output directory python app.py /path/to/your/projects --output ~/my_pip_backup # With custom thread count python app.py /path/to/your/projects --workers 16 # Resume failed downloads python app.py /path/to/your/projects --resume
~/pip_packages/ ├── requests/ │ └── 2.28.1/ │ ├── requests-2.28.1-py3-none-any.whl │ └── .download_report.json ├── django/ │ └── 4.2.1/ │ ├── Django-4.2.1-py3-none-any.whl │ └── .download_report.json ├── global_download_report.json └── all_requirements.txt
Local report (in each package folder):
{
"package": "requests",
"version": "2.28.1",
"download_time": "2024-01-15T10:30:00",
"source_projects": ["project_a", "project_b"],
"status": "success",
"files": ["requests-2.28.1-py3-none-any.whl"]
}
Global report (global_download_report.json):
- Overall statistics
- Detailed information for each package
- Rate limiting and cancellation markers
| Parameter | Short | Description |
|---|---|---|
directory | Directory to search for requirements.txt | |
--output | -o | Output folder (default ~/pip_packages) |
--workers | -w | Number of threads (default CPU×2) |
--resume | -r | Resume mode for failed packages |
- Packages requiring compilation (with C extensions) may fail without system dependencies
- Non-standard links (git+, http) are ignored
- PyPI rate limiting is possible with very large number of requests
- Language: Python 3.8+
- Platform: Windows / Linux / macOS
- License: BSD 3-Clause
By using this software, you agree to the full disclaimer terms.
Software provided "AS IS" without warranty. You assume all risks.
Full legal disclaimer: See DISCLAIMER.md
License: BSD 3-Clause License