From 1,000s of raw images to a training-ready dataset — pixel-curator (similarity training curation tool)

pixel-curator — curation tooling for similarity training (identity, subject, and style models) Repo: https://github.com/snoozey-mf-bnui/pixel-curator What it is pixel-curator is a local-first CLI that takes a raw folder of images (or video) and produces a curated, scored, captioned dataset ready to export directly into DreamBooth notebook training or in-house Kohya LoRA training. It's built primarily for similarity training — the case where you have hundreds or thousands of images of one specific subject and need a model that reliably reproduces it. Identity, character, person, product, or a specific style. That's the workflow it's optimized around, though the output is generic enough for other training setups. The problem it addresses If you've collected 1,000+ images of a subject, you already know the pipeline breaks down long before training starts: A large fraction are near-duplicates — burst shots, adjacent video frames, minor crops Some are technically unusable — low resolution, soft focus, compression artifacts The distribution is usually skewed — hundreds of shots from one angle, almost none from another Captioning is a second full project on its own Handing a folder of 1,792 images to a Colab notebook is a fast route to disconnects, VRAM exhaustion, and daily-limit lockouts Realistically, a well-curated 200–500 image subset outperforms a messy 2,000 on identity training. Most of the work is figuring out which subset that is. Pipeline 1. Ingest and prefilter Drops technically unusable frames — resolution failures, decode errors, corrupted files — before anything expensive runs. 2. Deduplication Perceptual hash dedup collapses near-identical images. On datasets sourced from video, bursts, or scrapes, this stage alone typically removes a large majority of the raw set. 3. Similarity clustering CLIP embeddings plus k-means, with embedding map visualization. This surfaces the actual distribution of your dataset — angle, lighting, composition, framing — so imbalance is visible before training rather than diagnosed after a failed run. 4. Quality scoring and classification Optical quality scoring assigns per-image status, including face-aware quality where relevant. Low-quality images are filtered rather than left to dilute the training signal. 5. Semantic filtering CLIP text filtering, e.g. automatically removing images with watermarks or text overlays without manual review. 6. Tagging and captioning WD14 tagger (auto-downloads) and Florence-2 natural-language captioning, written as .txt sidecars alongside each image. 7. Human review round-trip Exports a static HTML report and a spreadsheet. You mark keep/reject, import back, and the decisions are applied. This is also the workflow for trimming class/regularization images — hand-trimming generated class images is a well-known fix for weak identity results, and this makes it practical at scale. 8. Export package kohya --mode dreambooth produces the numbered instance/ structure that standard Colab DreamBooth notebooks expect: text instance/ 000001.jpg 000001.txt 000002.jpg 000002.txt dataset_config.toml package kohya --mode lora supports multi-concept LoRA packaging with concept_/ folders for in-house Kohya setups. Practical characteristics Fully local — no cloud, no accounts, no uploads CLI, so it scripts, batches, and runs over SSH Dry-run by default on any operation that mutates files Resumable via SQLite manifest — long stages can be interrupted and restarted without redoing work Designed to run on modest hardware Note on the earlier post I posted this here previously and the GitHub link pointed at the wrong directory — the repository has been corrected and the link above is the right one. Apologies to anyone who clicked through and found nothing useful. If you tried it before and it didn't work, it should now. Feedback Actively developed. If you run it against a dataset and something breaks or behaves unexpectedly, open an issue or leave a comment — the failure cases are the most useful thing I can get right now. https://github.com/snoozey-mf-bnui/pixel-curator If anyone needs help with creating a notebook for google colab dreambooth training, also lmk i will assist you through dms. ENJOY! submitted by /u/Impossible-Car-2809 [link] [comments]

Source: r/StableDiffusion | 2026-10-13