数据净化工匠
原名：nemo-curator
Curate LLM training data: dedupe, filter, PII redaction.
- 分类
- 数据分析
- 版本
- v1.0.1
- 作者
- 弈韬(@ra1nzzz)
- 下载
- 1
- 收藏
- 0
- 发布
- 2026-08-25
- 更新
- 2026-09-13
- TRACE 评分
- 3.6 / 5
内容概览
NVIDIA's toolkit for preparing high-quality training data for LLMs. Use NeMo Curator when: - Preparing LLM training data from web scrapes (Common Crawl) - Need fast deduplication (16× faster than CPU) - Curating multi-modal datasets (text, images, video, audio) - Filtering low-quality or toxic content - Scaling data processing across GPU cluster Performance : - 16× faster fuzzy deduplication (8TB RedPajama v2) - 40% lower TCO vs CPU alternatives - Near-linear scaling across GPU nodes Use alternatives instead : - datatrove : CPU-based, open-source data processing - dolma : Allen AI's data toolkit - Ray Data : General ML data processing (no curation focus) Major version rewrite (1.x): NeMo Curator was rewritten around a Ray-based pipeline/stage architecture . The old DocumentDataset + nemo curator.modules. / ScoreFilter / Modify call-the-object-on-a-dataset API from 0.x is gone. In 1.x you…