EleutherAI Releases Common Pile v0.1, an 8TB Dataset for AI Training

EleutherAI has announced the release of the Common Pile v0.1, an 8TB dataset of openly licensed and public domain text, designed for training large language models. This dataset, curated in collaboration with several academic and research institutions, aims to provide a comprehensive resource for AI research and development detailed in a company blog post.

The Common Pile v0.1 includes content from 30 diverse sources, such as research papers, code, books, and educational materials. It serves as a successor to the Pile, EleutherAI's previous dataset, and is intended to address the need for large-scale, openly licensed data in AI training. Alongside the dataset, EleutherAI has released two models, Comma v0.1-1T and Comma v0.1-2T, trained on 1 trillion and 2 trillion tokens respectively, demonstrating competitive performance with models trained on unlicensed data.

This release underscores EleutherAI's commitment to open science and transparency in AI research, providing tools and resources that enable researchers to build and evaluate AI models under consistent data conditions. The dataset and associated models are available on platforms like arXiv, Hugging Face, and GitHub, facilitating widespread access and collaboration in the AI community.