I'm looking for recent papers that develop small efficient datasets for training coding…
- User
I'm looking for recent papers that develop small efficient datasets for training coding models (LLMs) using data curator models
- Assistant
Recent research has focused on developing small, efficient datasets for training coding models using data curation techniques. Notable contributions include:
-
The MiniPile Challenge for Data-Efficient Language Models: This study introduces MiniPile, a 6GB subset of the 825GB Pile corpus, curated through a three-step data filtering process involving embedding inference, clustering, and low-quality cluster removal. Pre-training language models on MiniPile resulted in only a 1.9% to 2.5% performance drop on benchmarks compared to models trained on significantly larger datasets[1].
-
TinyHelen's First Curriculum: Training and Evaluating Tiny Language Models in a Simpler Language Environment: This research presents a pipeline for refining text data by eliminating noise, minimizing vocabulary, and maintaining genre-specific patterns. The resulting leaner datasets, such as Leaner-Pretrain and Leaner-Instruct, enhance learning efficiency, enabling tiny language models to outperform those trained on original datasets in instruction-following tasks[2].
-
DataDAM: Efficient Dataset Distillation with Attention Matching: This paper introduces DataDAM, a method that learns synthetic images by matching the spatial attention maps of real and synthetic data across different neural network layers. DataDAM outperforms prior methods on several datasets, including CIFAR10/100 and ImageNet-1K, achieving improvements of up to 6.5% and 4.1%, respectively[3].
These studies demonstrate the potential of curated, smaller datasets in training efficient coding models without significant performance degradation.
-
- User
I'm looking for recent papers that develop small efficient coding-focused datasets for training code generation models (LLMs) where the datasets are minimized by filtering using data curator models
- Assistant
Recent research has focused on developing small, efficient datasets for training code generation models by employing data curation techniques. Notable contributions include:
-
Every Sample Matters: Leveraging Mixture-of-Experts and High-Quality Data for Efficient and Accurate Code LLM: This study introduces Ling-Coder-Lite, a code language model that utilizes a Mixture-of-Experts (MoE) architecture combined with high-quality data curation methods, particularly those based on program analytics. By filtering and curating the training data, the model achieves competitive performance on 12 coding benchmarks while reducing deployment resources by 50% compared to similar-sized dense models[1].
-
SEED: Domain-Specific Data Curation With Large Language Models: SEED presents an approach that automatically generates domain-specific data curation solutions using Large Language Models (LLMs). The system compiles user-described tasks into hybrid pipelines that combine LLM querying with cost-effective alternatives, such as vector-based caching and small models trained on LLM-annotated data. This method effectively reduces the number of LLM calls while maintaining state-of-the-art performance across various data curation tasks[2].
-
OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs: This research introduces OpenCodeInstruct, a dataset comprising 5 million diverse samples, each including a programming question, solution, test cases, execution feedback, and LLM-generated quality assessments. The dataset is curated through a detailed methodology involving seed data curation, synthetic instruction and solution generation, and filtering. Fine-tuning various base models with OpenCodeInstruct has demonstrated substantial performance improvements on popular benchmarks[3].
-
Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning: This paper explores data pruning methods aimed at enhancing the efficiency of model training for code LLMs. By integrating clustering and pruning metrics to selectively reduce training data, the study finds that benchmark performance can be largely preserved by training on only 10% of the data. Additionally, moderate pruning strategies not only reduce computational resources but also enhance the overall quality of code generation[4].
These studies demonstrate the effectiveness of data curation and pruning techniques in creating smaller, efficient datasets for training code generation models without significant performance degradation.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.