> ## Documentation Index
> Fetch the complete documentation index at: https://futureagi.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Dataset Optimization
> Automatically improve prompt templates stored in your datasets using evaluation-driven optimization algorithms.
Dataset Optimization is the process of automatically improving **prompt templates** stored within your datasets. It uses the same optimization algorithms as [Prompt Optimization](/future-agi/get-started/optimization/overview) but applies them to dataset columns rather than agent system prompts.
This is useful when you want to systematically enhance the quality of prompts in your training or evaluation data using a structured, data-driven approach.
***
## **How It Differs from Prompt Optimization**
* Targets **agent system prompts**
* Uses **simulation** (conversational) evaluation
* Best for improving **agent behavior**
* Input: Agent configuration
* Targets **dataset column prompts**
* Uses **direct** (input/output) evaluation
* Best for improving **training & eval data**
* Input: Dataset column + evaluation templates
***
## **Key Concepts**
### **Optimization Run**
An optimization run connects the following components:
* ๐ **Column** โ The dataset column containing the prompt template to optimize.
* โ๏ธ **Optimizer Algorithm** โ The strategy used to find better prompts (e.g., Bayesian Search, ProTeGi).
* ๐ **Evaluation Templates** โ The evaluations used to score how well each prompt variation performs.
* ๐ง **Teacher Model** โ The LLM used for optimization decisions (generating new prompt candidates).
* ๐ **Inference Model** โ The LLM used to execute prompts and generate outputs during each trial.
### **Trials**
Each optimization run consists of multiple **trials**:
* ๐ **Baseline Trial** โ The original prompt is evaluated first to establish a baseline score.
* ๐ **Variation Trials** โ New prompt variations generated by the optimizer algorithm.
* Each trial receives an **average score** based on the configured evaluation templates.
### **Evaluation Templates**
Evaluation templates define how each prompt variation is scored across dataset rows. You can use:
* โ
**Built-in templates** โ Pre-configured evaluations like `summary_quality`, `context_adherence`, `tone`, and more.
* ๐ ๏ธ **Custom templates** โ Define your own evaluation criteria tailored to your specific use case.
You can browse and create evaluation templates from the [Evaluations](/future-agi/get-started/evaluation/running-your-first-eval) section of the platform.
***
## **How It Works**
Navigate to your dataset and click the **Optimize** button in the top action bar. Select the column containing the prompt template you want to optimize, choose an optimizer algorithm, configure the teacher and inference models, and select your evaluation templates.
The system evaluates your original prompt against the dataset to establish a baseline score. This score serves as the benchmark for measuring improvements.
The optimizer generates new prompt variations, runs them against dataset rows (up to 50) using the inference model, and scores them using your evaluation templates. Each variation is saved as a trial with its results.
Compare the baseline against all optimized variations. The system highlights the best-performing prompt with measurable score improvements. ๐
Optimizations can be **paused and resumed** โ the optimizer state is persisted after each trial, so you won't lose progress if a run is interrupted.
***
## **Supported Optimizers**
All six optimization algorithms are available for dataset optimization:
| Optimizer | Speed | Quality | Best For |
| ------------------- | ----- | ------- | ------------------------------- |
| **Random Search** | โกโกโก | โญโญ | Quick baselines |
| **Bayesian Search** | โกโก | โญโญโญโญ | Few-shot learning tasks |
| **Meta-Prompt** | โกโก | โญโญโญโญ | Complex reasoning tasks |
| **ProTeGi** | โก | โญโญโญโญ | Fixing specific error patterns |
| **PromptWizard** | โก | โญโญโญโญ | Creative/open-ended exploration |
| **GEPA** | โก | โญโญโญโญโญ | Production deployments |
Speed: โก = Slow, โกโก = Medium, โกโกโก = Fast
Quality: โญ = Basic, โญโญโญโญโญ = Excellent
Start with **Random Search** for a quick baseline, then use a more advanced optimizer like **ProTeGi** or **GEPA** to push for higher quality. ๐
For detailed information on each algorithm, see the [Optimization Algorithms](/future-agi/get-started/optimization/optimizers/overview) page.
***
## **When to Use Each Optimizer**
**Best for**: Getting a fast baseline to compare against.
Generates random prompt variations using a teacher model. No learning from previous attempts, but very fast.
```python theme={null}
"optimizer_algorithm": "random_search"
"optimizer_config": { "num_variations": 5 }
```
**Best for**: Tasks that benefit from few-shot examples in the prompt.
Uses Bayesian optimization to intelligently select the best combination and number of few-shot examples.
```python theme={null}
"optimizer_algorithm": "bayesian"
"optimizer_config": { "n_trials": 20, "min_examples": 2, "max_examples": 8 }
```
**Best for**: Complex tasks where the prompt needs holistic redesign.
Analyzes failed examples, formulates hypotheses about what went wrong, and rewrites the entire prompt.
```python theme={null}
"optimizer_algorithm": "metaprompt"
"optimizer_config": { "num_rounds": 5 }
```
**Best for**: When you can identify clear failure patterns in outputs.
Generates critiques of failures and applies targeted improvements using beam search.
```python theme={null}
"optimizer_algorithm": "protegi"
"optimizer_config": { "num_rounds": 3, "beam_size": 3, "num_gradients": 4 }
```
**Best for**: Open-ended tasks where you want diverse prompt variations.
Combines mutation with different "thinking styles", then critiques and refines top performers.
```python theme={null}
"optimizer_algorithm": "promptwizard"
"optimizer_config": { "refine_iterations": 2, "mutate_rounds": 3 }
```
**Best for**: Production deployments requiring the highest quality results.
Uses evolutionary algorithms with reflective learning and mutation strategies. State-of-the-art performance.
```python theme={null}
"optimizer_algorithm": "gepa"
"optimizer_config": { "max_metric_calls": 150 }
```
***
## **Best Practices**
### ๐ฆ Dataset Size
* **Optimal range**: 15โ50 rows for optimization.
* The system evaluates up to **50 rows** per trial for efficiency.
* Smaller datasets run faster but may produce less reliable scores.
### ๐ Evaluation Templates
* Use **1โ3 evaluation templates** that directly measure what matters for your task.
* Avoid conflicting evaluations that may confuse the optimizer.
* Use clear pass/fail criteria where possible.
### ๐งญ Choosing an Optimizer
```
Do you need production-grade results?
โโ Yes โ Use GEPA ๐งฌ
โโ No
โ
Do you have few-shot examples?
โโ Yes โ Use Bayesian Search ๐
โโ No
โ
Is your task reasoning-heavy?
โโ Yes โ Use Meta-Prompt ๐ง
โโ No
โ
Do you have clear failure patterns?
โโ Yes โ Use ProTeGi ๐ฌ
โโ No โ Use Random Search ๐ฒ (baseline)
```
***
## **Troubleshooting**
**Cause**: Dataset may be too small or not diverse enough.
**Solution**: Use more diverse examples (15โ50 rows recommended) and ensure your evaluation templates clearly distinguish good from bad outputs.
**Cause**: Inconsistent or conflicting evaluation templates.
**Solution**: Simplify your evaluations โ use 1โ2 clear templates instead of many overlapping ones.
**Cause**: Too many dataset rows or a slow optimizer.
**Solution**: Reduce dataset size, or switch to a faster optimizer like Random Search or Bayesian Search.
**Cause**: API errors or timeout.
**Solution**: Create a new run with the same configuration โ the system can resume from where it left off using persisted optimizer state.
***
## **Next Steps**
Compare all available optimizers and understand how each one works.
Run optimizations programmatically with advanced configurations.