> ## Documentation Index
> Fetch the complete documentation index at: https://futureagi.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Meta-Prompt Optimizer

> A guide to the Meta-Prompt optimizer, which uses a teacher LLM for deep reasoning-based prompt refinement through systematic failure analysis and rewriting.

Meta-Prompt uses a powerful teacher LLM to analyze how your prompt performs, understand why it fails on specific examples, formulate hypotheses about improvements, and completely rewrite the prompt. This approach is inspired by the `promptim` library and excels at tasks requiring deep reasoning.

***

## **When to Use Meta-Prompt**

<CardGroup cols={2}>
  <Card title="✅ Best For" icon="check">
    * Complex reasoning tasks
    * Tasks where understanding failures helps
    * Refining well-scoped prompts
    * Deep iterative improvement
  </Card>

  <Card title="❌ Not Ideal For" icon="xmark">
    * Quick experiments (slower)
    * Simple classification tasks
    * Very large datasets (costly)
    * Tasks with unclear failure patterns
  </Card>
</CardGroup>

***

## **How It Works**

Meta-Prompt follows a systematic analysis-and-rewrite cycle:

<Steps>
  <Step title="Evaluate Current Prompt">
    Run the current prompt on a subset of your dataset and collect scores
  </Step>

  <Step title="Identify Failures">
    Focus on examples with low scores to understand what went wrong
  </Step>

  <Step title="Formulate Hypothesis">
    Teacher model analyzes failures and proposes a specific improvement theory
  </Step>

  <Step title="Rewrite Prompt">
    Generate a complete new prompt implementing the hypothesis
  </Step>

  <Step title="Repeat">
    Continue for multiple rounds, building on previous insights
  </Step>
</Steps>

<Info>
  Unlike optimizers that tweak parts of a prompt, Meta-Prompt rewrites the **entire** prompt each iteration based on deep analysis.
</Info>

***

## **Basic Usage**

```python theme={null}
from fi.opt.optimizers import MetaPromptOptimizer
from fi.opt.generators import LiteLLMGenerator
from fi.opt.datamappers import BasicDataMapper
from fi.opt.base.evaluator import Evaluator

# Setup teacher model (use a powerful model for analysis)
teacher = LiteLLMGenerator(
    model="gpt-4o",
    prompt_template="{prompt}"
)

# Setup evaluator
evaluator = Evaluator(
    eval_template="summary_quality",
    eval_model_name="turing_flash",
    fi_api_key="your_key",
    fi_secret_key="your_secret"
)

# Setup data mapper
data_mapper = BasicDataMapper(
    key_map={"input": "text", "output": "generated_output"}
)

# Create optimizer
optimizer = MetaPromptOptimizer(
    teacher_generator=teacher
)

# Run optimization
result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=dataset,
    initial_prompts=["Summarize this text: {text}"],
    task_description="Create concise, informative summaries",
    num_rounds=5,
    eval_subset_size=40
)

print(f"Improvement: {result.final_score:.2%}")
print(f"Best prompt:\n{result.best_generator.get_prompt_template()}")
```

***

## **Configuration Parameters**

### **Core Parameters**

<ParamField path="teacher_generator" type="LiteLLMGenerator" required>
  A powerful language model used for analyzing failures and generating improved prompts. Recommended: `gpt-4o`, `gpt-4-turbo`, or `claude-3-opus`.

  ```python theme={null}
  teacher = LiteLLMGenerator("gpt-4o", "{prompt}")
  ```
</ParamField>

<ParamField path="task_description" type="str" default="I want to improve my prompt.">
  Description of what you want the optimized prompt to achieve. More specific descriptions lead to better results.

  ```python theme={null}
  task_description="Generate summaries that capture key points while being under 50 words"
  ```
</ParamField>

<ParamField path="num_rounds" type="int" default="5">
  Number of analysis-and-rewrite iterations. More rounds can lead to better results but cost more.
</ParamField>

<ParamField path="eval_subset_size" type="int" default="40">
  Number of examples to evaluate each round. Smaller = faster but less reliable signal.
</ParamField>

***

## **The Meta-Prompt Process**

### **What the Teacher Model Sees**

In each round, the teacher model receives:

1. **Current Prompt** - The prompt being evaluated
2. **Previous Failed Attempts** - Prompts that performed worse (to avoid repeating mistakes)
3. **Performance Data** - Detailed results showing which examples failed and why
4. **Task Description** - Your goal for the optimization

### **What the Teacher Model Returns**

The teacher provides two things:

```json theme={null}
{
  "hypothesis": "The prompt fails on complex multi-sentence texts because it doesn't specify a structure. Adding explicit instruction to identify main points first should improve clarity.",
  "improved_prompt": "First identify the 2-3 main points in the following text. Then write a single concise sentence that captures these points:\n\n{text}"
}
```

***

## **Underlying Research**

The Meta-Prompt optimizer is inspired by meta-learning and reflective AI systems, where a model improves its own processes.

* **Meta-Learning**: The core idea is formalized in research like "[System Prompt Optimization with Meta-Learning](https://arxiv.org/abs/2505.09666)", which uses bilevel optimization. Another related work is "[metaTextGrad](https://arxiv.org/abs/2505.18524)", which optimizes both prompts and their surrounding structures.
* **Industry Tools**: This reflective approach is used in tools like Google's Vertex AI Prompt Optimizer and is a key feature in advanced models for self-improvement.
* **Frameworks**: The concept is explored in libraries like `promptim` and is classified in surveys as a leading LLM-driven optimization method.

***

## **Advanced Examples**

### **With Detailed Task Description**

```python theme={null}
optimizer = MetaPromptOptimizer(
    teacher_generator=LiteLLMGenerator("gpt-4o", "{prompt}")
)

result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=dataset,
    initial_prompts=[initial_prompt],
    
    # Provide detailed context
    task_description="""
    I want to extract structured information from customer support tickets.
    The prompt should:
    - Identify the main issue
    - Extract customer sentiment (positive/negative/neutral)
    - Determine urgency level (low/medium/high)
    - Suggest appropriate department routing
    
    The output must be in JSON format and handle incomplete information gracefully.
    """,
    
    num_rounds=7,
    eval_subset_size=30
)
```

***

### **With More Rounds for Complex Tasks**

```python theme={null}
# For very complex tasks, use more rounds
optimizer = MetaPromptOptimizer(
    teacher_generator=LiteLLMGenerator("gpt-4o", "{prompt}")
)

result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=complex_dataset,
    initial_prompts=[initial_prompt],
    task_description=detailed_description,
    num_rounds=10,  # More iterations for complex refinement
    eval_subset_size=50  # More examples for reliable signal
)

# Analyze the evolution
for i, iteration in enumerate(result.history):
    print(f"\nRound {i+1} Score: {iteration.average_score:.4f}")
    print(f"Prompt: {iteration.prompt[:150]}...")
```

***

### **Combining with Other Optimizers**

Use Meta-Prompt for deep refinement after initial exploration:

```python theme={null}
# Stage 1: Quick exploration
random_result = random_search_optimizer.optimize(...)

# Stage 2: Deep refinement on best candidate
meta_optimizer = MetaPromptOptimizer(
    teacher_generator=LiteLLMGenerator("gpt-4o", "{prompt}")
)

final_result = meta_optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=dataset,
    initial_prompts=[random_result.best_generator.get_prompt_template()],
    task_description="Refine for clarity and consistency",
    num_rounds=5
)
```

***

## **Understanding the Results**

### **Tracking Hypothesis Evolution**

Meta-Prompt's hypotheses show its reasoning process:

```python theme={null}
result = optimizer.optimize(...)

# View the optimization journey
for i, iteration in enumerate(result.history):
    print(f"\n{'='*60}")
    print(f"Round {i+1}")
    print(f"Score: {iteration.average_score:.4f}")
    print(f"\nPrompt:\n{iteration.prompt}")
    
    # Note: Hypothesis is internal to teacher model, 
    # but you can infer it from prompt evolution
```

### **Analyzing Improvement Patterns**

```python theme={null}
scores = [iteration.average_score for iteration in result.history]

import matplotlib.pyplot as plt
plt.plot(scores, marker='o')
plt.xlabel('Round')
plt.ylabel('Score')
plt.title('Meta-Prompt Optimization Progress')
plt.show()

# Calculate improvement
initial_score = scores[0]
final_score = scores[-1]
improvement = ((final_score - initial_score) / initial_score) * 100
print(f"Total improvement: {improvement:.1f}%")
```

***

## **Performance Tips**

<AccordionGroup>
  <Accordion title="Use a powerful teacher model" icon="brain">
    Meta-Prompt's quality depends heavily on the teacher model's reasoning ability. Use `gpt-4o`, `claude-3-opus`, or similar high-end models.
  </Accordion>

  <Accordion title="Provide detailed task descriptions" icon="file-lines">
    Specific task descriptions help the teacher make targeted improvements. Include constraints, desired output format, and edge cases to handle.
  </Accordion>

  <Accordion title="Start with 5 rounds" icon="rotate">
    5 rounds is usually enough for meaningful improvement. Increase to 7-10 only for very complex tasks where you see continued progress.
  </Accordion>

  <Accordion title="Balance eval subset size" icon="scale-balanced">
    * Too small (\< 20): Unreliable signal, may optimize for noise
    * Too large (> 50): Slow and expensive
    * Sweet spot: 30-40 examples
  </Accordion>

  <Accordion title="Analyze failed examples" icon="magnifying-glass">
    Look at low-scoring examples in each round to understand what the optimizer is trying to fix:

    ```python theme={null}
    for iteration in result.history:
        failures = [r for r in iteration.individual_results if r.score < 0.5]
        print(f"Round failures: {len(failures)}")
        for f in failures[:3]:  # Show first 3
            print(f"  - Score: {f.score:.2f}, Reason: {f.reason}")
    ```
  </Accordion>
</AccordionGroup>

***

## **Common Patterns**

### **Complex Reasoning Tasks**

```python theme={null}
dataset = [
    {
        "problem": "Multi-step math word problem...",
        "solution": "Step-by-step solution..."
    }
]

optimizer = MetaPromptOptimizer(
    teacher_generator=LiteLLMGenerator("gpt-4o", "{prompt}")
)

result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=BasicDataMapper({
        "input": "problem",
        "output": "generated_output"
    }),
    dataset=dataset,
    initial_prompts=["Solve this problem: {problem}"],
    task_description="""
    Generate step-by-step solutions that:
    - Show clear reasoning at each step
    - Explain why each step is necessary
    - Arrive at the correct final answer
    """,
    num_rounds=8
)
```

### **Creative Writing with Constraints**

```python theme={null}
optimizer = MetaPromptOptimizer(
    teacher_generator=LiteLLMGenerator("gpt-4o", "{prompt}")
)

result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=creative_dataset,
    initial_prompts=["Write a story based on: {prompt}"],
    task_description="""
    Generate engaging short stories (200-300 words) that:
    - Have a clear beginning, middle, and end
    - Include vivid sensory details
    - Match the tone specified in the prompt
    - Are appropriate for a general audience
    """,
    num_rounds=6,
    eval_subset_size=25
)
```

### **Data Transformation Tasks**

```python theme={null}
optimizer = MetaPromptOptimizer(
    teacher_generator=LiteLLMGenerator("gpt-4o", "{prompt}")
)

result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=transformation_dataset,
    initial_prompts=["Convert this data: {input_data}"],
    task_description="""
    Transform unstructured text into JSON format with these fields:
    - name (string)
    - date (YYYY-MM-DD format)
    - amount (number)
    - category (one of: personal, business, travel)
    
    Handle missing fields by using null. Infer dates from context when possible.
    """,
    num_rounds=5
)
```

***

## **Troubleshooting**

<AccordionGroup>
  <Accordion title="Scores plateau after few rounds" icon="chart-line">
    **Problem**: Improvement stops after 2-3 rounds

    **Solution**:

    * Your initial prompt might already be good - check if score is already high
    * Make task description more specific to guide further refinement
    * Try a different teacher model for fresh perspective
    * Increase `eval_subset_size` for more reliable signal
  </Accordion>

  <Accordion title="Prompts become too verbose" icon="text-size">
    **Problem**: Each iteration adds more instructions, making prompts unwieldy

    **Solution**:

    * Add to task description: "Keep the prompt concise and under 200 words"
    * Manually select a mid-optimization prompt that balances quality and length
    * Use fewer rounds (3-4 instead of 7-8)
  </Accordion>

  <Accordion title="High API costs" icon="dollar-sign">
    **Problem**: Optimization is expensive with GPT-4

    **Solution**:

    * Reduce `num_rounds` to 3-5
    * Decrease `eval_subset_size` to 20-30
    * Use `gpt-4o-mini` as teacher for initial experiments
    * Run on a smaller dataset subset first to validate approach
  </Accordion>

  <Accordion title="Inconsistent improvements" icon="arrow-trend-up">
    **Problem**: Score goes up and down between rounds

    **Solution**:

    * Increase `eval_subset_size` for more stable measurements
    * Check if your evaluation metric is too noisy
    * Ensure dataset examples are high-quality and representative
    * Consider using a different evaluation metric
  </Accordion>
</AccordionGroup>

***

## **Comparison with Other Optimizers**

| Aspect                 | Meta-Prompt        | Bayesian Search    | ProTeGi              |
| ---------------------- | ------------------ | ------------------ | -------------------- |
| **Approach**           | Analysis & rewrite | Few-shot selection | Error-driven fixing  |
| **Best for**           | Complex reasoning  | Structured tasks   | Systematic debugging |
| **Speed**              | Medium             | Fast               | Slow                 |
| **Prompt changes**     | Complete rewrites  | Example selection  | Targeted edits       |
| **Teacher dependency** | High               | Medium             | High                 |

***

## **Next Steps**

<CardGroup cols={2}>
  <Card title="Try ProTeGi" icon="microscope" href="/future-agi/get-started/optimization/optimizers/protegi">
    For more systematic error analysis
  </Card>

  <Card title="Compare All Optimizers" icon="scale-balanced" href="/future-agi/get-started/optimization/optimizers/overview">
    See which optimizer fits your needs
  </Card>
</CardGroup>