ProTeGi Optimizer

ProTeGi (Prompt optimization with Textual Gradients) systematically improves prompts by identifying failure patterns, generating targeted critiques, and applying specific fixes. It uses beam search to maintain multiple candidate prompts and progressively refines them.

When to Use ProTeGi

✅ Best For

Debugging specific failure modes
Systematic error correction
Tasks with clear failure patterns
Iterative refinement workflows

❌ Not Ideal For

Quick experiments (multi-stage process)
Tasks where failures are random
Very small datasets
Budget-constrained projects

How It Works

ProTeGi follows a structured expansion and selection process:

Identify Failures

Run current prompts and identify examples with low scores

Generate Critiques

Teacher model analyzes failures and generates multiple specific critiques (“gradients”)

Apply Improvements

For each critique, generate improved prompt variations

Beam Selection

Evaluate all candidates and keep top N prompts

Iterate

Repeat expansion from the best performing prompts

ProTeGi maintains a “beam” of candidate prompts throughout optimization, preventing premature convergence to local optima.

Basic Usage

from fi.opt.optimizers import ProTeGi
from fi.opt.generators import LiteLLMGenerator
from fi.opt.datamappers import BasicDataMapper
from fi.opt.base.evaluator import Evaluator

# Setup teacher model
teacher = LiteLLMGenerator(
    model="gpt-4o",
    prompt_template="{prompt}"
)

# Setup evaluator
evaluator = Evaluator(
    eval_template="context_relevance",
    eval_model_name="turing_flash",
    fi_api_key="your_key",
    fi_secret_key="your_secret"
)

# Setup data mapper
data_mapper = BasicDataMapper(
    key_map={"input": "question", "output": "generated_output"}
)

# Create optimizer
optimizer = ProTeGi(
    teacher_generator=teacher,
    num_gradients=4,
    errors_per_gradient=4,
    prompts_per_gradient=1,
    beam_size=4
)

# Run optimization
result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=dataset,
    initial_prompts=["Answer the question: {question}"],
    num_rounds=3,
    eval_subset_size=32
)

Underlying Research

ProTeGi introduces a novel, gradient-inspired approach to prompt optimization, adapting concepts from numerical optimization to natural language.

Core Paper: The method originates from the paper “Automatic Prompt Optimization with “Gradient Descent” and Beam Search”, which details how to create “textual gradients” (critiques) to guide prompt improvement.
Extensions: The core idea has been extended in subsequent research, such as “Momentum-Aided Gradient Descent Prompt Optimization”, which incorporates momentum to accelerate convergence.
Classification: In surveys on automatic prompt engineering, ProTeGi is categorized as a pioneering gradient-based method for its innovative approach to error-driven refinement.

Configuration Parameters

Core Parameters

LiteLLMGenerator

required

Powerful model for generating critiques and improved prompts. Recommended: gpt-4o, claude-3-opus.

int

default:"4"

Number of distinct critiques to generate for each prompt. More gradients = more diverse improvement directions.

int

default:"4"

Number of failed examples shown to teacher when generating each critique. Higher = more context but more expensive.

int

default:"1"

Number of new prompts to generate from each critique. Set to 2-3 for more exploration.

int

default:"4"

Number of top-performing prompts to keep each round. Larger beam = more diversity but slower.

Get Started

Guides

When to Use ProTeGi

✅ Best For

❌ Not Ideal For

How It Works

Basic Usage

Underlying Research

Configuration Parameters

Core Parameters

**Optimization Parameters

​When to Use ProTeGi

✅ Best For

❌ Not Ideal For

​How It Works

​Basic Usage

​Underlying Research

​Configuration Parameters

​Core Parameters

​**Optimization Parameters

When to Use ProTeGi

How It Works

Basic Usage

Underlying Research

Configuration Parameters

Core Parameters

**Optimization Parameters