Random Search Optimizer

Random Search is a gradient-free method that generates a set of random variations of an initial prompt using a powerful “teacher” LLM. It then evaluates each variation against a dataset and selects the best-performing one. It’s a fast, straightforward, and often surprisingly effective way to explore different prompt phrasings and establish a strong performance baseline.

When to Use Random Search

✅ Best For

Establishing a quick baseline
Simple tasks like summarization or classification
Broad, unbiased exploration of the prompt space
Projects with a low computational budget

❌ Not Ideal For

Complex, nuanced, or multi-step reasoning tasks
Directed, efficient optimization when failure modes are known
Tasks requiring highly structured or constrained prompts
Finding the absolute, state-of-the-art best prompt

How It Works

The Random Search process is simple and effective, involving three main steps:

1. Generate Variations

You provide an initial prompt. The optimizer then uses a powerful teacher_model (like GPT-4o) to generate a specified num_variations of diverse rewrites of that prompt.

2. Evaluate All Variations

The optimizer iterates through each generated variation. For each one, it generates outputs for all examples in your dataset and scores them using the provided evaluator.

3. Select the Best

The variation that achieves the highest average score across the entire dataset is chosen as the best prompt. The process concludes, and this top-performing prompt is returned.

Basic Usage

from fi.opt.optimizers import RandomSearchOptimizer
from fi.opt.generators import LiteLLMGenerator
from fi.opt.datamappers import BasicDataMapper
from fi.opt.base.evaluator import Evaluator

# 1. Define the generator with the initial prompt to be optimized
initial_generator = LiteLLMGenerator(
    model="gpt-4o-mini",
    prompt_template="Summarize this article: {article}"
)

# 2. Setup the evaluator to score prompt performance
evaluator = Evaluator(
    eval_template="summary_quality",
    eval_model_name="turing_flash",
    fi_api_key="your_key",
    fi_secret_key="your_secret"
)

# 3. Setup the data mapper
data_mapper = BasicDataMapper(
    key_map={"input": "article", "output": "generated_output"}
)

# 4. Initialize the Random Search optimizer
# It needs the generator to optimize, a powerful teacher model, and the number of variations to try.
optimizer = RandomSearchOptimizer(
    generator=initial_generator,
    teacher_model="gpt-4o",
    num_variations=10
)

# 5. Run the optimization
result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=my_dataset
)

print(f"Best prompt found: {result.best_generator.get_prompt_template()}")
print(f"Final score: {result.final_score:.4f}")

Configuration Parameters

BaseGenerator

required

The generator instance that you want to optimize. The optimizer will modify the prompt template within this object.

str

default:"gpt-5"

The powerful language model used to generate the prompt variations. The quality of the random search depends heavily on this model’s ability to create diverse and sensible rewrites. Recommended: gpt-4o, claude-3-opus.

int

default:"5"

The number of different prompt variations the teacher model will generate. This parameter controls the trade-off between the breadth of the search and the computational cost/time of the optimization.

Dict

default:"{}"

A dictionary of additional arguments to pass to the teacher model during variation generation. This is useful for controlling parameters like temperature to influence the creativity of the variations.

# Example: Increase temperature for more creative variations
optimizer = RandomSearchOptimizer(
    ...,
    teacher_model_kwargs={"temperature": 1.2}
)

Underlying Research

Random search is a foundational technique in hyperparameter tuning, valued for its simplicity and surprising effectiveness, often outperforming more structured methods like grid search.

Baseline Strength: Research like “Random Sampling as a Strong Baseline for Prompt Optimisation” demonstrates that even simple random sampling can be a highly competitive method for improving prompts.
Broad Applicability: It is frequently used as the first step in prompt optimization toolkits to get a sense of the landscape. Its ability to avoid getting stuck in local optima makes it a valuable tool in the discrete and high-dimensional space of prompt engineering.

Get Started

Guides

When to Use Random Search

✅ Best For

❌ Not Ideal For

How It Works

Basic Usage

Configuration Parameters

Underlying Research

Next Steps

Try Bayesian Search

Compare All Optimizers

​When to Use Random Search

✅ Best For

❌ Not Ideal For

​How It Works

​Basic Usage

​Configuration Parameters

​Underlying Research

​Next Steps

Try Bayesian Search

Compare All Optimizers

When to Use Random Search

How It Works

Basic Usage

Configuration Parameters

Underlying Research

Next Steps