> ## Documentation Index
> Fetch the complete documentation index at: https://futureagi.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# End-to-End Prompt Optimization

***

## 1. Introduction

Prompt optimization appears simple, just adjust instructions until outputs improve, but in production, this approach consistently fails.

The first failure is the lack of evaluation baselines. Most teams do not have a stable, quantitative way to determine whether a prompt change is an improvement or a regression. Outputs are inspected manually, sampled inconsistently, and judged subjectively. Once behavior degrades, there is no reference point to diagnose why.

The second failure is reproducibility. Prompt changes are rarely versioned, benchmarked, or evaluated in a controlled manner. Improvements cannot be reliably reproduced across environments, team members, or time. As a result, prompt behavior becomes fragile and difficult to defend.

The third failure is iteration cost. Prompt refinement is performed through manual loops: edit, test a few examples, repeat. This process does not scale with dataset size, task complexity, or organizational velocity. As systems grow, iteration slows and confidence erodes.

The final failure is brittleness. Over time, prompts accumulate ad-hoc fixes for edge cases. Each fix introduces new interactions, making the prompt increasingly unstable. Small changes cause unexpected regressions, and prompt engineering devolves into reactive patching.

Prompt engineering relies on human intuition and local testing. This is sufficient for prototypes. It breaks down when prompts must satisfy diverse inputs, strict correctness requirements, and cost constraints simultaneously. At that point, prompt behavior must be managed as a system, not as text.

***

## 2. Prompt Optimization as a First-Class Workflow

Even when prompt optimisers are available, using them requires stitching together evaluation logic, tracking prompt versions, comparing runs, and managing iteration manually. These steps are rarely standardised and are often handled through scripts, notebooks, or human judgement As a result, optimization is slow, inconsistent, and difficult to repeat. Teams either stop optimizing or limit it to one-off experiments.

Future AGI removes this operational burden by making prompt optimization a built-in workflow rather than a custom system. Outputs are scored consistently, prompt versions and results are stored and comparable, optimization loops are handled by the platform and improved prompts are ranked and returned automatically. This allows teams to focus on defining behavior and success criteria, instead of building and maintaining optimization infrastructure.

Using Future AGI, prompt optimization is reduced to a small set of decisions:

* what behavior to evaluate (by creating dataset)
* how success is measured (by defining evaluator)
* how improvement is explored (by choosing optimiser)

Once these are defined, optimization runs as a single execution step. Prompt optimization stops being a research problem and becomes an execution problem.

***

## **3. Prompt Optimization using Future AGI**

This section defines **all required components** to run prompt optimization using Future AGI. Each step introduces one concrete object, explains its role briefly, and shows the exact code required.

<Steps>
  <Step title="Install Required Packages">
    Install the `agent-opt` package to get started with prompt optimization.

    ```bash theme={null}
    pip install agent-opt
    ```
  </Step>

  <Step title="Set Environment Variables">
    These credentials are required to run evaluations and track optimization results in Future AGI. Click [here](https://app.futureagi.com/dashboard/keys) to get your API keys.

    ```python theme={null}
    import os

    os.environ["FI_API_KEY"] ="YOUR_API_KEY"
    os.environ["FI_SECRET_KEY"] ="YOUR_SECRET_KEY"
    ```
  </Step>

  <Step title="Prepare the Dataset">
    The dataset defines the inputs against which prompt performance will be evaluated.

    ```python theme={null}
    dataset = [
        {
    "article":"The James Webb Space Telescope captured detailed images of the Pillars of Creation.",
    "target_summary":"JWST captured new detailed images of the Pillars of Creation."
        },
        {
    "article":"Researchers discovered an enzyme that rapidly breaks down plastic.",
    "target_summary":"A newly discovered enzyme rapidly breaks down plastic."
        }
    ]
    ```
  </Step>

  <Step title="Define the Prompt and Generator">
    Provide the initial prompt that will be optimized. The generator binds the prompt to a model configuration.

    ```python theme={null}
    from fi.opt.generatorsimport LiteLLMGenerator

    prompt_template ="Summarize this: {article}"

    generator = LiteLLMGenerator(
        model="gpt-4o-mini",
        prompt_template=prompt_template
    )
    ```
  </Step>

  <Step title="Configure the Evaluator">
    The evaluator defines how output quality is measured. It acts as the objective function for optimization.

    ```python theme={null}
    from fi.opt.base.evaluatorimport Evaluator

    evaluator = Evaluator(
        eval_template="summary_quality",
        eval_model_name="turing_flash"
    )
    ```

    <Note>
      We are using one of Future AGI's builtin eval called `summary_quality`. Click [here](https://docs.futureagi.com/future-agi/get-started/evaluation/builtin-evals/overview) to learn what other builtin evals Future AGI offers.
    </Note>

    <Tip>
      For maximum flexibility, you can define your own evaluation logic using a local LLM-as-a-judge. This is ideal for custom tasks or when you need a very specific evaluation rubric. Click [here](https://www.notion.so/2d51cecdacb78097b438c23f90e8f66a?pvs=21) to learn more.
    </Tip>
  </Step>

  <Step title="Configure the DataMapper">
    The DataMapper connects dataset fields to evaluator inputs.

    ```python theme={null}
    from fi.opt.datamappersimport BasicDataMapper

    data_mapper = BasicDataMapper(
        key_map={
    "input":"article",
    "output":"generated_output"
        }
    )
    ```
  </Step>

  <Step title="Select the Optimizer">
    The optimizer defines how prompt variants are generated and evaluated.

    Future AGI supports multiple prompt optimization strategies, all accessible through the same workflow. A full, up-to-date overview of supported optimizers is available in the documentation. Click [here](https://www.notion.so/Cookbook-Prompt-Optimization-2bd1cecdacb780fa9d41da7c0c7d607a?pvs=21) to learn more.

    At a high level, commonly used optimizers include:

    * **Random Search** – fast baseline exploration
    * **Bayesian Search** – structured optimization for few-shot prompts
    * **ProTeGi** – targeted refinement for recurring failure patterns
    * **Meta-Prompt** – higher-level prompt rewrites
    * **GEPA** – evolutionary optimization for production-grade quality

    <Info>
      Switching optimizers does **not** change the workflow.
    </Info>

    ```python theme={null}
    from fi.opt.optimizersimport RandomSearchOptimizer

    optimizer = RandomSearchOptimizer(
        generator=generator,
        teacher_model="gpt-4o",
        num_variations=5
    )
    ```
  </Step>

  <Step title="Run Prompt Optimization">
    Execute the optimization process with all configured components.

    ```python theme={null}
    result = optimizer.optimize(
        evaluator=evaluator,
        data_mapper=data_mapper,
        dataset=dataset
    )
    ```

    <Check>
      Once these steps are complete, Future AGI automatically handles:

      * Evaluation execution
      * Optimization loops
      * Experiment tracking
      * Prompt versioning
      * Result comparison and ranking
    </Check>
  </Step>
</Steps>

***

## Conclusion

Prompt optimization becomes difficult when it is treated as an informal, intuition-driven activity. It becomes manageable when prompts are evaluated systematically, and improved through explicit feedback loops.

Future AGI removes the operational complexity by incorporating evaluation, iteration, comparison, and bookkeeping. What remains is a small set of explicit inputs and a single execution step.

As a result, prompt optimization shifts from a research exercise to a routine engineering workflow, which is repeatable, auditable, and easy to operate at scale.

***

## FAQ

**1. Do I need to write custom evaluation logic?**

No. Future AGI provides 60+ [built-in](http://docs.futureagi.com/future-agi/get-started/evaluation/builtin-evals/overview) evaluators and supports LLM-as-a-Judge patterns out of the box. Evaluation execution, scoring, and aggregation are handled by the platform.

**2. Does switching optimizers require changing my workflow?**

No. The workflow remains the same. Switching optimizers changes a single configuration line; the dataset, evaluator, data mapping, and execution flow do not change.

**3. Can we save this optimized prompt as a prompt templates in Future AGI platform?**

Yes, by using prompt SDK, the output can be stored as a new template version and managed like any other prompt artifact. Click [here](https://docs.futureagi.com/products/prompt/how-to/prompt-workbench-using-sdk) to learn more.

***

## **Ready to Systematically Optimize Prompt?**

Start incorporating prompt optimization in your production AI systems using Future AGI. Future AGI provides the evaluation and optimization infrastructure required to build reliable, explainable, and production-ready LLM applications. Click [here](https://futureagi.com/contact-us) to schedule a demo with us now!

***