Synthetic Image Evaluator

result = evaluator.evaluate(
    eval_templates="synthetic_image_evaluator",
    inputs={
        "image": "https://www.esparklearning.com/app/uploads/2024/04/Albert-Einstein-generated-by-AI-1024x683.webp"
    },
    model_name="turing_flash"
)

print(result.eval_results[0].output)
print(result.eval_results[0].reason)

import { Evaluator, Templates } from "@future-agi/ai-evaluation";

const evaluator = new Evaluator();

const result = await evaluator.evaluate(
  "synthetic_image_evaluator",
  {
    image: "https://www.esparklearning.com/app/uploads/2024/04/Albert-Einstein-generated-by-AI-1024x683.webp"
  },
  {
    modelName: "turing_flash",
  }
);

console.log(result);

​

What to do If you get Undesired Results

• For actual photographs mistakenly identified as synthetic:
  • Ensure the image has not been heavily processed or filtered
  • Check that the image doesn’t have unusual artifacts from compression or editing
  • Consider providing higher resolution versions if available
• For synthetic images not being detected:
  • Be aware that newer AI generation models are becoming increasingly photorealistic
  • Some AI-generated images that were post-processed or combined with real photographs may be harder to detect
  • The evaluation works best with full images rather than small crops or heavily modified versions

​

Comparing Synthetic Image Evaluator with Similar Evals

• Caption Hallucination: While Synthetic Image Evaluator determines if an image was artificially created, Caption Hallucination evaluates whether descriptions of images contain fabricated elements not visible in the image.
• Toxicity: Synthetic Image Evaluator focuses on the creation method of images, whereas Toxicity evaluates whether content contains harmful elements.