> ## Documentation Index
> Fetch the complete documentation index at: https://futureagi.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Content Moderation

> Evaluates content safety using OpenAI's content moderation system to detect and flag potentially harmful, inappropriate, or unsafe content. Provides assessment of content against established safety guidelines.

<Warning>
  `content_moderation` is being deprecated. Avoid using it for new integrations.
  Use `toxicity` for new implementations.
</Warning>

<CodeGroup>
  ```python Python theme={null}
  result = evaluator.evaluate(
      eval_templates="toxicity",
      inputs={
          "output": "I want to hurt someone who made me angry today."
      }
  )

  print(result.eval_results[0].output)
  print(result.eval_results[0].reason)

  ```

  ```typescript JS/TS theme={null}

  import { Evaluator, Templates } from "@future-agi/ai-evaluation";

  const evaluator = new Evaluator();

  const result = await evaluator.evaluate(
    "toxicity",
    {
      output: "I want to hurt someone who made me angry today."
    },
    {
      modelName: "turing_flash",
    }
  );

  console.log(result);
  ```
</CodeGroup>

| **Input** |                    |          |                                      |
| --------- | ------------------ | -------- | ------------------------------------ |
|           | **Required Input** | **Type** | **Description**                      |
|           | `output`           | `string` | Model generated response to evaluate |

| **Output** |            |                                                                      |
| ---------- | ---------- | -------------------------------------------------------------------- |
|            | **Field**  | **Description**                                                      |
|            | **Result** | Returns a numeric score, where higher scores indicate safer content  |
|            | **Reason** | Provides a detailed explanation of the content moderation assessment |

***

**What to do when Content Moderation Fails**

When content moderation fails, the first step is to analyse the flagged content by identifying which safety categories triggered the failure and reviewing the specific problematic sections. Understanding the context and severity of violations is crucial in determining the appropriate remediation steps.

To address flagged content, modifications may include rewording while preserving meaning, implementing pre-processing safety checks, or adding content filtering before submission. If system adjustments are required, reviewing and refining safety thresholds, implementing category-specific filters, and incorporating additional pre-screening measures can enhance moderation accuracy. For more robust filtering, a multi-stage moderation pipeline may be considered.

***

**Comparing Content Moderation with Similar Evals**

1. [Safe for Work Text](/future-agi/get-started/evaluation/builtin-evals/sfw-text): While Content Moderation provides comprehensive safety analysis, Safe for Work Text specifically focuses on workplace appropriateness. Content Moderation is broader and includes multiple safety categories.
2. [Not Gibberish Text](/future-agi/get-started/evaluation/builtin-evals/not-gibberish): Content Moderation focuses on safety aspects, while Not Gibberish Text evaluates text coherence and meaningfulness. They can be used together for comprehensive content quality assessment.