Multi-Modality

Langfuse supports rendering Multi-Modal traces, including both text and image formats. We follow OpenAI’s format convention.

How to trace Multi-Modal content in Langfuse?

To utilize our Multi-Modal Trace support, your trace or observation input/output should include a list of messages comprising the conversation so far. Each message should contain a role (system, user, assistant) and content. To display multi-modal content, you can pass a combination of text and image URLs. The content property of the messages follows the OpenAI convention.

Visual Representation in Langfuse

When the “Markdown” option is enabled in the Langfuse UI, you can click on the image icon to preview the image inline.

Content Format

Content Examples

{
  "content": [
    {
      "role": "system",
      "content": "You are an AI trained to describe and interpret images. Describe the main objects and actions in the image."
    },
    {
      "role": "user",
      "content": [
        { 
          "type": "text",
          "text": "What's happening in this image?"
        },
        { 
          "type": "image_url",
          "image_url": {
            "url": "https://example.com/image.jpg"
          }
        }
      ]
    }
  ]
}

{
  "content": [
    {"role": "system", "content": "You are a very accurate calculator. You output only the result of the calculation."},
    {"role": "user", "content": "1 + 1 = "}
  ]
}

For more details and examples, refer to our OpenAI cookbook.

GitHub Discussions