searxng/docs/dev/plugins/ai_summary.rst

.. _ai_summary plugin:

==========
AI summary
==========

.. sidebar:: Further reading ..

   - :ref:`settings ai_summary`
   - :ref:`dev plugin`
   - :ref:`result types`

.. contents::
   :depth: 2
   :local:
   :backlinks: entry

The AI summary plugin shows a generated answer above the search results.  The
text comes from an LLM server the administrator runs; see :ref:`settings
ai_summary` for how to set one up.

The interesting part of this plugin is *when* things happen.  Generating an
answer takes seconds, and a search engine that waits seconds before painting
anything is a broken search engine.  So the plugin never blocks the result
page: it renders an empty box, and the browser fills that box afterwards from a
second, streaming request.


Request flow
============

.. _ai_summary dataflow:

.. kernel-render:: DOT
   :alt: Data flow between browser, SearXNG and the LLM server
   :caption: A search that produces a summary: two requests, not one

   digraph ai_summary {
     rankdir=LR;
     graph [fontname="sans-serif", ranksep=1.1, nodesep=0.4];
     node  [fontname="sans-serif", fontsize=11, shape=box, style="rounded,filled",
            fillcolor="#f4f4f4", color="#999999"];
     edge  [fontname="sans-serif", fontsize=9, color="#666666"];

     browser  [label="browser\n(simple theme)"];
     searxng  [label="SearXNG"];
     engines  [label="search engines", fillcolor="#ffffff"];
     llm      [label="LLM server\nOllama, vLLM, ...", fillcolor="#ffffff"];

     browser -> searxng [label=" 1  GET /search"];
     searxng -> engines [label=" 2  query"];
     searxng -> browser [label=" 3  result page,\l    empty summary box\l", constraint=false];
     browser -> searxng [label=" 4  POST /ai_summary\l    (query + results)\l"];
     searxng -> llm     [label=" 5  POST /v1/chat/completions"];
     llm     -> searxng [label=" 6  SSE token stream", constraint=false];
     searxng -> browser [label=" 7  NDJSON token stream", constraint=false];
   }

Steps 1--3 are an ordinary SearXNG search.  The plugin's
:py:obj:`post_search <searx.plugins.ai_summary.SXNGPlugin.post_search>` hook
adds an empty :py:obj:`searx.result_types.AiSummary` placeholder to the answer
area and returns immediately, so the page is not delayed.

Steps 4--7 happen in the browser after the page has painted.
``client/simple/src/js/plugin/AiSummary.ts`` posts to the ``/ai_summary``
endpoint, which opens a streaming request to the LLM server and re-emits the
tokens as they arrive.  The user sees the answer being written.

Two format changes happen along the way.  The LLM server speaks `SSE`_
(``data: {...}`` lines, terminated by ``data: [DONE]``), because that is what
the OpenAI chat completions API specifies.  SearXNG re-emits that to the
browser as `NDJSON`_ -- one JSON object per line, ``{"delta": "..."}`` for each
chunk of text and a final ``{"done": true}``.  NDJSON is used because the
browser reads the body with ``fetch`` and a stream reader, where SSE's
``EventSource`` would be the wrong tool: ``EventSource`` cannot issue a POST.


When no summary is generated
============================

:py:obj:`post_search <searx.plugins.ai_summary.SXNGPlugin.post_search>` skips
the placeholder entirely for:

- page two and beyond -- a summary belongs with the first impression of a query
- anything but the *general* category
- non-HTML output formats (the JSON, CSV and RSS APIs)
- queries where an engine already produced an infobox (wikipedia, wikidata) or
  an instant answer (e.g. ddg definitions)
- an empty query, or no LLM server configured

The infobox rule mirrors what the big engines do: if the query is a lookup of a
well known entity, that entity's own data is a better answer than a generated
paragraph.


Where the API key goes
======================

The administrator's ``api_key`` is only ever sent to the administrator's
``base_url``.  This matters because users may set their own server in the
``ai_summary_server`` preference: without the check, any user of the instance
could point that preference at a host they control and collect the instance's
key from the ``Authorization`` header.  Users authenticate to their own server
with their own ``ai_summary_api_key`` preference, which is stored in a cookie
and deliberately excluded from the shareable preferences URL.

:py:obj:`_server_api_key <searx.plugins.ai_summary._server_api_key>` implements
the rule; a URL carrying credentials in its userinfo is rejected outright,
because HTTP clients turn that into an ``Authorization`` header of the user's
choosing.


Implementation notes
====================

The ``/ai_summary`` route is registered in :py:obj:`searx.webapp`, next to the
favicon proxy, rather than in the plugin's ``init()``.  Flask does not allow
``add_url_rule`` after the first request has been handled, and registering it
from a plugin breaks the test suite.

Requests to the LLM server bypass :py:obj:`searx.network` and are sent with a
plain :py:obj:`httpx.Client`.  The outgoing proxy configuration is deliberately
not applied: an LLM server usually sits on localhost or in the local network,
which is exactly what an outgoing proxy is configured to avoid.

The streaming response uses ``direct_passthrough``, so the generator must yield
``bytes`` -- werkzeug asserts on ``str``, and the Flask test client does not
catch it.

.. _SSE: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events
.. _NDJSON: https://github.com/ndjson/ndjson-spec

Reference
=========

.. automodule:: searx.plugins.ai_summary
   :members: