.. _ai_summary plugin: ========== AI Summary ========== .. sidebar:: Further reading .. - :ref:`Configuration ` - :ref:`dev plugin` - :ref:`result types` The AI Summary plugin shows a generated answer above the search results. The text comes from an LLM server the administrator runs; see :ref:`its configuration ` for how to set one up. The interesting part of this plugin is *when* things happen. Generating an answer takes seconds, and a search engine that waits seconds before painting anything is a broken search engine. So the plugin never blocks the result page: it renders an empty box, and the browser fills that box afterwards from a second, streaming request. Request flow ============ .. _ai_summary dataflow: .. kernel-render:: DOT :alt: Data flow between browser, SearXNG and the LLM server :caption: A search that produces a summary: two requests, not one digraph ai_summary { rankdir=LR; graph [fontname="sans-serif", ranksep=1.1, nodesep=0.4]; node [fontname="sans-serif", fontsize=11, shape=box, style="rounded,filled", fillcolor="#f4f4f4", color="#999999"]; edge [fontname="sans-serif", fontsize=9, color="#666666"]; browser [label="browser\n(simple theme)"]; searxng [label="SearXNG"]; engines [label="search engines", fillcolor="#ffffff"]; llm [label="LLM server\nOllama, vLLM, ...", fillcolor="#ffffff"]; browser -> searxng [label=" 1 GET /search"]; searxng -> engines [label=" 2 query"]; searxng -> browser [label=" 3 result page,\l empty summary box\l", constraint=false]; browser -> searxng [label=" 4 POST /ai_summary\l (query + results)\l"]; searxng -> llm [label=" 5 POST /v1/chat/completions"]; llm -> searxng [label=" 6 SSE token stream", constraint=false]; searxng -> browser [label=" 7 NDJSON token stream", constraint=false]; } Steps 1--3 are an ordinary SearXNG search. The plugin's :py:obj:`post_search ` hook adds an empty :py:obj:`searx.result_types.AiSummary` placeholder to the answer area and returns immediately, so the page is not delayed. Steps 4--7 happen in the browser after the page has painted. ``client/simple/src/js/plugin/AiSummary.ts`` posts to the ``/ai_summary`` endpoint, which opens a streaming request to the LLM server and re-emits the tokens as they arrive. The user sees the answer being written. Two format changes happen along the way. The LLM server speaks `SSE`_ (``data: {...}`` lines, terminated by ``data: [DONE]``), because that is what the OpenAI chat completions API specifies. SearXNG re-emits that to the browser as `NDJSON`_ -- one JSON object per line, ``{"delta": "..."}`` for each chunk of text and a final ``{"done": true}``. NDJSON is used because the browser reads the body with ``fetch`` and a stream reader, where SSE's ``EventSource`` would be the wrong tool: ``EventSource`` cannot issue a POST. When no summary is generated ============================ :py:obj:`post_search ` skips the placeholder entirely for: - page two and beyond -- a summary belongs with the first impression of a query - anything but the *general* category - non-HTML output formats (the JSON, CSV and RSS APIs) - queries where an engine already produced an infobox (wikipedia, wikidata) or an instant answer (e.g. ddg definitions) - an empty query, or no LLM server configured The infobox rule mirrors what the big engines do: if the query is a lookup of a well known entity, that entity's own data is a better answer than a generated paragraph. Where the API key goes ====================== The administrator's ``api_key`` is only ever sent to the administrator's ``base_url``. This matters because users may set their own server in the ``ai_summary_server`` preference: without the check, any user of the instance could point that preference at a host they control and collect the instance's key from the ``Authorization`` header. Users authenticate to their own server with their own ``ai_summary_api_key`` preference, which is stored in a cookie and deliberately excluded from the shareable preferences URL. :py:obj:`_server_api_key ` implements the rule; a URL carrying credentials in its userinfo is rejected outright, because HTTP clients turn that into an ``Authorization`` header of the user's choosing. Implementation notes ==================== The ``/ai_summary`` route is registered in :py:obj:`searx.webapp`, next to the favicon proxy, rather than in the plugin's ``init()``. Flask does not allow ``add_url_rule`` after the first request has been handled, and registering it from a plugin breaks the test suite. Requests to the LLM server bypass :py:obj:`searx.network` and are sent with a plain :py:obj:`httpx.Client`. The outgoing proxy configuration is deliberately not applied: an LLM server usually sits on localhost or in the local network, which is exactly what an outgoing proxy is configured to avoid. The streaming response uses ``direct_passthrough``, so the generator must yield ``bytes`` -- werkzeug asserts on ``str``, and the Flask test client does not catch it. .. _SSE: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events .. _NDJSON: https://github.com/ndjson/ndjson-spec Reference ========= .. automodule:: searx.plugins.ai_summary :members: