jasonwitty
37d7d11959
Both pages carried design rationale and implementation notes that belong in a review conversation rather than in documentation: why NDJSON is sent to the browser instead of SSE, why the endpoint is registered in webapp.py, why the outgoing proxy is bypassed. They are gone, along with the troubleshooting section. The section about public instances no longer opens by saying the plugin is meant for private instances, which contradicted the instructions that followed it. It now explains why the server, model and key are user preferences at all -- so that they can be changed at home without editing settings.yml -- and what that means once other people can reach the instance. "AI Summary" is the plugin, "AI summary" is the text it produces, and an "instance" is always a SearXNG one; a page that also talks about LLM servers cannot leave that to context. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
104 lines
4.0 KiB
ReStructuredText
104 lines
4.0 KiB
ReStructuredText
.. _ai_summary plugin:
|
|
|
|
==========
|
|
AI Summary
|
|
==========
|
|
|
|
.. sidebar:: Further reading ..
|
|
|
|
- :ref:`Configuration <settings ai_summary>`
|
|
- :ref:`dev plugin`
|
|
- :ref:`result types`
|
|
|
|
The AI Summary plugin shows a generated answer above the ordinary search
|
|
results. The text comes from an LLM server run by the administrator, which
|
|
speaks the `OpenAI chat completions API`_ -- `Ollama`_, Hugging Face TGI,
|
|
LiteLLM, vLLM, llama.cpp and anything else implementing that specification.
|
|
See :ref:`its configuration <settings ai_summary>` for how to set one up.
|
|
|
|
The summary is generated asynchronously: the result page is delivered without
|
|
delay and carries an empty placeholder, which the browser fills from a second,
|
|
streaming request.
|
|
|
|
Request flow
|
|
============
|
|
|
|
.. _ai_summary dataflow:
|
|
|
|
.. kernel-render:: DOT
|
|
:alt: Data flow between browser, SearXNG and the LLM server
|
|
:caption: A search that produces a summary: two requests, not one
|
|
|
|
digraph ai_summary {
|
|
rankdir=LR;
|
|
graph [fontname="sans-serif", ranksep=1.1, nodesep=0.4];
|
|
node [fontname="sans-serif", fontsize=11, shape=box, style="rounded,filled",
|
|
fillcolor="#f4f4f4", color="#999999"];
|
|
edge [fontname="sans-serif", fontsize=9, color="#666666"];
|
|
|
|
browser [label="browser"];
|
|
searxng [label="SearXNG"];
|
|
engines [label="search engines", fillcolor="#ffffff"];
|
|
llm [label="LLM server\nOllama, vLLM, ...", fillcolor="#ffffff"];
|
|
|
|
browser -> searxng [label=" 1 GET /search"];
|
|
searxng -> engines [label=" 2 query"];
|
|
searxng -> browser [label=" 3 result page,\l empty summary box\l", constraint=false];
|
|
browser -> searxng [label=" 4 POST /ai_summary\l (query + results)\l"];
|
|
searxng -> llm [label=" 5 POST /v1/chat/completions"];
|
|
llm -> searxng [label=" 6 SSE token stream", constraint=false];
|
|
searxng -> browser [label=" 7 NDJSON token stream", constraint=false];
|
|
}
|
|
|
|
Steps 1 to 3 are an ordinary SearXNG search. :py:obj:`SXNGPlugin.post_search
|
|
<searx.plugins.ai_summary.SXNGPlugin.post_search>` adds an empty
|
|
:py:obj:`searx.result_types.AiSummary` placeholder to the answer area and
|
|
returns; the result page is not delayed.
|
|
|
|
Steps 4 to 7 run in the browser once the page is rendered.
|
|
``client/simple/src/js/plugin/AiSummary.ts`` posts to the ``/ai_summary``
|
|
endpoint (registered in :py:obj:`searx.webapp`), which opens a streaming
|
|
request to the LLM server and re-emits the tokens as they arrive.
|
|
|
|
The two streams use different formats. The LLM server sends `SSE`_ --
|
|
``data: {...}`` lines terminated by ``data: [DONE]``. SearXNG re-emits them to
|
|
the browser as `NDJSON`_: one JSON object per line, ``{"delta": "..."}`` for
|
|
each chunk of text and a final ``{"done": true}``.
|
|
|
|
When no summary is generated
|
|
============================
|
|
|
|
:py:obj:`SXNGPlugin.post_search
|
|
<searx.plugins.ai_summary.SXNGPlugin.post_search>` adds no placeholder for:
|
|
|
|
- page two and beyond
|
|
- categories other than *general*
|
|
- non-HTML output formats (the JSON, CSV and RSS APIs)
|
|
- queries an engine already answered with an infobox (wikipedia, wikidata) or
|
|
an instant answer (e.g. ddg definitions)
|
|
- an empty query, or no LLM server configured
|
|
|
|
API keys
|
|
========
|
|
|
|
The administrator's ``api_key`` is sent only to the configured ``base_url``.
|
|
Users who point the ``ai_summary_server`` preference at a server of their own
|
|
authenticate it with their own ``ai_summary_api_key`` preference, which is
|
|
stored in a cookie and excluded from the preferences URL; the administrator's
|
|
key is never sent to such a server. A server URL carrying credentials in its
|
|
userinfo is ignored, and the administrator's default is used instead.
|
|
|
|
:py:obj:`_server_api_key <searx.plugins.ai_summary._server_api_key>` implements
|
|
the rule.
|
|
|
|
Reference
|
|
=========
|
|
|
|
.. automodule:: searx.plugins.ai_summary
|
|
:members:
|
|
|
|
.. _Ollama: https://ollama.com/
|
|
.. _OpenAI chat completions API: https://platform.openai.com/docs/api-reference/chat
|
|
.. _SSE: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events
|
|
.. _NDJSON: https://github.com/ndjson/ndjson-spec
|