104 lines
4.0 KiB
ReStructuredText
104 lines
4.0 KiB
ReStructuredText
|
.. _ai_summary plugin:
|
||
|
|
|
||
|
|
==========
|
||
|
|
AI Summary
|
||
|
|
==========
|
||
|
|
|
||
|
|
.. sidebar:: Further reading ..
|
||
|
|
|
||
|
|
- :ref:`Configuration <settings ai_summary>`
|
||
|
|
- :ref:`dev plugin`
|
||
|
|
- :ref:`result types`
|
||
|
|
|
||
|
|
The AI Summary plugin shows a generated answer above the ordinary search
|
||
|
|
results. The text comes from an LLM server run by the administrator, which
|
||
|
|
speaks the `OpenAI chat completions API`_ -- `Ollama`_, Hugging Face TGI,
|
||
|
|
LiteLLM, vLLM, llama.cpp and anything else implementing that specification.
|
||
|
|
See :ref:`its configuration <settings ai_summary>` for how to set one up.
|
||
|
|
|
||
|
|
The summary is generated asynchronously: the result page is delivered without
|
||
|
|
delay and carries an empty placeholder, which the browser fills from a second,
|
||
|
|
streaming request.
|
||
|
|
|
||
|
|
Request flow
|
||
|
|
============
|
||
|
|
|
||
|
|
.. _ai_summary dataflow:
|
||
|
|
|
||
|
|
.. kernel-render:: DOT
|
||
|
|
:alt: Data flow between browser, SearXNG and the LLM server
|
||
|
|
:caption: A search that produces a summary: two requests, not one
|
||
|
|
|
||
|
|
digraph ai_summary {
|
||
|
|
rankdir=LR;
|
||
|
|
graph [fontname="sans-serif", ranksep=1.1, nodesep=0.4];
|
||
|
|
node [fontname="sans-serif", fontsize=11, shape=box, style="rounded,filled",
|
||
|
|
fillcolor="#f4f4f4", color="#999999"];
|
||
|
|
edge [fontname="sans-serif", fontsize=9, color="#666666"];
|
||
|
|
|
||
|
|
browser [label="browser"];
|
||
|
|
searxng [label="SearXNG"];
|
||
|
|
engines [label="search engines", fillcolor="#ffffff"];
|
||
|
|
llm [label="LLM server\nOllama, vLLM, ...", fillcolor="#ffffff"];
|
||
|
|
|
||
|
|
browser -> searxng [label=" 1 GET /search"];
|
||
|
|
searxng -> engines [label=" 2 query"];
|
||
|
|
searxng -> browser [label=" 3 result page,\l empty summary box\l", constraint=false];
|
||
|
|
browser -> searxng [label=" 4 POST /ai_summary\l (query + results)\l"];
|
||
|
|
searxng -> llm [label=" 5 POST /v1/chat/completions"];
|
||
|
|
llm -> searxng [label=" 6 SSE token stream", constraint=false];
|
||
|
|
searxng -> browser [label=" 7 NDJSON token stream", constraint=false];
|
||
|
|
}
|
||
|
|
|
||
|
|
Steps 1 to 3 are an ordinary SearXNG search. :py:obj:`SXNGPlugin.post_search
|
||
|
|
<searx.plugins.ai_summary.SXNGPlugin.post_search>` adds an empty
|
||
|
|
:py:obj:`searx.result_types.AiSummary` placeholder to the answer area and
|
||
|
|
returns; the result page is not delayed.
|
||
|
|
|
||
|
|
Steps 4 to 7 run in the browser once the page is rendered.
|
||
|
|
``client/simple/src/js/plugin/AiSummary.ts`` posts to the ``/ai_summary``
|
||
|
|
endpoint (registered in :py:obj:`searx.webapp`), which opens a streaming
|
||
|
|
request to the LLM server and re-emits the tokens as they arrive.
|
||
|
|
|
||
|
|
The two streams use different formats. The LLM server sends `SSE`_ --
|
||
|
|
``data: {...}`` lines terminated by ``data: [DONE]``. SearXNG re-emits them to
|
||
|
|
the browser as `NDJSON`_: one JSON object per line, ``{"delta": "..."}`` for
|
||
|
|
each chunk of text and a final ``{"done": true}``.
|
||
|
|
|
||
|
|
When no summary is generated
|
||
|
|
============================
|
||
|
|
|
||
|
|
:py:obj:`SXNGPlugin.post_search
|
||
|
|
<searx.plugins.ai_summary.SXNGPlugin.post_search>` adds no placeholder for:
|
||
|
|
|
||
|
|
- page two and beyond
|
||
|
|
- categories other than *general*
|
||
|
|
- non-HTML output formats (the JSON, CSV and RSS APIs)
|
||
|
|
- queries an engine already answered with an infobox (wikipedia, wikidata) or
|
||
|
|
an instant answer (e.g. ddg definitions)
|
||
|
|
- an empty query, or no LLM server configured
|
||
|
|
|
||
|
|
API keys
|
||
|
|
========
|
||
|
|
|
||
|
|
The administrator's ``api_key`` is sent only to the configured ``base_url``.
|
||
|
|
Users who point the ``ai_summary_server`` preference at a server of their own
|
||
|
|
authenticate it with their own ``ai_summary_api_key`` preference, which is
|
||
|
|
stored in a cookie and excluded from the preferences URL; the administrator's
|
||
|
|
key is never sent to such a server. A server URL carrying credentials in its
|
||
|
|
userinfo is ignored, and the administrator's default is used instead.
|
||
|
|
|
||
|
|
:py:obj:`_server_api_key <searx.plugins.ai_summary._server_api_key>` implements
|
||
|
|
the rule.
|
||
|
|
|
||
|
|
Reference
|
||
|
|
=========
|
||
|
|
|
||
|
|
.. automodule:: searx.plugins.ai_summary
|
||
|
|
:members:
|
||
|
|
|
||
|
|
.. _Ollama: https://ollama.com/
|
||
|
|
.. _OpenAI chat completions API: https://platform.openai.com/docs/api-reference/chat
|
||
|
|
.. _SSE: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events
|
||
|
|
.. _NDJSON: https://github.com/ndjson/ndjson-spec
|