diff --git a/docs/admin/settings/settings_ai_summary.rst b/docs/admin/settings/settings_ai_summary.rst index 37d355f1d..83487d2af 100644 --- a/docs/admin/settings/settings_ai_summary.rst +++ b/docs/admin/settings/settings_ai_summary.rst @@ -13,11 +13,10 @@ Configuration of the :ref:`AI Summary plugin `, which shows a short AI generated answer above the search results. -The text is produced by an **LLM server that you run** -- SearXNG does not ship -a model and does not talk to any AI provider on its own. Anything that speaks -the `OpenAI chat completions API`_ works: `Ollama`_, vLLM, llama.cpp, LM Studio, -Hugging Face TGI and others. The plugin is not activated by default. - +The text is produced by an LLM server that you run; SearXNG ships no model and +contacts no AI provider of its own. Any server implementing the `OpenAI chat +completions API`_ works: `Ollama`_, vLLM, llama.cpp, LM Studio, Hugging Face +TGI and others. The plugin is not activated by default. .. _ai_summary quickstart: @@ -39,8 +38,8 @@ A local setup on the same machine as SearXNG, in four steps. ollama pull gemma3:4b ``gemma3:4b`` needs roughly 4 GB of memory and runs on CPU if you have no GPU. -On a small machine use ``gemma3:1b`` instead; any model in the `Ollama -library`_ works. +On a smaller machine use ``gemma3:1b``; any model in the `Ollama library`_ +works. **3. Configure SearXNG** @@ -71,20 +70,18 @@ Add this to your ``settings.yml``: .. warning:: A ``plugins:`` block **replaces** the default list, it is not merged into it - (:ref:`settings plugins`). If you list only the AI Summary plugin, every - other plugin is switched off. That is why the block above repeats the - defaults -- drop the lines for plugins you do not want. + (:ref:`settings plugins`). Listing only the AI Summary plugin switches + every other plugin off, which is why the block above repeats the defaults -- + drop the lines for plugins you do not want. **4. Restart SearXNG** and search for something. -The summary appears above the results while it is still being written. If -nothing appears, see :ref:`ai_summary troubleshooting`. - +The summary appears above the results while it is still being written. Options ======= -Only ``base_url`` is required; a model is needed too, but if you leave ``model`` +Only ``base_url`` is required. A model is needed too, but if ``model`` is left empty the first entry of ``models`` is used. .. code:: yaml @@ -97,13 +94,12 @@ empty the first entry of ``models`` is used. .. autoclass:: searx.ai_summary.SettingsAISummary :members: +Servers that require authentication +=================================== -Servers that need a password -============================ - -Some servers require authentication -- vLLM and llama.cpp when started with -``--api-key``, a gateway such as LiteLLM, or any LLM server placed behind an -authenticating reverse proxy. Give SearXNG the key with ``api_key``: +vLLM and llama.cpp started with ``--api-key``, a gateway such as LiteLLM, or an +LLM server behind an authenticating reverse proxy all expect a key. Set it +with ``api_key``: .. code:: yaml @@ -112,59 +108,47 @@ authenticating reverse proxy. Give SearXNG the key with ``api_key``: api_key: "sk-..." model: "gemma3:4b" -The key is sent as an ``Authorization: Bearer`` header, and only to the server -in ``base_url``. Users who point the ``ai_summary_server`` preference at a -server of their own never receive it; they set their own key in the -``ai_summary_api_key`` preference instead. - -SearXNG has no separate secret store, so the key sits in ``settings.yml`` -- -make sure that file is readable only by the user SearXNG runs as. +The key is sent as an ``Authorization: Bearer`` header and only to the server in +``base_url``. Users who configure a server of their own in the +``ai_summary_server`` preference never receive it; they set their own key in the +``ai_summary_api_key`` preference. +SearXNG has no separate secret store, so the key is held in ``settings.yml`` -- +that file should be readable only by the user SearXNG runs as. .. _ai_summary grounding: Grounding ========= -Grounding decides *what the model is told*: +With ``grounding`` enabled, which is the default, the query **and the top search +results** (title, URL and snippet, at most ``max_context_items`` of them) are +sent to the LLM server, and the answer reflects what the search found. With it +disabled only the query is sent and the model answers from its training data. +Users can change this in the ``ai_summary_grounding`` preference. -``grounding: true`` (the default) - The query **and the top search results** (title, URL and snippet, at most - ``max_context_items`` of them) are sent to the LLM server. Answers reflect - what the search actually found, so they are more accurate and more current. +What leaves the network therefore depends on where the LLM server runs: with a +server on localhost or in the local network, nothing does. -``grounding: false`` - Only the query is sent, and the model answers from its own training data. - Faster and cheaper, but the answer can be outdated or invented. +Public SearXNG instances +======================== -Users can switch this in their preferences (``ai_summary_grounding``). +The server URL, model and API key are user preferences so that someone running +SearXNG at home can switch models or debug their LLM server from the +preferences page, without editing ``settings.yml`` and restarting. -How much this matters for privacy depends entirely on where the LLM server runs. -With a server on localhost or in your own network, nothing leaves that network. -With a hosted server, the query and the result snippets are sent to the -provider. This is the main reason a local server is recommended. - - -Public instances -================ +On public SearXNG instances, those same preferences let any visitor choose the +address the summary request is sent to. The request is made by the SearXNG +host, so a visitor can use it to reach machines on your network that they have +no route to themselves -- an `SSRF`_ vector. .. attention:: - The plugin is designed for private instances. Read this before enabling it - on a public one. + Lock ``ai_summary_server`` on any SearXNG instance that is reachable by + people outside your household. -Two things change on a public instance: - -**Every search costs real work.** A summary is a full LLM inference. Traffic -that was cheap to serve becomes expensive, on hardware you pay for. - -**Users can choose the server SearXNG talks to.** The ``ai_summary_server`` -preference makes your instance send requests to an address of the user's -choosing, which is a `SSRF`_ vector: it can be pointed at services inside your -network that are not reachable from outside. - -Lock the preferences so that only your configuration is used -(:ref:`settings preferences`): +Locking a preference makes SearXNG use your configured value and ignore the +user's (:ref:`settings preferences`): .. code:: yaml @@ -175,46 +159,11 @@ Lock the preferences so that only your configuration is used - ai_summary_model - ai_summary_grounding -Locking ``ai_summary_server`` closes the SSRF vector; locking -``ai_summary_api_key`` stops users making your instance send an -``Authorization`` header of their choosing to a host of their choosing. - - -.. _ai_summary troubleshooting: - -Troubleshooting -=============== - -**No summary appears at all.** - The plugin deliberately stays quiet in several cases: on page two and beyond, - outside the *general* category, and when an engine already answered the query - with an infobox or an instant answer. Try a question-like query on the first - page. Check that the plugin is enabled both in ``settings.yml`` and in your - own preferences. - -**The model list in the preferences is empty.** - SearXNG asks the server for its models (``GET /v1/models``) once at startup. - An empty list means that request failed -- the server was not running yet, - the URL is wrong, or it needs an ``api_key``. The log records the reason; - restart SearXNG after fixing it. - -**The summary box shows an error.** - SearXNG could not reach the LLM server, or the server rejected the request. - Check ``base_url`` from the SearXNG machine, confirm the model name exists on - that server (``ollama list``), and check the SearXNG log -- it records the - status and the message the server replied with, which usually names the cause. - -**The first search after a while fails, the next one works.** - An idle LLM server unloads the model and has to load it again, and it sends - nothing at all while doing so. If that takes longer than ``read_timeout`` - (30 s by default) the request is abandoned. SearXNG repeats the request once, - which covers a normal load, but a large model on slow storage can need more: - raise ``read_timeout``, or keep the model in memory -- with Ollama, set - ``OLLAMA_KEEP_ALIVE`` (for example ``-1`` to never unload it). - -**The summary starts, then stops mid-sentence.** - The answer exceeded ``stream_timeout`` (120 s by default). Large models on - CPU are slow; either raise the limit or use a smaller model. +``ai_summary_server`` is the one that matters: locking it closes the SSRF +vector. Locking ``ai_summary_api_key`` additionally stops visitors making your +SearXNG instance send an ``Authorization`` header of their choosing to a host of +their choosing. ``ai_summary_model`` and ``ai_summary_grounding`` are about +cost and consistency rather than security. .. _Ollama: https://ollama.com/ .. _Ollama library: https://ollama.com/library diff --git a/docs/dev/plugins/ai_summary.rst b/docs/dev/plugins/ai_summary.rst index f8830bea5..142ee3ec5 100644 --- a/docs/dev/plugins/ai_summary.rst +++ b/docs/dev/plugins/ai_summary.rst @@ -11,23 +11,14 @@ AI Summary - :ref:`result types` The AI Summary plugin shows a generated answer above the ordinary search -results, unless an engine has already answered the query directly -- with a -Wikipedia infobox, for instance, or an instant answer. It is meant to run -against a local LLM server and speaks the `OpenAI chat completions API`_, so it -works with `Ollama`_, Hugging Face TGI, LiteLLM, vLLM, llama.cpp and anything -else that implements that specification. See :ref:`its configuration ` for how to set one up. - -The purpose of the summary is not to tell you what the model memorised during -training, but to summarise the up to date results your query actually returned. -That is what the *grounding* setting does, and why it is enabled by default. - -Generating an answer takes seconds, and a search engine that waits seconds -before painting anything is a broken search engine. The summary is therefore -produced asynchronously: the plugin renders an empty box, the result page is -delivered immediately, and the browser fills that box from a second, streaming -request. The results below stay readable and scrollable the whole time. +results. The text comes from an LLM server run by the administrator, which +speaks the `OpenAI chat completions API`_ -- `Ollama`_, Hugging Face TGI, +LiteLLM, vLLM, llama.cpp and anything else implementing that specification. +See :ref:`its configuration ` for how to set one up. +The summary is generated asynchronously: the result page is delivered without +delay and carries an empty placeholder, which the browser fills from a second, +streaming request. Request flow ============ @@ -45,7 +36,7 @@ Request flow fillcolor="#f4f4f4", color="#999999"]; edge [fontname="sans-serif", fontsize=9, color="#666666"]; - browser [label="browser\n(simple theme)"]; + browser [label="browser"]; searxng [label="SearXNG"]; engines [label="search engines", fillcolor="#ffffff"]; llm [label="LLM server\nOllama, vLLM, ...", fillcolor="#ffffff"]; @@ -59,84 +50,54 @@ Request flow searxng -> browser [label=" 7 NDJSON token stream", constraint=false]; } -Steps 1--3 are an ordinary SearXNG search. The plugin's -:py:obj:`post_search ` hook -adds an empty :py:obj:`searx.result_types.AiSummary` placeholder to the answer -area and returns immediately, so the page is not delayed. +Steps 1 to 3 are an ordinary SearXNG search. :py:obj:`SXNGPlugin.post_search +` adds an empty +:py:obj:`searx.result_types.AiSummary` placeholder to the answer area and +returns; the result page is not delayed. -Steps 4--7 happen in the browser after the page has painted. +Steps 4 to 7 run in the browser once the page is rendered. ``client/simple/src/js/plugin/AiSummary.ts`` posts to the ``/ai_summary`` -endpoint, which opens a streaming request to the LLM server and re-emits the -tokens as they arrive. The user sees the answer being written. - -Two format changes happen along the way. The LLM server speaks `SSE`_ -(``data: {...}`` lines, terminated by ``data: [DONE]``), because that is what -the OpenAI chat completions API specifies. SearXNG re-emits that to the -browser as `NDJSON`_ -- one JSON object per line, ``{"delta": "..."}`` for each -chunk of text and a final ``{"done": true}``. NDJSON is used because the -browser reads the body with ``fetch`` and a stream reader, where SSE's -``EventSource`` would be the wrong tool: ``EventSource`` cannot issue a POST. +endpoint (registered in :py:obj:`searx.webapp`), which opens a streaming +request to the LLM server and re-emits the tokens as they arrive. +The two streams use different formats. The LLM server sends `SSE`_ -- +``data: {...}`` lines terminated by ``data: [DONE]``. SearXNG re-emits them to +the browser as `NDJSON`_: one JSON object per line, ``{"delta": "..."}`` for +each chunk of text and a final ``{"done": true}``. When no summary is generated ============================ -:py:obj:`post_search ` skips -the placeholder entirely for: +:py:obj:`SXNGPlugin.post_search +` adds no placeholder for: -- page two and beyond -- a summary belongs with the first impression of a query -- anything but the *general* category +- page two and beyond +- categories other than *general* - non-HTML output formats (the JSON, CSV and RSS APIs) -- queries where an engine already produced an infobox (wikipedia, wikidata) or +- queries an engine already answered with an infobox (wikipedia, wikidata) or an instant answer (e.g. ddg definitions) - an empty query, or no LLM server configured -The infobox rule mirrors what the big engines do: if the query is a lookup of a -well known entity, that entity's own data is a better answer than a generated -paragraph. +API keys +======== - -Where the API key goes -====================== - -The administrator's ``api_key`` is only ever sent to the administrator's -``base_url``. This matters because users may set their own server in the -``ai_summary_server`` preference: without the check, any user of the instance -could point that preference at a host they control and collect the instance's -key from the ``Authorization`` header. Users authenticate to their own server -with their own ``ai_summary_api_key`` preference, which is stored in a cookie -and deliberately excluded from the shareable preferences URL. +The administrator's ``api_key`` is sent only to the configured ``base_url``. +Users who point the ``ai_summary_server`` preference at a server of their own +authenticate it with their own ``ai_summary_api_key`` preference, which is +stored in a cookie and excluded from the preferences URL; the administrator's +key is never sent to such a server. A server URL carrying credentials in its +userinfo is ignored, and the administrator's default is used instead. :py:obj:`_server_api_key ` implements -the rule; a URL carrying credentials in its userinfo is rejected outright, -because HTTP clients turn that into an ``Authorization`` header of the user's -choosing. - - -Implementation notes -==================== - -The ``/ai_summary`` route is registered in :py:obj:`searx.webapp`, next to the -favicon proxy, rather than in the plugin's ``init()``. Flask does not allow -``add_url_rule`` after the first request has been handled, and registering it -from a plugin breaks the test suite. - -Requests to the LLM server bypass :py:obj:`searx.network` and are sent with a -plain :py:obj:`httpx.Client`. The outgoing proxy configuration is deliberately -not applied: an LLM server usually sits on localhost or in the local network, -which is exactly what an outgoing proxy is configured to avoid. - -The streaming response uses ``direct_passthrough``, so the generator must yield -``bytes`` -- werkzeug asserts on ``str``, and the Flask test client does not -catch it. - -.. _Ollama: https://ollama.com/ -.. _OpenAI chat completions API: https://platform.openai.com/docs/api-reference/chat -.. _SSE: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events -.. _NDJSON: https://github.com/ndjson/ndjson-spec +the rule. Reference ========= .. automodule:: searx.plugins.ai_summary :members: + +.. _Ollama: https://ollama.com/ +.. _OpenAI chat completions API: https://platform.openai.com/docs/api-reference/chat +.. _SSE: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events +.. _NDJSON: https://github.com/ndjson/ndjson-spec