diff --git a/AUTHORS.rst b/AUTHORS.rst index 23d8d6db6..6bcc3ba92 100644 --- a/AUTHORS.rst +++ b/AUTHORS.rst @@ -179,3 +179,4 @@ features or generally made SearXNG better: - Tommaso Colella `` - @AgentScrubbles - Filip Mikina `` +- Jason Witty `` diff --git a/docs/admin/settings/settings_ai_summary.rst b/docs/admin/settings/settings_ai_summary.rst index c521c2244..d5609c64e 100644 --- a/docs/admin/settings/settings_ai_summary.rst +++ b/docs/admin/settings/settings_ai_summary.rst @@ -4,80 +4,215 @@ ``ai_summary:`` =============== -Default configuration of the :ref:`AI summary plugin `. -Users configure the LLM server URL (any server implementing the OpenAI chat -completions API: Ollama, vLLM, llama.cpp, LM Studio, Hugging Face TGI, ...) -and the model in the *AI Summary* tab of their preferences; the values below -only act as instance wide defaults. +.. sidebar:: Further reading .. + + - :ref:`ai_summary plugin` + - :ref:`settings plugins` + - :ref:`settings preferences` + +Configuration of the :ref:`AI summary plugin `, which shows a +short AI generated answer above the search results. + +The text is produced by an **LLM server that you run** -- SearXNG does not ship +a model and does not talk to any AI provider on its own. Anything that speaks +the `OpenAI chat completions API`_ works: `Ollama`_, vLLM, llama.cpp, LM Studio, +Hugging Face TGI and others. The plugin is not activated by default. + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _ai_summary quickstart: + +Quickstart +========== + +A local setup on the same machine as SearXNG, in four steps. + +**1. Install Ollama** + +.. code:: sh + + curl -fsSL https://ollama.com/install.sh | sh + +**2. Download a model** + +.. code:: sh + + ollama pull gemma3:4b + +``gemma3:4b`` needs roughly 4 GB of memory and runs on CPU if you have no GPU. +On a small machine use ``gemma3:1b`` instead; any model in the `Ollama +library`_ works. + +**3. Configure SearXNG** + +Add this to your ``settings.yml``: .. code:: yaml ai_summary: base_url: "http://127.0.0.1:11434" - model: "llama3.2:3b" + model: "gemma3:4b" -An LLM server that requires authentication -- e.g. vLLM or llama.cpp started -with ``--api-key``, or a server behind an authenticating reverse proxy -- is -configured with an ``api_key``: + plugins: + searx.plugins.ai_summary.SXNGPlugin: + active: true + # keep the plugins you already had, see the warning below + searx.plugins.calculator.SXNGPlugin: {active: true} + searx.plugins.hash_plugin.SXNGPlugin: {active: true} + searx.plugins.self_info.SXNGPlugin: {active: true} + searx.plugins.unit_converter.SXNGPlugin: {active: true} + searx.plugins.ahmia_filter.SXNGPlugin: {active: true} + searx.plugins.hostnames.SXNGPlugin: {active: true} + searx.plugins.time_zone.SXNGPlugin: {active: true} + searx.plugins.tracker_url_remover.SXNGPlugin: {active: true} + searx.plugins.infinite_scroll.SXNGPlugin: {active: false} + searx.plugins.oa_doi_rewrite.SXNGPlugin: {active: false} + searx.plugins.tor_check.SXNGPlugin: {active: false} + +.. warning:: + + A ``plugins:`` block **replaces** the default list, it is not merged into it + (:ref:`settings plugins`). If you list only the AI summary plugin, every + other plugin is switched off. That is why the block above repeats the + defaults -- drop the lines for plugins you do not want. + +**4. Restart SearXNG** and search for something. + +The summary appears above the results while it is still being written. If +nothing appears, see :ref:`ai_summary troubleshooting`. + + +Options +======= + +Only ``base_url`` is required; a model is needed too, but if you leave ``model`` +empty the first entry of ``models`` is used. + +.. code:: yaml + + ai_summary: + base_url: "http://127.0.0.1:11434" + model: "gemma3:4b" + grounding: true + +.. autoclass:: searx.ai_summary.SettingsAISummary + :members: + + +Servers that need a password +============================ + +Some servers require authentication -- vLLM and llama.cpp when started with +``--api-key``, a gateway such as LiteLLM, or any LLM server placed behind an +authenticating reverse proxy. Give SearXNG the key with ``api_key``: .. code:: yaml ai_summary: base_url: "http://127.0.0.1:8000" api_key: "sk-..." - model: "llama3.2:3b" + model: "gemma3:4b" -The key is sent in an ``Authorization: Bearer`` header and only to the -``base_url`` above. A user who points the ``ai_summary_server`` preference at -a server of their own never gets the administrator's key; for such a server -the user configures their own key in the ``ai_summary_api_key`` preference. -SearXNG has no indirection for secrets in ``settings.yml``, so the file -holding the key should be readable by the SearXNG process only. +The key is sent as an ``Authorization: Bearer`` header, and only to the server +in ``base_url``. Users who point the ``ai_summary_server`` preference at a +server of their own never receive it; they set their own key in the +``ai_summary_api_key`` preference instead. + +SearXNG has no separate secret store, so the key sits in ``settings.yml`` -- +make sure that file is readable only by the user SearXNG runs as. -The *AI Summary* tab of the preferences is only shown when the plugin is -activated in ``settings.yml`` (``active: true``); an instance that does not -offer AI summaries does not show the tab at all. .. _ai_summary grounding: Grounding ========= -With ``grounding`` (the default) the query **and the top search results** -(title, URL and snippet of at most ``max_context_items`` results) are sent to -the LLM server; without it only the query is sent and the model answers from -its own knowledge. Grounded answers are more accurate and more current, at -the cost of a longer prompt. +Grounding decides *what the model is told*: -Which data leaves the SearXNG instance therefore depends on where the LLM -server runs: with a server on localhost or in the local network nothing leaves -the network, with a hosted server the query and the result snippets are sent -to that provider. This is why the plugin is not activated by default and why -its documentation recommends a local LLM server. Users can opt out of -grounding in their preferences (``ai_summary_grounding``). +``grounding: true`` (the default) + The query **and the top search results** (title, URL and snippet, at most + ``max_context_items`` of them) are sent to the LLM server. Answers reflect + what the search actually found, so they are more accurate and more current. + +``grounding: false`` + Only the query is sent, and the model answers from its own training data. + Faster and cheaper, but the answer can be outdated or invented. + +Users can switch this in their preferences (``ai_summary_grounding``). + +How much this matters for privacy depends entirely on where the LLM server runs. +With a server on localhost or in your own network, nothing leaves that network. +With a hosted server, the query and the result snippets are sent to the +provider. This is the main reason a local server is recommended. + + +Public instances +================ .. attention:: - A user configurable server URL allows any user of the instance to make the - SearXNG server send requests to a URL of their choice (`SSRF`_), and each - summary is real LLM work. This plugin is intended for private instances -- - on a public instance lock the related preferences (:ref:`settings - preferences`): + The plugin is designed for private instances. Read this before enabling it + on a public one. - .. code:: yaml +Two things change on a public instance: - preferences: - lock: - - ai_summary_server - - ai_summary_api_key - - ai_summary_model - - ai_summary_grounding +**Every search costs real work.** A summary is a full LLM inference. Traffic +that was cheap to serve becomes expensive, on hardware you pay for. - Locking ``ai_summary_server`` and ``ai_summary_api_key`` matters most: an - unlocked pair lets any user of the instance make SearXNG send an - ``Authorization`` header of their choosing to a host of their choosing. +**Users can choose the server SearXNG talks to.** The ``ai_summary_server`` +preference makes your instance send requests to an address of the user's +choosing, which is a `SSRF`_ vector: it can be pointed at services inside your +network that are not reachable from outside. +Lock the preferences so that only your configuration is used +(:ref:`settings preferences`): + +.. code:: yaml + + preferences: + lock: + - ai_summary_server + - ai_summary_api_key + - ai_summary_model + - ai_summary_grounding + +Locking ``ai_summary_server`` closes the SSRF vector; locking +``ai_summary_api_key`` stops users making your instance send an +``Authorization`` header of their choosing to a host of their choosing. + + +.. _ai_summary troubleshooting: + +Troubleshooting +=============== + +**No summary appears at all.** + The plugin deliberately stays quiet in several cases: on page two and beyond, + outside the *general* category, and when an engine already answered the query + with an infobox or an instant answer. Try a question-like query on the first + page. Check that the plugin is enabled both in ``settings.yml`` and in your + own preferences. + +**The model list in the preferences is empty.** + SearXNG asks the server for its models (``GET /v1/models``) once at startup. + An empty list means that request failed -- the server was not running yet, + the URL is wrong, or it needs an ``api_key``. The log records the reason; + restart SearXNG after fixing it. + +**The summary box shows an error.** + SearXNG could not reach the LLM server, or the server rejected the request. + Check ``base_url`` from the SearXNG machine, confirm the model name exists on + that server (``ollama list``), and check the SearXNG log. + +**The summary starts, then stops mid-sentence.** + The answer exceeded ``stream_timeout`` (120 s by default). Large models on + CPU are slow; either raise the limit or use a smaller model. + +.. _Ollama: https://ollama.com/ +.. _Ollama library: https://ollama.com/library +.. _OpenAI chat completions API: https://platform.openai.com/docs/api-reference/chat .. _SSRF: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery - -.. autoclass:: searx.ai_summary.SettingsAISummary - :members: diff --git a/docs/dev/plugins/ai_summary.rst b/docs/dev/plugins/ai_summary.rst index 4e488230c..0305121fd 100644 --- a/docs/dev/plugins/ai_summary.rst +++ b/docs/dev/plugins/ai_summary.rst @@ -4,5 +4,134 @@ AI summary ========== +.. sidebar:: Further reading .. + + - :ref:`settings ai_summary` + - :ref:`dev plugin` + - :ref:`result types` + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +The AI summary plugin shows a generated answer above the search results. The +text comes from an LLM server the administrator runs; see :ref:`settings +ai_summary` for how to set one up. + +The interesting part of this plugin is *when* things happen. Generating an +answer takes seconds, and a search engine that waits seconds before painting +anything is a broken search engine. So the plugin never blocks the result +page: it renders an empty box, and the browser fills that box afterwards from a +second, streaming request. + + +Request flow +============ + +.. _ai_summary dataflow: + +.. kernel-render:: DOT + :alt: Data flow between browser, SearXNG and the LLM server + :caption: A search that produces a summary: two requests, not one + + digraph ai_summary { + rankdir=LR; + graph [fontname="sans-serif", ranksep=1.1, nodesep=0.4]; + node [fontname="sans-serif", fontsize=11, shape=box, style="rounded,filled", + fillcolor="#f4f4f4", color="#999999"]; + edge [fontname="sans-serif", fontsize=9, color="#666666"]; + + browser [label="browser\n(simple theme)"]; + searxng [label="SearXNG"]; + engines [label="search engines", fillcolor="#ffffff"]; + llm [label="LLM server\nOllama, vLLM, ...", fillcolor="#ffffff"]; + + browser -> searxng [label=" 1 GET /search"]; + searxng -> engines [label=" 2 query"]; + searxng -> browser [label=" 3 result page,\l empty summary box\l", constraint=false]; + browser -> searxng [label=" 4 POST /ai_summary\l (query + results)\l"]; + searxng -> llm [label=" 5 POST /v1/chat/completions"]; + llm -> searxng [label=" 6 SSE token stream", constraint=false]; + searxng -> browser [label=" 7 NDJSON token stream", constraint=false]; + } + +Steps 1--3 are an ordinary SearXNG search. The plugin's +:py:obj:`post_search ` hook +adds an empty :py:obj:`searx.result_types.AiSummary` placeholder to the answer +area and returns immediately, so the page is not delayed. + +Steps 4--7 happen in the browser after the page has painted. +``client/simple/src/js/plugin/AiSummary.ts`` posts to the ``/ai_summary`` +endpoint, which opens a streaming request to the LLM server and re-emits the +tokens as they arrive. The user sees the answer being written. + +Two format changes happen along the way. The LLM server speaks `SSE`_ +(``data: {...}`` lines, terminated by ``data: [DONE]``), because that is what +the OpenAI chat completions API specifies. SearXNG re-emits that to the +browser as `NDJSON`_ -- one JSON object per line, ``{"delta": "..."}`` for each +chunk of text and a final ``{"done": true}``. NDJSON is used because the +browser reads the body with ``fetch`` and a stream reader, where SSE's +``EventSource`` would be the wrong tool: ``EventSource`` cannot issue a POST. + + +When no summary is generated +============================ + +:py:obj:`post_search ` skips +the placeholder entirely for: + +- page two and beyond -- a summary belongs with the first impression of a query +- anything but the *general* category +- non-HTML output formats (the JSON, CSV and RSS APIs) +- queries where an engine already produced an infobox (wikipedia, wikidata) or + an instant answer (e.g. ddg definitions) +- an empty query, or no LLM server configured + +The infobox rule mirrors what the big engines do: if the query is a lookup of a +well known entity, that entity's own data is a better answer than a generated +paragraph. + + +Where the API key goes +====================== + +The administrator's ``api_key`` is only ever sent to the administrator's +``base_url``. This matters because users may set their own server in the +``ai_summary_server`` preference: without the check, any user of the instance +could point that preference at a host they control and collect the instance's +key from the ``Authorization`` header. Users authenticate to their own server +with their own ``ai_summary_api_key`` preference, which is stored in a cookie +and deliberately excluded from the shareable preferences URL. + +:py:obj:`_server_api_key ` implements +the rule; a URL carrying credentials in its userinfo is rejected outright, +because HTTP clients turn that into an ``Authorization`` header of the user's +choosing. + + +Implementation notes +==================== + +The ``/ai_summary`` route is registered in :py:obj:`searx.webapp`, next to the +favicon proxy, rather than in the plugin's ``init()``. Flask does not allow +``add_url_rule`` after the first request has been handled, and registering it +from a plugin breaks the test suite. + +Requests to the LLM server bypass :py:obj:`searx.network` and are sent with a +plain :py:obj:`httpx.Client`. The outgoing proxy configuration is deliberately +not applied: an LLM server usually sits on localhost or in the local network, +which is exactly what an outgoing proxy is configured to avoid. + +The streaming response uses ``direct_passthrough``, so the generator must yield +``bytes`` -- werkzeug asserts on ``str``, and the Flask test client does not +catch it. + +.. _SSE: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events +.. _NDJSON: https://github.com/ndjson/ndjson-spec + +Reference +========= + .. automodule:: searx.plugins.ai_summary :members: diff --git a/searx/plugins/ai_summary.py b/searx/plugins/ai_summary.py index 02767eeab..b46c846bf 100644 --- a/searx/plugins/ai_summary.py +++ b/searx/plugins/ai_summary.py @@ -1,64 +1,22 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -"""Plugin that displays an AI generated summary of the search query at the top -of the result page. The summary is generated by a (local) LLM server that -implements the `OpenAI chat completions API`_ -- e.g. `Ollama`_, vLLM, -llama.cpp, LM Studio or Hugging Face TGI. +"""Implementation of the AI summary plugin, which shows a generated answer above +the search results. The answer comes from an LLM server that implements the +`OpenAI chat completions API`_ (Ollama, vLLM, llama.cpp, LM Studio, Hugging Face +TGI, ...) and that the administrator runs. -The LLM server URL and the model are configured by the user in the *AI -Summary* tab of the preferences (``ai_summary_server``, ``ai_summary_model``); -the administrator can configure instance wide defaults in the ``ai_summary:`` -section and lock the preferences via :ref:`settings preferences`. +- :ref:`ai_summary plugin` describes the design and the request flow. +- :ref:`settings ai_summary` describes how to configure it. -.. attention:: +This module holds the plugin itself and the ``/ai_summary`` endpoint +(:py:obj:`ai_summary_view`, registered in :py:obj:`searx.webapp`). The endpoint +streams the answer to the browser, so that the result page is never delayed by +the LLM; :py:obj:`SXNGPlugin.post_search` only adds an empty +:py:obj:`searx.result_types.AiSummary` placeholder for the client to fill. - A user configurable server URL allows any user of the instance to make the - SearXNG server send requests to a URL of their choice (`SSRF`_), and each - summary is real LLM work. This plugin is intended for private instances -- - on a public instance, lock the ``ai_summary_server``, ``ai_summary_model`` - and ``ai_summary_grounding`` preferences and configure the ``ai_summary:`` - section instead. +Settings of the ``ai_summary:`` section are defined in +:py:obj:`searx.ai_summary.SettingsAISummary`. -The result page is never delayed by this plugin: it only places an empty -placeholder (:py:obj:`searx.result_types.AiSummary`) in the answer area, which -is filled asynchronously by the client (``client/simple/src/js/plugin/ -AiSummary.ts``) from the ``/ai_summary`` endpoint (registered in -:py:obj:`searx.webapp`). The endpoint re-emits the SSE token stream of the -LLM server's ``/v1/chat/completions`` to the client as `NDJSON`_. - -A summary is only generated on the first page of a *general* search and only -if no engine has contributed an infobox (e.g. wikipedia / wikidata) or an -instant answer (e.g. ddg definitions) -- in these cases the query is most -likely a lookup of a well known term that is already answered. - -The requests to the LLM server are sent directly (not via -:py:obj:`searx.network`), an outgoing proxy configuration is deliberately not -applied to reach an LLM server in the local network. - -A server that requires authentication (e.g. vLLM or llama.cpp started with -``--api-key``, or an LLM server behind an authenticating reverse proxy) is -configured with an ``api_key``. The administrator's key is only sent to the -server in ``base_url``, never to a server a user configured; for their own -server users configure their own key in the ``ai_summary_api_key`` preference -(:py:obj:`_server_api_key`). - -Configuration of the defaults (:py:obj:`searx.ai_summary.SettingsAISummary`): - -.. code:: yaml - - ai_summary: - base_url: "http://127.0.0.1:11434" - model: "llama3.2:3b" - -.. code:: yaml - - plugins: - searx.plugins.ai_summary.SXNGPlugin: - active: false - -.. _Ollama: https://ollama.com/ .. _OpenAI chat completions API: https://platform.openai.com/docs/api-reference/chat -.. _NDJSON: https://github.com/ndjson/ndjson-spec -.. _SSRF: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery """ import typing as t