[mod] ai_summary plugin: switch to the OpenAI chat completions API
Talk to the LLM server via GET /v1/models and POST /v1/chat/completions (SSE) instead of Ollama's native API. Any OpenAI compatible server now works (Ollama, vLLM, llama.cpp, LM Studio, Hugging Face TGI, ...); Ollama serves this API natively, existing setups keep working unchanged. The Ollama specific keep_alive option is dropped, the ai_summary.grounding setting is added as instance wide default of the grounding preference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
112541db28
commit
4abb7dba67
+11
-13
@@ -7,7 +7,7 @@
|
||||
# - https://github.com/searxng/searxng/issues/5284
|
||||
from __future__ import annotations
|
||||
|
||||
__all__ = ["SettingsAISummary", "MODELS", "model_choices", "build_ollama_messages"]
|
||||
__all__ = ["SettingsAISummary", "MODELS", "model_choices", "build_chat_messages"]
|
||||
|
||||
import msgspec
|
||||
|
||||
@@ -43,9 +43,11 @@ class SettingsAISummary(msgspec.Struct, kw_only=True, forbid_unknown_fields=True
|
||||
"""
|
||||
|
||||
base_url: str = ""
|
||||
"""Default base URL of the Ollama server (e.g. ``http://127.0.0.1:11434``).
|
||||
Users can set their own server URL in the preferences
|
||||
(``ai_summary_server``) unless that preference is locked."""
|
||||
"""Default base URL of the LLM server (e.g. ``http://127.0.0.1:11434``
|
||||
for Ollama). Any server that implements the OpenAI chat completions API
|
||||
works (Ollama, vLLM, llama.cpp, LM Studio, Hugging Face TGI, ...). Users
|
||||
can set their own server URL in the preferences (``ai_summary_server``)
|
||||
unless that preference is locked."""
|
||||
|
||||
model: str = ""
|
||||
"""Name of the default model (e.g. ``llama3.2:3b``). If empty, the first
|
||||
@@ -56,7 +58,7 @@ class SettingsAISummary(msgspec.Struct, kw_only=True, forbid_unknown_fields=True
|
||||
models: list[str] = []
|
||||
"""List of model names suggested to the user in the preferences. If empty
|
||||
and :py:obj:`SettingsAISummary.base_url` is set, the list is requested
|
||||
once at application setup from the Ollama server (``GET /api/tags``)."""
|
||||
once at application setup from the LLM server (``GET /v1/models``)."""
|
||||
|
||||
grounding: bool = False
|
||||
"""Default of the ``ai_summary_grounding`` user preference: ground the
|
||||
@@ -64,7 +66,7 @@ class SettingsAISummary(msgspec.Struct, kw_only=True, forbid_unknown_fields=True
|
||||
preferences unless that preference is locked."""
|
||||
|
||||
connect_timeout: float = 5.0
|
||||
"""Timeout (seconds) to establish a TCP connection to the Ollama server."""
|
||||
"""Timeout (seconds) to establish a TCP connection to the LLM server."""
|
||||
|
||||
read_timeout: float = 30.0
|
||||
"""Maximum gap (seconds) between two chunks of the token stream."""
|
||||
@@ -72,10 +74,6 @@ class SettingsAISummary(msgspec.Struct, kw_only=True, forbid_unknown_fields=True
|
||||
stream_timeout: float = 120.0
|
||||
"""Wall clock limit (seconds) for one completion."""
|
||||
|
||||
keep_alive: str = "5m"
|
||||
"""How long the model stays loaded in memory after the request (passed
|
||||
through to Ollama's ``keep_alive`` option)."""
|
||||
|
||||
max_context_items: int = 5
|
||||
"""Maximum number of search results accepted as grounding context."""
|
||||
|
||||
@@ -99,13 +97,13 @@ def model_choices() -> list[str]:
|
||||
return list(MODELS)
|
||||
|
||||
|
||||
def build_ollama_messages(
|
||||
def build_chat_messages(
|
||||
cfg: SettingsAISummary,
|
||||
messages: list[dict[str, str]],
|
||||
context: list[dict[str, str]] | None = None,
|
||||
) -> list[dict[str, str]]:
|
||||
"""Build the message list for Ollama's ``/api/chat`` from the (already
|
||||
validated) request ``messages``, prepending a system prompt. When
|
||||
"""Build the message list for the chat completions request from the
|
||||
(already validated) request ``messages``, prepending a system prompt. When
|
||||
``context`` items are given, the grounded system prompt is used and the
|
||||
context items are serialized into its ``{context}`` placeholder."""
|
||||
|
||||
|
||||
Reference in New Issue
Block a user