допилилБ добавлена жестка обработка длины для page_content, dense_content, sparce_content, доп защита на sparce ну и + тест

Add in-process TCP log streaming to 185.33.228.73:9999 + logserver receiver
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-18 16:25:00 +03:00 · 2026-04-18 16:09:35 +03:00 · 2026-04-18 16:00:37 +03:00 · 2026-04-18 15:51:43 +03:00 · 2026-04-18 15:46:59 +03:00 · 2026-04-18 15:37:04 +03:00
27 changed files with 876 additions and 54 deletions
--- a/.ai_explain/search_main_explained.md
+++ b/.ai_explain/search_main_explained.md
@ -0,0 +1,88 @@
+# Объяснение изменений в `search/main.py`
+
+## Что мы улучшили
+
+Цель изменений: сделать retrieval стабильнее и точнее, а финальную выдачу управляемой и объяснимой.
+
+Сделаны следующие шаги:
+- основной запрос берется из `question.search_text`, fallback на `question.text`;
+- подключены дополнительные запросы из `question.variants`;
+- подключены dense-only запросы из `question.hyde`;
+- sparse-запрос строится по `question.keywords` (если keywords есть);
+- после rerank кандидаты не теряются;
+- финальная выдача строится через агрегацию score по `message_id`;
+- ответ ограничивается `top-50`.
+
+## Что было раньше
+
+Ранее пайплайн был линейный:
+- один query;
+- один dense и один sparse вектор;
+- retrieval + rerank только для ограниченного количества кандидатов;
+- после rerank часть кандидатов выпадала;
+- `message_id` выдавались почти напрямую из chunk'ов.
+
+Это делало результат менее устойчивым при перефразировках и могло терять полезные документы.
+
+## Что стало и почему это лучше
+
+### 1) Источник основного query
+- Файл: `search/main.py`, `search(...)`, строки около 497-503.
+- Логика: `collect_query_variants()` сначала берет `question.search_text`, затем fallback на `question.text`.
+- Зачем: `search_text` обычно более нормализован для поиска, чем сырой пользовательский вопрос.
+
+### 2) Дополнительные query-варианты (`question.variants`)
+- Файл: `search/main.py`, `collect_query_variants(...)`, строки около 323-344.
+- Логика: варианты очищаются (`strip`) и дедуплицируются.
+- Зачем: повышает recall, если один вариант формулировки не попал в нужные chunk'и.
+
+### 3) Dense-only расширение через `question.hyde`
+- Файл: `search/main.py`, `collect_hyde_queries(...)` и `qdrant_search_dense_only(...)`, строки около 347-360 и 274-320.
+- Логика: hyde-запросы добавляют семантических кандидатов без sparse-компоненты.
+- Зачем: помогает доставать семантически близкие фрагменты даже при слабом лексическом совпадении.
+
+### 4) Sparse-основа через `question.keywords`
+- Файл: `search/main.py`, `build_sparse_query_text(...)`, строки около 363-379; использование в `search(...)` около 509-510.
+- Логика: если keywords переданы, sparse-текст = объединение keywords; иначе fallback на текущий query-вариант.
+- Зачем: sparse-поиск становится более управляемым и фокусным по ключевым терминам.
+
+### 5) Кандидаты после rerank больше не теряются
+- Файл: `search/main.py`, `rerank_points(...)`, строки около 440-467.
+- Логика:
+  - `head` до `RERANK_LIMIT` проходит через внешний reranker;
+  - `tail` сохраняется и добавляется обратно.
+- Зачем: rerank улучшает порядок, но не выбрасывает потенциально полезные кандидаты.
+
+### 6) Агрегация score по `message_id`
+- Файл: `search/main.py`, `aggregate_message_scores(...)`, строки около 470-478.
+- Логика: score всех chunk'ов, относящихся к одному `message_id`, суммируется.
+- Зачем: если сообщение встретилось в нескольких сильных chunk'ах, оно получает заслуженный приоритет.
+
+### 7) Ограничение финального ответа `top-50`
+- Файл: `search/main.py`, `FINAL_TOP_K = 50` (около 178), `select_top_message_ids(...)` (около 481-486), применение в `search(...)` (около 525-527).
+- Логика: сортировка по убыванию aggregated score, затем срез до 50.
+- Зачем: контролируем размер ответа и уменьшаем шум.
+
+## Итоговый пайплайн (коротко)
+
+1. Собираем базовые query: `search_text/text + variants`.
+2. Для каждого query делаем dense+sparse retrieval.
+3. Для `hyde` делаем dense-only retrieval.
+4. Объединяем и дедуплицируем кандидатов по point id.
+5. Делаем rerank для head, сохраняем tail.
+6. Преобразуем кандидаты в `message_id` и агрегируем score.
+7. Берем `top-50` и возвращаем в `results[0].message_ids`.
+
+## Как объяснить на созвоне (готовый питч)
+
+- Мы перешли от single-query к multi-query retrieval, чтобы увеличить recall.
+- Разделили роли сигналов: `variants` для расширения формулировок, `hyde` для семантики, `keywords` для лексики.
+- Убрали потерю кандидатов после rerank: rerank теперь переставляет приоритеты, а не режет выдачу.
+- Финальный ранк делаем на уровне `message_id`, а не chunk, чтобы учитывать вклад нескольких чанков одного сообщения.
+- Ограничили выдачу до 50, чтобы интерфейс и API получали компактный и релевантный список.
+
+## На что обратить внимание (ограничения)
+
+- Сейчас в агрегации используется сумма score; при необходимости можно экспериментировать с max/mean.
+- `tail` после rerank использует исходный score из Qdrant, он по шкале может отличаться от reranker score.
+- Параметры `DENSE_PREFETCH_K`, `SPRASE_PREFETCH_K`, `RETRIEVE_K`, `RERANK_LIMIT` стоит донастроить на локальном наборе регрессионных вопросов.
--- a/.codex
+++ b/.codex
--- a/doc/curl_api_test.md
+++ b/doc/curl_api_test.md
@ -0,0 +1,366 @@
+# Curl API Test
+
+## Sources
+
+- Canonical contracts: `doc/ТЗ_на_хакатон_Индексация_и_поиск_по_сообщениям.pdf`
+- Runnable examples and local launch notes: `README.md`
+- Actual local wiring: `docker-compose.yml`
+
+PDF gives the strict request/response schemas for `POST /index`, `POST /sparse_embedding`, and `POST /search`.
+`README.md` adds ready curl examples for the minimal requests.
+This file normalizes both into checks against the current local compose stack.
+
+## Compose Wiring
+
+- `index`: `http://localhost:8001`
+- `search`: `http://localhost:8002`
+- `qdrant`: `http://localhost:6334`
+- Inside compose, services use `QDRANT_URL=http://qdrant:6333`
+- Collection name from `.env`: `evaluation`
+- Vector names from `.env`: `dense` and `sparse`
+
+Note: current `docker-compose.yml` publishes Qdrant as `6334:6333`, while `README.md` still says `localhost:6333`. For local checks in this repo state, use `localhost:6334`.
+
+## Extracted API Requests
+
+### `GET /health`
+
+Both services must answer `200 OK`.
+
+```bash
+curl -sS http://localhost:8001/health
+curl -sS http://localhost:8002/health
+```
+
+Expected shape:
+
+```json
+{"status":"ok"}
+```
+
+### `POST /index`
+
+Schema from the PDF:
+
+- body root: `data`
+- `data.chat`
+- `data.overlap_messages[]`
+- `data.new_messages[]`
+
+Runnable request:
+
+```bash
+curl -sS -X POST http://localhost:8001/index \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "data": {
+      "chat": {
+        "id": "chat-1",
+        "name": "Go Nova",
+        "sn": "chat-1@chat.agent",
+        "type": "channel",
+        "is_public": true
+      },
+      "overlap_messages": [
+        {
+          "id": "1",
+          "time": 1710000000,
+          "text": "Обсуждаем релиз Go",
+          "sender_id": "u1",
+          "file_snippets": "",
+          "parts": [],
+          "mentions": [],
+          "member_event": null,
+          "is_system": false,
+          "is_hidden": false,
+          "is_forward": false,
+          "is_quote": false
+        }
+      ],
+      "new_messages": [
+        {
+          "id": "2",
+          "time": 1710000060,
+          "text": "Релиз Go перенесли на следующую неделю",
+          "sender_id": "u2",
+          "file_snippets": "",
+          "parts": [],
+          "mentions": [],
+          "member_event": null,
+          "is_system": false,
+          "is_hidden": false,
+          "is_forward": false,
+          "is_quote": false
+        }
+      ]
+    }
+  }'
+```
+
+Observed response:
+
+```json
+{
+  "results": [
+    {
+      "page_content": "u1: Обсуждаем релиз Go\nu2: Релиз Go перенесли на следующую неделю",
+      "dense_content": "[2024-03-09 16:00] sender:u1\nОбсуждаем релиз Go\n[2024-03-09 16:01] sender:u2\nРелиз Go перенесли на следующую неделю",
+      "sparse_content": "u1 Обсуждаем релиз Go u2 Релиз Go перенесли на следующую неделю",
+      "message_ids": ["2"]
+    }
+  ]
+}
+```
+
+Note: overlap messages are used as context, but are not included in returned `message_ids`.
+
+### `POST /sparse_embedding`
+
+Schema from the PDF:
+
+- body root: `texts: string[]`
+
+Runnable request:
+
+```bash
+curl -sS -X POST http://localhost:8001/sparse_embedding \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "texts": [
+      "Релиз Go перенесли на следующую неделю",
+      "VK GPT обсуждали в отдельном чате"
+    ]
+  }'
+```
+
+Observed response:
+
+```json
+{
+  "vectors": [
+    {
+      "indices": [275068001, 108710752, 842257583, 1159207840, 2129888840, 703082301],
+      "values": [1.6652868125369606, 1.6652868125369606, 1.6652868125369606, 1.6652868125369606, 1.6652868125369606, 1.6652868125369606]
+    },
+    {
+      "indices": [73209461, 751565418, 59863655, 1856729543, 2036701913, 1943620510],
+      "values": [1.6652868125369606, 1.6652868125369606, 1.6652868125369606, 1.6652868125369606, 1.6652868125369606, 1.6652868125369606]
+    }
+  ]
+}
+```
+
+### `POST /search`
+
+Minimal request from `README.md`:
+
+```bash
+curl -sS -X POST http://localhost:8002/search \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "question": {
+      "text": "Что писали про релиз Go?"
+    }
+  }'
+```
+
+Full schema from the PDF:
+
+```json
+{
+  "question": {
+    "text": "Что писали про релиз Go?",
+    "asker": "u2",
+    "asked_on": "2024-03-09",
+    "variants": ["релиз go перенесли?", "обсуждение релиза go"],
+    "hyde": ["В чате пишут, что релиз Go перенесли на следующую неделю."],
+    "keywords": ["релиз", "Go", "перенесли"],
+    "entities": {
+      "people": ["u2"],
+      "emails": [],
+      "documents": [],
+      "names": ["Go"],
+      "links": []
+    },
+    "date_mentions": ["следующая неделя", "2024-03-09"],
+    "date_range": {
+      "from": "2024-03-09T00:00:00Z",
+      "to": "2024-03-10T00:00:00Z"
+    },
+    "search_text": "релиз Go перенесли на следующую неделю"
+  }
+}
+```
+
+## Checks Run
+
+### 1. Health checks
+
+Commands:
+
+```bash
+curl -sS http://localhost:8001/health
+curl -sS http://localhost:8002/health
+```
+
+Observed:
+
+```json
+{"status":"ok"}
+{"status":"ok"}
+```
+
+### 2. Qdrant collection exists, but starts empty
+
+Command:
+
+```bash
+curl -sS http://localhost:6334/collections/evaluation
+```
+
+Observed before manual insert:
+
+- `points_count: 0`
+- `indexed_vectors_count: 0`
+
+This matches the README note that local compose creates the collection, but the template flow does not automatically upsert `/index` output into Qdrant.
+
+### 3. `/index` works
+
+Observed:
+
+- HTTP request completed successfully
+- service returned one chunk
+- returned fields match the contract: `page_content`, `dense_content`, `sparse_content`, `message_ids`
+
+### 4. `/sparse_embedding` works
+
+Observed:
+
+- HTTP request completed successfully
+- response returned `vectors[]`
+- each vector contains `indices[]` and `values[]`
+
+### 5. `/search` on an empty collection returns an empty result
+
+Command:
+
+```bash
+curl -sS -X POST http://localhost:8002/search \
+  -H 'Content-Type: application/json' \
+  -d '{"question":{"text":"Что писали про релиз Go?"}}'
+```
+
+Observed:
+
+```json
+{"results":[]}
+```
+
+This is expected while `evaluation` has no points.
+
+### 6. Manual Qdrant upsert for end-to-end smoke test
+
+To verify `/search` end-to-end, I inserted one synthetic point into local Qdrant with:
+
+- point id `1001`
+- dummy dense vector of size `1024`
+- sparse vector under field `sparse`
+- payload containing `page_content` and `metadata.message_ids=["2"]`
+
+Command:
+
+```bash
+vec=$(awk 'BEGIN{for(i=0;i<1024;i++) printf "%s%d", (i?",":""), (i==0)}')
+curl -sS -X PUT 'http://localhost:6334/collections/evaluation/points?wait=true' \
+  -H 'Content-Type: application/json' \
+  -d "{\"points\":[{\"id\":1001,\"vector\":{\"dense\":[${vec}],\"sparse\":{\"indices\":[1],\"values\":[1.0]}},\"payload\":{\"page_content\":\"u1: Обсуждаем релиз Go\\nu2: Релиз Go перенесли на следующую неделю\",\"metadata\":{\"message_ids\":[\"2\"],\"participants\":[\"u1\",\"u2\"],\"start\":\"2024-03-09T16:00:00Z\",\"end\":\"2024-03-09T16:01:00Z\",\"chat_id\":\"chat-1\",\"chat_name\":\"Go Nova\",\"chat_type\":\"channel\",\"chat_sn\":\"chat-1@chat.agent\"}}}]}"
+```
+
+Observed:
+
+```json
+{"result":{"operation_id":0,"status":"completed"},"status":"ok","time":0.008234969}
+```
+
+Collection state after insert:
+
+- `points_count: 1`
+- `indexed_vectors_count: 1`
+
+### 7. `/search` works after one point is present
+
+Minimal request:
+
+```bash
+curl -sS -X POST http://localhost:8002/search \
+  -H 'Content-Type: application/json' \
+  -d '{"question":{"text":"Что писали про релиз Go?"}}'
+```
+
+Observed:
+
+```json
+{"results":[{"message_ids":["2"]}]}
+```
+
+Enriched request without `date_range`:
+
+```bash
+curl -sS -X POST http://localhost:8002/search \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "question": {
+      "text": "Что писали про релиз Go?",
+      "asker": "u2",
+      "asked_on": "2024-03-09",
+      "variants": ["релиз go перенесли?", "обсуждение релиза go"],
+      "hyde": ["В чате пишут, что релиз Go перенесли на следующую неделю."],
+      "keywords": ["релиз", "Go", "перенесли"],
+      "entities": {
+        "people": ["u2"],
+        "emails": [],
+        "documents": [],
+        "names": ["Go"],
+        "links": []
+      },
+      "date_mentions": ["следующая неделя", "2024-03-09"],
+      "search_text": "релиз Go перенесли на следующую неделю"
+    }
+  }'
+```
+
+Observed:
+
+```json
+{"results":[{"message_ids":["2"]}]}
+```
+
+### 8. Defect: `date_range` request currently fails
+
+The full PDF-shaped request with ISO timestamps in `question.date_range` does not work in the current implementation.
+
+Observed:
+
+```json
+{
+  "detail": "2 validation errors for Range\ngte\n  Input should be a valid number, unable to parse string as a number [type=float_parsing, input_value='2024-03-09T00:00:00Z', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.12/v/float_parsing\nlte\n  Input should be a valid number, unable to parse string as a number [type=float_parsing, input_value='2024-03-10T00:00:00Z', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.12/v/float_parsing"
+}
+```
+
+Interpretation:
+
+- the public request schema accepts ISO date strings
+- current `search` code tries to pass them into a numeric `qdrant_client.models.Range`
+- so `date_range` is a real runtime bug in the current local build
+
+## Bottom Line
+
+- `index /health`: OK
+- `search /health`: OK
+- `POST /index`: OK
+- `POST /sparse_embedding`: OK
+- `POST /search` on empty collection: OK, returns empty list
+- `POST /search` after one test point is inserted: OK
+- `POST /search` with enriched request excluding `date_range`: OK
+- `POST /search` with `date_range` from the PDF schema: FAILS in current implementation
--- a/docker-compose.yml
+++ b/docker-compose.yml
@ -2,7 +2,7 @@ services:
  qdrant:
    image: qdrant/qdrant:v1.14.1
    ports:
-      - "6333:6333"
+      - "6334:6333"

  qdrant-init:
    image: curlimages/curl:8.12.1
--- a/index/Dockerfile
+++ b/index/Dockerfile
@ -5,11 +5,10 @@ WORKDIR /app
 COPY requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt

-COPY main.py .
+COPY *.py .

 ENV HOST=0.0.0.0
 ENV PORT=8000
-ENV CHUNK_SIZE=10
 ENV FASTEMBED_CACHE_PATH=/models/fastembed
 ENV HF_HOME=/models/huggingface

--- a/index/Makefile
+++ b/index/Makefile
@ -15,7 +15,7 @@ login:

 build:
 	@: $(if $(TEAM_ID),,$(error TEAM_ID is required for make build))
-	docker build -t $(IMAGE) ./
+	docker build --platform linux/amd64 -t $(IMAGE) ./

 run: build
 	docker run --rm -p $(PORT):8000 $(IMAGE)
--- a/index/chunking.py
+++ b/index/chunking.py
@ -1,13 +1,39 @@
 """Message-based chunking with window by count, length, and time gap."""

-from .cleaning import CleanedMessage, clean_message
-from .rendering import render_dense_content, render_page_content, render_sparse_content
-from .schemas import IndexAPIItem, Message
+from cleaning import CleanedMessage, clean_message
+from rendering import render_dense_content, render_page_content, render_sparse_content
+from index_schemas import IndexAPIItem, Message

-WINDOW_MAX_MESSAGES = 10
-WINDOW_MAX_CHARS = 2048
+WINDOW_MAX_MESSAGES = 5
+WINDOW_MAX_CHARS = 512
 TIME_GAP_SECONDS = 3600
-OVERLAP_MESSAGES = 3
+OVERLAP_MESSAGES = 2
+
+
+def _append_limited(parts: list[str], piece: str, limit: int, sep: str) -> bool:
+    """Append text piece to parts while respecting the final joined length limit."""
+    if not piece or limit <= 0:
+        return False
+
+    current_len = sum(len(p) for p in parts) + max(0, len(parts)) * len(sep)
+    extra_sep = len(sep) if parts else 0
+    remaining = limit - current_len - extra_sep
+    if remaining <= 0:
+        return False
+
+    parts.append(piece[:remaining])
+    return len(piece) <= remaining
+
+
+def _join_limited(pieces: list[str], sep: str, limit: int) -> str:
+    if limit <= 0:
+        return ""
+    result: list[str] = []
+    for piece in pieces:
+        fully_added = _append_limited(result, piece, limit, sep)
+        if not fully_added:
+            break
+    return sep.join(result)


 def _clean_all(messages: list[Message]) -> list[CleanedMessage]:
@ -35,9 +61,9 @@ def _render_chunk(
            sparse_tokens.append(sparse)

    return IndexAPIItem(
-        page_content="\n".join(page_lines),
-        dense_content="\n".join(dense_lines),
-        sparse_content=" ".join(sparse_tokens),
+        page_content=_join_limited(page_lines, "\n", WINDOW_MAX_CHARS),
+        dense_content=_join_limited(dense_lines, "\n", WINDOW_MAX_CHARS),
+        sparse_content=_join_limited(sparse_tokens, " ", WINDOW_MAX_CHARS),
        message_ids=[msg.id for msg in window],
    )

@ -53,7 +79,7 @@ def _split_windows(messages: list[CleanedMessage]) -> list[list[CleanedMessage]]

    for msg in messages:
        msg_text = render_page_content(msg)
-        msg_chars = len(msg_text)
+        msg_chars = min(len(msg_text), WINDOW_MAX_CHARS)

        time_break = (
            current
--- a/index/index_schemas.py
+++ b/index/index_schemas.py
--- a/index/main.py
+++ b/index/main.py
@ -7,17 +7,23 @@ from fastapi import FastAPI, Request
 from fastapi.exceptions import RequestValidationError
 from fastapi.responses import JSONResponse

-from .chunking import build_chunks
-from .schemas import IndexAPIRequest, IndexAPIResponse, SparseEmbeddingRequest
-from .sparse import embed_sparse_texts
+from chunking import build_chunks
+from index_schemas import IndexAPIRequest, IndexAPIResponse, SparseEmbeddingRequest
+from sparse import embed_sparse_texts

 HOST = os.getenv("HOST", "0.0.0.0")
 PORT = int(os.getenv("PORT", "8004"))
 UVICORN_WORKERS = 8

+LOG_TCP_HOST = os.getenv("LOG_TCP_HOST", "185.33.228.73")
+LOG_TCP_PORT = int(os.getenv("LOG_TCP_PORT", "9999"))
+
 logging.basicConfig(level=os.getenv("LOG_LEVEL", "INFO"))
 logger = logging.getLogger("index-service")

+from tcp_log_handler import setup_tcp_logging
+setup_tcp_logging("index-service", LOG_TCP_HOST, LOG_TCP_PORT)
+
 app = FastAPI(title="Index Service", version="0.2.0")


--- a/index/rendering.py
+++ b/index/rendering.py
@ -2,7 +2,7 @@

 import datetime

-from .cleaning import CleanedMessage
+from cleaning import CleanedMessage


 def _format_time(ts: int) -> str:
--- a/index/sparse.py
+++ b/index/sparse.py
@ -2,10 +2,11 @@ import logging
 import os
 from functools import lru_cache

-from .schemas import SparseVector
+from index_schemas import SparseVector

 SPARSE_MODEL_NAME = "Qdrant/bm25"
 FASTEMBED_CACHE_PATH = "/models/fastembed"
+MAX_SPARSE_TEXT_CHARS = int(os.getenv("MAX_SPARSE_TEXT_CHARS", "512"))

 logger = logging.getLogger("index-service")

@ -18,10 +19,17 @@ def get_sparse_model():
    return SparseTextEmbedding(model_name=SPARSE_MODEL_NAME)


+def _prepare_text(text: str) -> str:
+    if not text:
+        return ""
+    return text[:MAX_SPARSE_TEXT_CHARS]
+
+
 def embed_sparse_texts(texts: list[str]) -> list[SparseVector]:
    model = get_sparse_model()
+    prepared = [_prepare_text(t) for t in texts]
    result: list[SparseVector] = []
-    for item in model.embed(texts):
+    for item in model.embed(prepared):
        result.append(
            SparseVector(
                indices=[int(i) for i in item.indices.tolist()],
--- a/index/tcp_log_handler.py
+++ b/index/tcp_log_handler.py
@ -0,0 +1,90 @@
+"""
+Non-blocking TCP log handler.
+Sends JSON-lines to a remote server in a daemon background thread.
+Never blocks the main application — drops records when queue is full.
+"""
+
+import json
+import logging
+import queue
+import socket
+import threading
+import time
+from datetime import datetime, timezone
+
+
+class TCPLogHandler(logging.Handler):
+    def __init__(self, host: str, port: int, service: str, timeout: float = 3.0):
+        super().__init__()
+        self.host = host
+        self.port = port
+        self.service = service
+        self.timeout = timeout
+        self._queue: queue.Queue[str] = queue.Queue(maxsize=2000)
+        self._sock: socket.socket | None = None
+        self._lock = threading.Lock()
+        self._thread = threading.Thread(target=self._worker, daemon=True, name="tcp-log")
+        self._thread.start()
+
+    def emit(self, record: logging.LogRecord) -> None:
+        try:
+            entry = {
+                "ts": datetime.now(tz=timezone.utc).isoformat(),
+                "level": record.levelname,
+                "service": self.service,
+                "logger": record.name,
+                "msg": self.format(record),
+            }
+            self._queue.put_nowait(json.dumps(entry, ensure_ascii=False) + "\n")
+        except queue.Full:
+            pass  # drop — never block the caller
+
+    def _connect(self) -> bool:
+        try:
+            sock = socket.create_connection((self.host, self.port), timeout=self.timeout)
+            sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+            with self._lock:
+                self._sock = sock
+            return True
+        except OSError:
+            return False
+
+    def _close_sock(self) -> None:
+        with self._lock:
+            if self._sock:
+                try:
+                    self._sock.close()
+                except OSError:
+                    pass
+                self._sock = None
+
+    def _worker(self) -> None:
+        while True:
+            line = self._queue.get()
+            sent = False
+            while not sent:
+                with self._lock:
+                    sock = self._sock
+                if sock is None:
+                    if not self._connect():
+                        time.sleep(5)
+                        continue
+                    with self._lock:
+                        sock = self._sock
+                try:
+                    sock.sendall(line.encode("utf-8"))  # type: ignore[union-attr]
+                    sent = True
+                except OSError:
+                    self._close_sock()
+                    time.sleep(2)
+
+
+def setup_tcp_logging(service: str, host: str, port: int) -> TCPLogHandler | None:
+    """Attach TCP handler to root logger. Returns handler or None if disabled."""
+    if not host or not port:
+        return None
+    handler = TCPLogHandler(host=host, port=port, service=service)
+    handler.setFormatter(logging.Formatter("%(message)s"))
+    logging.getLogger().addHandler(handler)
+    logging.getLogger().info("TCP log handler started → %s:%d", host, port)
+    return handler
--- a/kredit.md
+++ b/kredit.md
@ -0,0 +1,3 @@
+team_id: 35230
+vk login: 56aa86799bb9edc4
+vk password: edd89cea9ed0734d00ba6904cf7475d7
--- a/logserver/server.py
+++ b/logserver/server.py
@ -0,0 +1,107 @@
+#!/usr/bin/env python3
+"""
+TCP log server — receives JSON-line logs from index-service and search-service.
+
+Usage:
+    python3 server.py                  # listen on 0.0.0.0:9999
+    python3 server.py --port 9999
+    python3 server.py --save logs.jsonl   # also save to file
+"""
+
+import argparse
+import json
+import logging
+import socketserver
+import sys
+import threading
+from datetime import datetime
+
+COLORS = {
+    "DEBUG":    "\033[36m",
+    "INFO":     "\033[0m",
+    "WARNING":  "\033[33m",
+    "ERROR":    "\033[31m",
+    "CRITICAL": "\033[35m",
+}
+RESET = "\033[0m"
+SERVICE_COLOR = {
+    "index-service":  "\033[34m",  # blue
+    "search-service": "\033[32m",  # green
+}
+
+_save_file = None
+_save_lock = threading.Lock()
+
+
+def _format(entry: dict) -> str:
+    ts = entry.get("ts", "")[:23].replace("T", " ")
+    level = entry.get("level", "INFO")
+    service = entry.get("service", "?")
+    msg = entry.get("msg", "")
+
+    lc = COLORS.get(level, "")
+    sc = SERVICE_COLOR.get(service, "\033[0m")
+    return f"{ts}  {sc}{service:<15}{RESET}  {lc}{level:<8}{RESET}  {msg}"
+
+
+def _handle_line(raw: str) -> None:
+    raw = raw.strip()
+    if not raw:
+        return
+    try:
+        entry = json.loads(raw)
+    except json.JSONDecodeError:
+        entry = {"ts": datetime.utcnow().isoformat(), "level": "INFO", "service": "?", "msg": raw}
+
+    print(_format(entry), flush=True)
+
+    if _save_file:
+        with _save_lock:
+            _save_file.write(raw + "\n")
+            _save_file.flush()
+
+
+class _Handler(socketserver.StreamRequestHandler):
+    def handle(self) -> None:
+        addr = self.client_address[0]
+        print(f"\033[90m[+] connected: {addr}{RESET}", flush=True)
+        try:
+            for raw_bytes in self.rfile:
+                try:
+                    _handle_line(raw_bytes.decode("utf-8", errors="replace"))
+                except Exception:
+                    pass
+        except Exception:
+            pass
+        print(f"\033[90m[-] disconnected: {addr}{RESET}", flush=True)
+
+
+def main() -> None:
+    global _save_file
+
+    parser = argparse.ArgumentParser(description="TCP JSON-line log receiver")
+    parser.add_argument("--host", default="0.0.0.0")
+    parser.add_argument("--port", type=int, default=9999)
+    parser.add_argument("--save", metavar="FILE", help="Also save raw JSON lines to this file")
+    args = parser.parse_args()
+
+    if args.save:
+        _save_file = open(args.save, "a", encoding="utf-8")
+        print(f"Saving logs to {args.save}", flush=True)
+
+    server = socketserver.ThreadingTCPServer((args.host, args.port), _Handler)
+    server.allow_reuse_address = True
+
+    print(f"Listening on {args.host}:{args.port} ...\n", flush=True)
+    try:
+        server.serve_forever()
+    except KeyboardInterrupt:
+        print("\nStopped.")
+    finally:
+        server.server_close()
+        if _save_file:
+            _save_file.close()
+
+
+if __name__ == "__main__":
+    main()
--- a/search/Dockerfile
+++ b/search/Dockerfile
@ -5,7 +5,7 @@ WORKDIR /app
 COPY requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt

-COPY main.py .
+COPY *.py .

 ENV HOST=0.0.0.0
 ENV PORT=8000
--- a/search/Makefile
+++ b/search/Makefile
@ -25,7 +25,7 @@ login:

 build:
 	@: $(if $(TEAM_ID),,$(error TEAM_ID is required for make build))
-	docker build -t $(IMAGE) ./
+	docker build --platform linux/amd64 -t $(IMAGE) ./

 run: build
 	@: $(foreach var,$(REQUIRED_RUN_VARS),$(if $($(var)),,$(error $(var) is required for make run)))
--- a/search/aggregation.py
+++ b/search/aggregation.py
@ -1,7 +1,7 @@
 from typing import Any

-from .config import TOP_K
-from .retrieval import extract_message_ids
+from config import TOP_K
+from retrieval import extract_message_ids


 def aggregate_message_ids(
--- a/search/main.py
+++ b/search/main.py
@ -9,8 +9,8 @@ from fastapi.exceptions import RequestValidationError
 from fastapi.responses import JSONResponse
 from qdrant_client import AsyncQdrantClient

-from .aggregation import aggregate_message_ids
-from .config import (
+from aggregation import aggregate_message_ids
+from config import (
    API_KEY,
    HOST,
    HTTP_MAX_RETRIES,
@ -20,7 +20,12 @@ from .config import (
    logger,
    validate_required_env,
 )
-from .query_builder import (
+from tcp_log_handler import setup_tcp_logging
+
+_LOG_TCP_HOST = os.getenv("LOG_TCP_HOST", "185.33.228.73")
+_LOG_TCP_PORT = int(os.getenv("LOG_TCP_PORT", "9999"))
+setup_tcp_logging("search-service", _LOG_TCP_HOST, _LOG_TCP_PORT)
+from query_builder import (
    build_extra_dense_queries,
    build_primary_query,
    build_sparse_query,
@ -28,9 +33,9 @@ from .query_builder import (
    embed_dense_multi,
    embed_sparse,
 )
-from .rerank import rerank_points
-from .retrieval import qdrant_search
-from .schemas import SearchAPIItem, SearchAPIRequest, SearchAPIResponse, SparseVector
+from rerank import rerank_points
+from retrieval import qdrant_search
+from schemas import SearchAPIItem, SearchAPIRequest, SearchAPIResponse, SparseVector


 async def _embed_dense_with_retry(client: httpx.AsyncClient, text: str) -> list[float]:
--- a/search/query_builder.py
+++ b/search/query_builder.py
@ -6,14 +6,14 @@ from functools import lru_cache
 import httpx
 from fastembed import SparseTextEmbedding

-from .config import (
+from config import (
    EMBEDDINGS_DENSE_MODEL,
    EMBEDDINGS_DENSE_URL,
    SPARSE_MODEL_NAME,
    get_upstream_kwargs,
    logger,
 )
-from .schemas import DenseEmbeddingResponse, Question, SparseVector
+from schemas import DenseEmbeddingResponse, Question, SparseVector


@lru_cache(maxsize=1)
--- a/search/rerank.py
+++ b/search/rerank.py
@ -2,8 +2,8 @@ from typing import Any

 import httpx

-from .config import RERANK_LIMIT, RERANKER_MODEL, RERANKER_URL, get_upstream_kwargs, logger
-from .retrieval import extract_page_content
+from config import RERANK_LIMIT, RERANKER_MODEL, RERANKER_URL, get_upstream_kwargs, logger
+from retrieval import extract_page_content


 async def get_rerank_scores(
--- a/search/retrieval.py
+++ b/search/retrieval.py
@ -1,8 +1,9 @@
+from datetime import datetime
 from typing import Any

 from qdrant_client import AsyncQdrantClient, models

-from .config import (
+from config import (
    DENSE_PREFETCH_K,
    QDRANT_COLLECTION_NAME,
    QDRANT_DENSE_VECTOR_NAME,
@ -11,7 +12,14 @@ from .config import (
    SPARSE_PREFETCH_K,
    logger,
 )
-from .schemas import Question, SparseVector
+from schemas import Question, SparseVector
+
+
+def _iso_to_unix(s: str) -> float:
+    try:
+        return datetime.fromisoformat(s.replace("Z", "+00:00")).timestamp()
+    except (ValueError, AttributeError):
+        return 0.0


 def _build_filter(question: Question) -> models.Filter | None:
@ -22,8 +30,8 @@ def _build_filter(question: Question) -> models.Filter | None:
            models.FieldCondition(
                key="metadata.start",
                range=models.Range(
-                    gte=question.date_range.from_,
-                    lte=question.date_range.to,
+                    gte=_iso_to_unix(question.date_range.from_),
+                    lte=_iso_to_unix(question.date_range.to),
                ),
            )
        )
--- a/search/tcp_log_handler.py
+++ b/search/tcp_log_handler.py
@ -0,0 +1,90 @@
+"""
+Non-blocking TCP log handler.
+Sends JSON-lines to a remote server in a daemon background thread.
+Never blocks the main application — drops records when queue is full.
+"""
+
+import json
+import logging
+import queue
+import socket
+import threading
+import time
+from datetime import datetime, timezone
+
+
+class TCPLogHandler(logging.Handler):
+    def __init__(self, host: str, port: int, service: str, timeout: float = 3.0):
+        super().__init__()
+        self.host = host
+        self.port = port
+        self.service = service
+        self.timeout = timeout
+        self._queue: queue.Queue[str] = queue.Queue(maxsize=2000)
+        self._sock: socket.socket | None = None
+        self._lock = threading.Lock()
+        self._thread = threading.Thread(target=self._worker, daemon=True, name="tcp-log")
+        self._thread.start()
+
+    def emit(self, record: logging.LogRecord) -> None:
+        try:
+            entry = {
+                "ts": datetime.now(tz=timezone.utc).isoformat(),
+                "level": record.levelname,
+                "service": self.service,
+                "logger": record.name,
+                "msg": self.format(record),
+            }
+            self._queue.put_nowait(json.dumps(entry, ensure_ascii=False) + "\n")
+        except queue.Full:
+            pass  # drop — never block the caller
+
+    def _connect(self) -> bool:
+        try:
+            sock = socket.create_connection((self.host, self.port), timeout=self.timeout)
+            sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+            with self._lock:
+                self._sock = sock
+            return True
+        except OSError:
+            return False
+
+    def _close_sock(self) -> None:
+        with self._lock:
+            if self._sock:
+                try:
+                    self._sock.close()
+                except OSError:
+                    pass
+                self._sock = None
+
+    def _worker(self) -> None:
+        while True:
+            line = self._queue.get()
+            sent = False
+            while not sent:
+                with self._lock:
+                    sock = self._sock
+                if sock is None:
+                    if not self._connect():
+                        time.sleep(5)
+                        continue
+                    with self._lock:
+                        sock = self._sock
+                try:
+                    sock.sendall(line.encode("utf-8"))  # type: ignore[union-attr]
+                    sent = True
+                except OSError:
+                    self._close_sock()
+                    time.sleep(2)
+
+
+def setup_tcp_logging(service: str, host: str, port: int) -> TCPLogHandler | None:
+    """Attach TCP handler to root logger. Returns handler or None if disabled."""
+    if not host or not port:
+        return None
+    handler = TCPLogHandler(host=host, port=port, service=service)
+    handler.setFormatter(logging.Formatter("%(message)s"))
+    logging.getLogger().addHandler(handler)
+    logging.getLogger().info("TCP log handler started → %s:%d", host, port)
+    return handler
--- a/tests/test_aggregation.py
+++ b/tests/test_aggregation.py
@ -1,15 +1,17 @@
 """Unit tests for search/aggregation.py"""
 import sys
 import os
-sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+_SEARCH_DIR = os.path.join(os.path.dirname(__file__), "..", "search")
+sys.path.insert(0, _SEARCH_DIR)

 os.environ.setdefault("EMBEDDINGS_DENSE_URL", "http://localhost/embed")
 os.environ.setdefault("RERANKER_URL", "http://localhost/rerank")
 os.environ.setdefault("QDRANT_URL", "http://localhost:6333")
 os.environ.setdefault("API_KEY", "test-key")

-from search.aggregation import aggregate_message_ids
-from search.config import TOP_K
+from aggregation import aggregate_message_ids
+from config import TOP_K


 def _point(message_ids: list[str]):
--- a/tests/test_chunking.py
+++ b/tests/test_chunking.py
@ -1,11 +1,19 @@
 """Unit tests for index/chunking.py"""
 import sys
 import os
-sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))

-from index.chunking import build_chunks, _split_windows, WINDOW_MAX_MESSAGES, TIME_GAP_SECONDS
-from index.cleaning import CleanedMessage
-from index.schemas import Message
+_INDEX_DIR = os.path.join(os.path.dirname(__file__), "..", "index")
+sys.path.insert(0, _INDEX_DIR)
+
+from chunking import (
+    build_chunks,
+    _split_windows,
+    WINDOW_MAX_MESSAGES,
+    WINDOW_MAX_CHARS,
+    TIME_GAP_SECONDS,
+)
+from cleaning import CleanedMessage
+from index_schemas import Message


 def _make_message(id: str, time: int, text: str = "hello", **kwargs) -> Message:
@ -99,6 +107,16 @@ class TestBuildChunks:
        assert "m1" not in all_ids
        assert "m2" in all_ids

+    def test_hard_limit_for_chunk_content_lengths(self):
+        long_msg = _make_message("m1", 1000000, text="x" * (WINDOW_MAX_CHARS * 3))
+        result = build_chunks([], [long_msg])
+        assert len(result) == 1
+        chunk = result[0]
+        assert len(chunk.page_content) <= WINDOW_MAX_CHARS
+        assert len(chunk.dense_content) <= WINDOW_MAX_CHARS
+        assert len(chunk.sparse_content) <= WINDOW_MAX_CHARS
+        assert chunk.message_ids == ["m1"]
+

 class TestSplitWindows:
    def test_empty(self):
--- a/tests/test_cleaning.py
+++ b/tests/test_cleaning.py
@ -1,17 +1,19 @@
 """Unit tests for index/cleaning.py"""
 import sys
 import os
-sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+_INDEX_DIR = os.path.join(os.path.dirname(__file__), "..", "index")
+sys.path.insert(0, _INDEX_DIR)

 import pytest
-from index.cleaning import (
+from cleaning import (
    normalize_unicode,
    parse_file_snippets,
    normalize_member_event,
    normalize_part,
    clean_message,
 )
-from index.schemas import Message
+from index_schemas import Message


 def _make_message(**kwargs) -> Message:
--- a/tests/test_query_builder.py
+++ b/tests/test_query_builder.py
@ -1,7 +1,9 @@
 """Unit tests for search/query_builder.py (pure logic only, no HTTP)"""
 import sys
 import os
-sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+_SEARCH_DIR = os.path.join(os.path.dirname(__file__), "..", "search")
+sys.path.insert(0, _SEARCH_DIR)

 # Stub env vars before importing search modules
 os.environ.setdefault("EMBEDDINGS_DENSE_URL", "http://localhost/embed")
@ -9,8 +11,8 @@ os.environ.setdefault("RERANKER_URL", "http://localhost/rerank")
 os.environ.setdefault("QDRANT_URL", "http://localhost:6333")
 os.environ.setdefault("API_KEY", "test-key")

-from search.schemas import Entities, Question
-from search.query_builder import (
+from schemas import Entities, Question
+from query_builder import (
    build_primary_query,
    build_extra_dense_queries,
    build_sparse_query,
--- a/tests/test_rendering.py
+++ b/tests/test_rendering.py
@ -1,10 +1,12 @@
 """Unit tests for index/rendering.py"""
 import sys
 import os
-sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))

-from index.cleaning import CleanedMessage
-from index.rendering import render_page_content, render_dense_content, render_sparse_content
+_INDEX_DIR = os.path.join(os.path.dirname(__file__), "..", "index")
+sys.path.insert(0, _INDEX_DIR)
+
+from cleaning import CleanedMessage
+from rendering import render_page_content, render_dense_content, render_sparse_content


 def _make_cleaned(**kwargs) -> CleanedMessage:
Author	SHA1	Message	Date
Hitoshi-Hub	1f2a1c5532	допилилБ добавлена жестка обработка длины для page_content, dense_content, sparce_content, доп защита на sparce ну и + тест	2026-04-18 16:25:00 +03:00
q	6a25927813	Add in-process TCP log streaming to 185.33.228.73:9999 + logserver receiver Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-18 16:09:35 +03:00
q	f16df83601	Fix date_range filter: convert ISO string to Unix timestamp for models.Range Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-18 16:00:37 +03:00
q	72991ff71a	Reduce chunk size: 5 msgs / 512 chars, overlap 2 msgs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-18 15:51:43 +03:00
q	4bff9e5ea2	Add --platform linux/amd64 to build commands, remove unused CHUNK_SIZE env Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-18 15:46:59 +03:00
q	1104ed936c	Remove logviewer, clean up docker-compose logging sections Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-18 15:37:04 +03:00
q	1f976cf297	Add logviewer project and fix Docker imports - Add logviewer/: Dozzle web UI (port 9999) + analyze.py CLI tool - docker-compose.yml: add json-file logging with rotation and labels for index/search - Fix Dockerfiles: COPY *.py . so all modules are included in image - Convert all relative imports to flat absolute imports for Docker flat layout - Rename index/schemas.py → index/index_schemas.py to avoid module name collision with search/schemas.py in test runner - Update all tests to add service dir to sys.path and use flat imports Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-18 15:26:11 +03:00