Larger chunks give the reranker more context per candidate and reduce
chunk count (~2x fewer), so each message_id is better represented.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Improvements to RERANK_LIMIT, search_text, variants made score worse.
Reverting to investigate better approach.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- RERANK_LIMIT 10→60: rerank more candidates → better NDCG ordering
- Dense query uses search_text if available (semantically richer than text)
- Sparse query always appends keywords on top of base text
- Multi-vector: use variants[:2] + hyde[:2] as extra dense queries
(previously only hyde[:2])
- Index UVICORN_WORKERS 8→4: matches 4-core constraint, saves ~1GB RAM
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both services are now single-file (main.py only), exactly matching
the Lotus solution structure that passes the test stand:
- index: char-based sliding window chunking (256/128), is_system+is_hidden
filter, render_message consistent with Lotus, UVICORN_WORKERS=8
- search: validate_required_env at module level, embed_dense_batch for
HyDE, 429 retry on reranker, RRF fusion without per-query filter
- Dockerfiles: COPY main.py . (no extra modules to import)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Lotus explicitly filters both flags before building chunks. Our _clean_all
was only filtering by is_empty, so system/hidden messages with content
(e.g. member_event) were included in chunks and polluted the index.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- embed_dense_multi now sends one batch request (N texts → 1 API call) instead of N parallel
requests, avoiding rate-limit errors when question has variants/hyde
- Extra dense embeddings (variants/hyde) wrapped in try/except so primary query always succeeds
- Reranker now retries up to 5 times with exponential backoff on 429, matching Lotus reference
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add logviewer/: Dozzle web UI (port 9999) + analyze.py CLI tool
- docker-compose.yml: add json-file logging with rotation and labels for index/search
- Fix Dockerfiles: COPY *.py . so all modules are included in image
- Convert all relative imports to flat absolute imports for Docker flat layout
- Rename index/schemas.py → index/index_schemas.py to avoid module name collision with search/schemas.py in test runner
- Update all tests to add service dir to sys.path and use flat imports
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
