AI-Powered Search Engine

Part of the worlds-biggest-software-project initiative.

An open-source hybrid search engine that combines vector similarity, BM25 keyword scoring, and RAG-based answer synthesis for accurate, citation-grounded results over private corpora.

Traditional keyword search fails when users cannot phrase their query with the exact terminology used in the source material, while pure semantic search struggles with precise term lookup and can surface irrelevant adjacent topics. This project pairs hybrid retrieval with semantic reranking and retrieval-augmented generation so teams can search private content and get synthesised, cited answers rather than a list of links. It targets engineering teams, knowledge-base operators, and product builders who need search that works on their own corpus.

Why AI-Powered Search Engine?

Algolia NeuralSearch locks vector search behind its Elevate tier ($50K+/yr) and offers no self-hosting option, putting hybrid AI search out of reach for SMBs.
Elasticsearch gates security features (RBAC, field-level security, audit logging) behind paid Platinum/Enterprise tiers, and its Elastic Licence 2.0 / SSPL split deters enterprise legal teams.
Meilisearch (SSPL) and Typesense (GPL-3.0) carry copyleft / SaaS-redistribution restrictions that complicate commercial hosting.
Pinecone, Perplexity, and Azure AI Search are fully proprietary SaaS with no self-hosting path, and Perplexity has no private-corpus indexing at all.
Existing engines under-serve GraphRAG, sentence-level citation grounding, query analytics for zero-result queries, access-control-aware retrieval, and bring-your-own-corpus connectors.

Key Features

Hybrid Retrieval and Ranking

BM25 full-text search combined with dense vector search, merged using Reciprocal Rank Fusion (RRF)
Sparse vector support (SPLADE++ or BM25-as-sparse-vector) for hybrid precision
Semantic reranking via a cross-encoder model on top-k retrieval candidates
Per-result relevance score exposure for transparency and tuning
Faceted filtering with structured metadata predicates

RAG Answer Synthesis

LLM-based answer synthesis over the top retrieved passages
Inline citations linking generated content back to source passages
Sentence-level citation grounding (backlog) for fine-grained provenance
Zero-result recovery patterns for queries where retrieval fails

Ingestion and Indexing

Real-time document ingestion with chunking and embedding pipeline
Connector framework for ingesting from wikis, databases, and cloud storage
Multimodal index roadmap covering text, images, and audio transcripts in a unified collection
Multi-tenant API key scoping for data isolation

Developer Experience and Operations

REST API with an OpenAPI 3.1 specification
Python and JavaScript SDKs at minimum
Query analytics dashboard surfacing zero-result and low-confidence queries
Evaluation suite exposing NDCG@k, recall@k, and answer faithfulness scores
Access-control-aware retrieval with document-level permission filtering

AI-Native Advantage

AI capabilities are core, not bolted on: semantic reranking with cross-encoders re-scores retrieval candidates for final ordering, and an RAG layer synthesises a direct, cited answer from the top results. Beyond retrieval, LLMs can power query rewriting and expansion, automate chunking-strategy selection by reading document structure, score index quality to flag documents that degrade results, and infer personalisation signals from session context without manual rule configuration. A GraphRAG layer is on the backlog to answer multi-hop questions that require connecting facts across documents.

Tech Stack & Deployment

The project is intended to be self-hostable, with a managed deployment option as a follow-on. Building on Apache 2.0 / BSD-3-Clause components such as OpenSearch, Qdrant, Weaviate, and FAISS, plus Lucene-derived indexing, avoids the licence restrictions that affect Elasticsearch, Meilisearch, and Typesense. Reciprocal Rank Fusion is in the public domain and HNSW is published in the academic literature with no known patent encumbrance. Integration follows the now-standard pattern of a REST API with an OpenAPI 3.1 spec, official Python and JavaScript SDKs, and native vector-store integrations with LangChain and LlamaIndex.

Market Context

Hybrid search combining semantic and keyword retrieval has become the dominant pattern in enterprise and developer-facing search products as of 2025-2026, with RAG-based summarisation now table-stakes for AI search products. Incumbent pricing ranges from Algolia NeuralSearch at $50K+/yr (Elevate tier) to Perplexity Sonar at $5 per 1,000 search requests, with Elasticsearch and Azure AI Search gating advanced features behind paid tiers. Primary buyers are engineering teams building search into SaaS products, knowledge-base and documentation teams, and enterprise platform groups deploying RAG over internal corpora.

Candidate metadata: complexity 6/10, domain availability High, demand High (Knowledge & Information category).

Project Status

This project is in the research and specification phase.
Contributions, feedback, and domain expertise are welcome.

Contributing

We welcome contributions from developers, domain experts, and potential users. See CONTRIBUTING.md for guidelines.

Important: All contributions must be your own original work or clearly attributed open-source material with a compatible licence. Copyright infringement and licence violations will not be tolerated and will result in immediate removal of the offending contribution. If you are unsure whether a piece of code, text, or other material is safe to contribute, open an issue and ask before submitting.

Licence

Licence to be determined. See discussion for context.