LangChain

Use polign_db as a LangChain vector store. langchain-polign gives you PolignVectorStore with metadata filters, MMR, and keyword and hybrid search, and needs nothing extra on the server.

Before you start

The package talks to a running polign_db server. For a first try, start one on your machine:

polign-server -store fs:/var/lib/polign

It listens on http://localhost:23000. Get started covers installing the binary and pointing it at a bucket.

Install and use

pip install langchain-polign

from langchain_openai import OpenAIEmbeddings
from langchain_polign import PolignVectorStore

store = PolignVectorStore(
    embedding=OpenAIEmbeddings(),
    collection="docs",
    url="http://localhost:23000",
)

store.add_texts(
    ["cats purr", "dogs bark"],
    metadatas=[{"lang": "en", "score": 0.9}, {"lang": "en", "score": 0.4}],
)

store.similarity_search("purring", k=1, filter={"score": {"$gte": 0.5}})
store.similarity_search_with_relevance_scores("barking", k=2)
store.max_marginal_relevance_search("animals", k=2, fetch_k=10)

store.delete(filter={"lang": "fr"})      # or store.delete(ids=[...])

retriever = store.as_retriever(search_kwargs={"k": 4, "filter": {"lang": "en"}})

Filters and scores

filter is the same metadata language as the Python client: a plain mapping is equality, and $eq, $ne, $in, $gt, $gte, $lt, $lte, $exists, $and, $or, and $not build richer predicates.

similarity_search_with_score returns the distance, where smaller is closer. similarity_search_with_relevance_scores turns it into a score between 0 and 1. The package passes LangChain's standard vector store test suite. Source and the full API are in the langchain-polign README.

Keyword and hybrid search

store.lexical_search("brown fox", k=5)            # keyword only
store.hybrid_search("brown fox", k=5, alpha=0.6)  # vector and keyword, weighted
store.hybrid_search("brown fox", k=5)             # vector and keyword, rank fusion

Keyword search needs a server started with -store. A new document becomes searchable by keyword once the server has written it to the bucket, about half a minute with default settings. Vector search sees it right away.

What gets stored

The document id becomes the record id, so adding the same id again replaces it.
The text is stored in the metadata key text. That is the field the server's keyword index reads, so keyword and hybrid search work without setup.
Metadata that is a string, number, boolean, or a flat list of those is stored as is and can be filtered on.
Nested objects and empty values are saved as text and restored when you read them back. They cannot be filtered on.
A collection is created by the first write and takes its size from the first vector.

Limits

Removing a whole collection needs the server's collection API (-byo-store). Without it, delete by filter.
MMR fetches the candidate vectors in one extra request, because search results do not include vectors.

Source & license

langchain-polign is open source under the Apache License 2.0 and lives next to the Python client in github.com/Polign/polign. Bug reports and pull requests go there.