encrypted-search-service
Substring search over personal data without reversible encryption — field-scoped, key-rotatable, and built so that a leaked index yields hashes rather than names.
Author and maintainer
Personal project, Apache License 2.0
Java 25, Spring Boot 4.1, OpenSearch
An index leak yields hashes, not names
Encryption breaks search, and search is not optional
Indonesia's Undang-Undang Perlindungan Data Pribadi pushed a question from the legal column into the engineering one: if names, phone numbers and identity numbers are sensitive personal data, they should not sit in a database as plaintext. Encrypting them is easy. What breaks is everything built on top.
A support dashboard over a million users needs to answer “find andi”. Against encrypted values there is no LIKE '%andi%' to run. The workaround — decrypt every row, compare in memory, return matches — is not a workaround at a million rows; it is a way of making security expensive enough that someone eventually turns it off.
Admin dashboards, support tooling and compliance investigations all fail the same way. Without search, a person scrolls page by page to find one record, which is not a system anyone will keep using.
Zero trust, stated up front
The database will leak
The design assumes a breach is possible rather than assuming it is not. That moves protection to the data level instead of the application boundary.
The index is not a safe place
A search index is a second copy of the data and gets treated as hostile ground. It stores tokens, never values.
Authorization stays where the data is
The service never holds credentials to the application's database. It answers with ids, and the application's own rules decide who may read them.
Your app then runs SELECT ... WHERE id IN (ids) — your authorization, your database, your data.
Trade-off: a bigger index, and n-gram frequency leakage inherent to any substring-searchable scheme.
Normalize, tokenize, hash
Normalization runs NFKD decomposition, strips combining marks and lowercases in a locale-independent pass, so “Åsa” and “asa” converge regardless of the server's default locale. All whitespace is removed, not only the leading and trailing kind — which is why searching “wind” matches “Pandu Windito” across the word boundary that used to be there.
The normalized value is then split into overlapping 2- and 3-character n-grams. Hashing a whole value only ever supports exact match; n-grams are what make partial and substring queries possible without the plaintext being present at query time.
Each n-gram is HMACed — with the field name as part of the input, not the n-gram alone. Searching email for “win” therefore cannot match a name containing “Winston”, because those two tokens are different hashes entirely. Query length decides the token set: one character is rejected, two characters search 2-grams, three or more search 3-grams.
What an index leak actually yields
{
"id": "b1f9c2:name",
"originId": "b1f9c2",
"fieldName": "name",
"tokens2": ["9f...", "3a...", "77..."],
"tokens3": ["9a7bf83c2e", "3a19c9a22a", "77d8e12f92"],
"keyVersion": 1,
"indexedAt": "2026-07-16T10:00:00Z"
}One document per (originId, fieldName) pair — which is what makes field-scoped querying possible. An attacker holding this cannot reconstruct the value, cannot tell which tokens came from another field, and cannot reuse a version-1 token against anything written after a rotation.
Four calls worth defending
The field name goes into the HMAC input, so cross-field correlation is impossible rather than merely prevented by query logic. One document per (originId, fieldName) is what makes it queryable.
Writes use one active version; reads accept any listed version — AND within a version, OR across them. Add the new key, let writes drift onto it, drop the old one when nothing depends on it.
Functionally close cousins. Elasticsearch's licensing history sits badly under a project whose whole premise is contribute freely, so the stack stays Apache-2.0 end to end.
The original sketch cached token computation. HMAC-SHA256 over a handful of n-grams costs microseconds — not worth a second stateful service to run and secure. Cut from v1, not forgotten.
Some metadata always leaks, and pretending otherwise is the real risk.
Token frequency and search patterns remain observable. That is true of every practical searchable-encryption scheme of this shape; removing it entirely needs techniques such as fully homomorphic encryption, which are still impractical for latency-sensitive systems. Writing the limit down is part of the design, not an apology for it.
The index also grows: a million users across three searchable fields is on the order of thirty million tokens. That is a real cost, and it happens to be precisely the workload an inverted index exists to serve.
Keeping the index in step with the source is left to the adopter — a synchronous call after the write, or a CDC event stream. Both are safe, because the write endpoint is a genuine idempotent upsert: the document id is derived from (originId, fieldName), so an at-least-once consumer can retry without deduplication logic of its own.
A sales performance dashboard recomputed everything on demand against a legacy system I was not allowed to touch, query directly, or ask for a push feed. Here is what I could change, and what it cost.