Where developers learn, build, and share. Your source for hands-on demos, cheat sheets, explainers and more.
Global
- Tweets 10,543
- Following 185
- Followers 65,883
- Likes 3,628
A bloom filter is a super fast, and efficient first-pass filtering algorithm.
It hashes each candidate into several required bit positions in a bit array. If any of those required bits are unset, the candidate is definitely absent; otherwise, the candidate may be present and needs verification.
With Elasticsearch's metrics store, it is used for the first stage of a two-stage deduplication process which doesn't even require an inverted index of object IDs. In fact, Elasticsearch doesn't even store a dedicated ID field!
- The `_id` field is now synthetically produced by combining the `_tsid`, and the `@ timestamp`
- The bloom filter works as a first-stage deduplication pass, and then Elasticsearch verifies the candidate point against the synthetic `_id`.
This preserves deduplication behavior without paying for a dedicated inverted index.
TSDB benchmarks show this reducing storage by ~5 bytes per sample.
Elasticsearch for metrics storage: go.es.io/4dSdPfF
3
3
18
2,236
We had Jev score Elasticsearch hybrid search results and let a short Python policy do the reranking.
• nDCG@10 went from 0.9351 to 0.9565 (250 Amazon Queries)
• Latency: 70–500ms end-to-end
• Cost: $0.042/M input tokens (outputs free)
No text parsing, just structured probabilities feeding business logic.
go.es.io/4hAKxDc
5
12
200
4,700,541
Before you run a single kNN query, you need to decide how Elasticsearch stores your vectors.
The dense_vector mapping has five key parameters that determine accuracy, memory usage, and query speed.
Most people copy a snippet from the docs - but knowing what each option does means you can tune for your use case instead of guessing.
Here is the full reference for ES 9.x.
Documentation: go.es.io/4xTiS6M
1
2
6
2,071
Sequence numbers (`_seq_no`) are a key mechanism for making Elasticsearch's distributed system work.
The primary shard assigns sequence numbers of write operations, and replicas use them to make sure that they have "caught up" to the primary.
They also help to power concurrency control for the system.
But, metrics are different. The primary need for a metrics store is append-only, meaning that once the initial replication is complete, there's no real need for further operations that need consistency.
So Elasticsearch can now treat `_seq_no` as vestigial, trimming them once the global replication checkpoint has passed the segment’s maximum sequence number. The trade-off is that optimistic concurrency control, single-document updates and `_seq_no` queries are unavailable.
But these are rarely used with metrics (if ever), and TSDB benchmarks show this reducing storage by over 2.5 bytes per sample! A big part of our reduction from 25 -> 3 bytes per sample.
7
3
37
1,008,625
Elasticsearch 9.5 ships with an improved doc-values codec - ES95.
Since we have no way of knowing what the name comes from, let's talk about the codec itself.
It is an adaptive codec that chooses its own adventure (i.e. compression path).
Monotonic timestamp and long-counter sequences are handled "SplitDelta", and "ALP" encodes suitable double-valued gauge fields efficiently.
The result is a smarter algorithm resulting in smaller footprints. A recent benchmark of 2+ billion high-cardinality data showed a reduction of 30+%, and huge reductions (90+%) for timestamp data!
Try it out: go.es.io/4xMi43D
4
1
13
2,577
Inverted indexes are one of the core pillars of information retrieval.
It is literally THE thing that powers keyword (lexical) searches, by mapping terms against matching document IDs.
But, does every job and every data type require an inverted index? As it turns out, maybe not. Time-series data is a great example. Time series queries often actually ask "What are the values between two points in time?", rather than "What entries contain words XYZ?"
Elasticsearch leverages these properties, by only using inverted indexes where they add value.
TSDS uses doc values and skippers for relevant time-series fields instead of separate backing indexes. Text fields retain inverted indexes where full-text search is required.
Right tools for the right job.
6
5
66
2,018,330
We indexed 138M vectors in under 10 minutes.
Benchmarked on MS MARCO: 138M vectors at 1,024 dims, from ~4 TB of documents, on 8 NVIDIA RTX PRO 6000 GPUs at 95% target recall:
Built with NVIDIA cuVS @NVIDIAAI
- Indexing throughput: 38K docs/s on CPU, 281K on GPU
- 6x lower p90 search latency under concurrent indexing load
- Force merge: roughly 4 hours down to 5 minutes
- Same recall profile as CPU-built indexes
Benchmarks, setup instructions, and the Rally track parameters: go.es.io/3VPp1n2
2
6
10
2,186
Supabase monitoring in Elastic, with no agent to install.
Built in collaboration with @supabase
Available for Supabase Pro, Team & Enterprise on Elastic Cloud.
Integration: go.es.io/4rCe8B6
1
10
2,180
What is a doc-value skipper?
It's a small, but powerful way to efficiently filter values in Elasticsearch. It asks "does this block of doc values contain values in the required range?" by cataloging the minima and maxima for blocks of documents.
If the answer is no, the query can skip the block. If yes, dig into the block to check for matches.
This is a great solution for time-series data, because it is sorted by `_tsid` and `@timestamp`. The resulting blocks are composed of related values; and in turn, they are great for time and dimension filters.
You might ask - why do this? The main benefit is efficiency. Going from relevant inverted-index and BKD-tree indexes to doc value skippers reduced storage by 10 bytes/point in one benchmark.
It's one of the key components that power Elasticsearch's optimisations for metrics, and for Columnar Mode.
Try it out go.es.io/47iOrvK
2
2
6
2,189
Elasticsearch has made huge strides on its metrics store.
Metrics are unique. They are by nature ordered, being a function of time. So, Elasticsearch's TSDB uses these:
- Doc value skippers - fast scans & filtering through time range and dimensions without separate indexes
- Synthetic `_id` from `_tsid` and `@ timestamp`, and bloom filters support fast, efficient deduplication checks
- `_seq_no` trimming after safe merges, leveraging the fundamentally different usage pattern for metrics data
- ES95 codec further improves compression efficiency in Elasticsearch 9.5 and onwards
Benchmarks show each metrics point being reduced in size from ~25 to ~3 bytes per point, a huge reduction.
It might be a great time to consolidate your logs and metrics store to one place.
25 to 3 bytes per data point: go.es.io/4dPQawe
What's new in 9.5, including the ES95 codec: go.es.io/4iIY127
2
5
27
2,460
Your agent gave a wrong answer and you can't reproduce it.
Same prompt, different path. It picked a different tool this run, got an empty result back, and answered anyway.
Re-reading the prompt won't show you that. The trace will.
The OpenTelemetry GenAI semantic conventions provide vendor agnostic telemetry specification, where spans represent model calls and tool calls with arguments for messages attached.
Elastic reads the gen_ai.* attributes and puts token usage on the waterfall beside the spans. A slow run becomes a visibly expensive one.
Worth knowing: the conventions are still moving. Pin your semconv version and expect attribute renames.
Once tool calls are spans, an agent failure is a trace you read top to bottom.
LLM tracing in Elastic APM: go.es.io/4rk8Qd7
4
2
12
2,348
Your user's stack trace died with the browser tab.
They report that the app froze. You have no telemetry, because console.log() wrote to a console you will never see.
Server side you would never accept this. On the client side most teams still do.
The OpenTelemetry JS SDK treats browser logs as a first class signal: @opentelemetry/sdk-logs plus the OTLP HTTP log exporter, pointed at your collector. Elastic ingests the OTLP payload as it arrives, so there's no vendor format to maintain.
Before you ship messages:
- Redact sensitive data and PII such as emails and form values
- Sample telemetry, every user is now a log producer
Wire trace context through and the user's console output arrives attached to the request that broke, and correlates with server side signals. That's what actually shortens the investigation.
3
2
12
2,362
Images fill your context window before the question does.
A single 384px page becomes hundreds of visual tokens. Send
all of them to the LLM and there's no room left to reason.
jina-vlm sits an attention-pooling connector between a
SigLIP2-So400M/14-384 encoder and Qwen3-1.7B-Base. The
connector weighs which visual tokens carry signal, then hands
the text model a smaller set.
Two-stage training, multilingual by design.
Pooling costs you fine detail. Small type inside a dense table
is still hard.
Most people blame the LLM for bad answers on mixed documents.
The trade usually happens at the connector.
Read the paper: go.es.io/3TysG89
7
1
12
2,379
The HNSW vs DiskBBQ call comes down to recall and budget.
HNSW scales logarithmically and it is genuinely fast, but only while the vectors stay resident. Let them spill and latency climbs exponentially. Indexing pays the same tax, because building the graph means searching the graph.
DiskBBQ clusters vectors with hierarchical k-means and keeps them BBQ-quantized on disk, scoring only the clusters a query touches.
Need 99%+ and have the memory budget? Stay on HNSW. Cost-sensitive and fine at 95%?Go with DiskBBQ.
go.es.io/3UQa1Fh
5
5
19
2,389
An AI agent is optimizing Elasticsearch. Engineers validate on improved performance
Coding agents are trained on tasks judged by LLMs and humans. Being convincing gets rewarded. That's a problem when the question is "did this get faster?"
One agent, two phases with different permissions. Exploration goes broad on a real workload and ranks opportunities; a human approves the benchmark and promotes one into a scoped exploitation task. A statistical acceptance rule the agent can't bypass decides whether it lands, and a human reviews before the PR.
All validated with Elasticsearch engineers in the loop
14
11
182
4,367,719