mirror of
https://github.com/grafana/mimir.git
synced 2026-02-13 08:14:03 +00:00
Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
8dbb85e739<!-- Thanks for sending a pull request! Before submitting:
1. Read our CONTRIBUTING.md guide
2. Rebase your PR if it gets out of sync with main
-->
#### What this PR does
Follow up of
https://github.com/grafana/mimir/pull/13806#discussion_r2660110934
The initial implementation in above PR was not accurate because we
didn't handle hash collision when increasing memory labels consumption
hence we can't accurately balancing reducing memory consumption when
handing series to the query engine. This PR do the following update to
make memory tracking for labels accurate:
* Handle hash collision and only increase memory consumption for unique
labels
* Add MemoryTrackingSeriesSet wrapper to wrap the merged result from
store-gateway and ingester whose results are already deduplicated the
labels via MergeSeriesSet
* Revert responsibility of QueryLimiter to the state before PR #13806,
just to handle series limit
* Labels deduplication and increasing memory consumption for labels are
handled by SeriesDeduplicator.
~For MQE, SeriesDeduplicator is initialised in Selector and passed into
context, hence deduplication will be effective per-selector. As an
example for query `foo` + `foo`, there will two SeriesDeduplicator for
each binary operand that will increase memory consumption for unique
labels for each binary operand. Once the series are fetched from
queriers (ingester and store gateway), we pass it to a wrapper
MemoryTrackingSeriesSet which will deccrease memory consumption for
already unique labels for the given selector.~
~For tenant federation case, we override the SeriesDeduplicator so that
it will unique for each tenant. As an example if there are two federated
tenants, there will be two SeriesDeduplicator that will do increase in
which each queriers result will be wrapped by different
MemoryTrackingSeriesSet for memory decrease.~
~Last we also add Deduplicator, together with
UnlimitedMemoryTrackingMiddleware to track memory consumption for Labels
in the case of `/series` and `/read` endpoint call. Without adding
Deduplicator there, we will get negative memory consumption when
iterating the series result from those 2 endpoints.~
`querier.MemoryTrackingQueryable` handles all SeriesLabelsDeduplicator
initialisation.
Some tests need to be updated by passing UnlimitedMemoryTracker to
context or by adding custom queryable to simulate when we have to
increase/decrease labels memory consumption.
## Memory Benchmark
We can see the more duplicates the B/op is smaller. No duplicates 21
B/op, 50% duplicates 10B/op and 90% duplicates 1 B/op only. Which means
less allocations the more duplicate labels in the results because we
reused the labels.
<details>
```
go test -bench='BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_(NoDuplicates|50pct|90pct)$' \ -benchmem \
-benchtime=5000x \
-count=5 \
-run=^$ \
./pkg/util/limiter/
goos: darwin
goarch: arm64
pkg: github.com/grafana/mimir/pkg/util/limiter
cpu: Apple M4 Pro
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_NoDuplicates-14 5000 19866 ns/op 21 B/op 0 allocs/op
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_NoDuplicates-14 5000 19553 ns/op 21 B/op 0 allocs/op
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_NoDuplicates-14 5000 19202 ns/op 21 B/op 0 allocs/op
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_NoDuplicates-14 5000 17423 ns/op 21 B/op 0 allocs/op
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_NoDuplicates-14 5000 17633 ns/op 21 B/op 0 allocs/op
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_90pct-14 5000 18806 ns/op 1 B/op 0 allocs/op
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_90pct-14 5000 19545 ns/op 1 B/op 0 allocs/op
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_90pct-14 5000 18882 ns/op 1 B/op 0 allocs/op
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_90pct-14 5000 20637 ns/op 1 B/op 0 allocs/op
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_90pct-14 5000 21143 ns/op 1 B/op 0 allocs/op
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_50pct-14 5000 17031 ns/op 10 B/op 0 allocs/op
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_50pct-14 5000 17139 ns/op 10 B/op 0 allocs/op
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_50pct-14 5000 16878 ns/op 10 B/op 0 allocs/op
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_50pct-14 5000 20526 ns/op 10 B/op 0 allocs/op
BenchmarkSeriesDeduplicator_Deduplicate_WithCallerDedup_50pct-14 5000 17965 ns/op 10 B/op 0 allocs/op
```
</details>
#### Which issue(s) this PR fixes or relates to
Fixes https://github.com/grafana/mimir-squad/issues/3280
#### Checklist
- [x] Tests updated.
- [ ] Documentation added.
- [ ] `CHANGELOG.md` updated - the order of entries should be
`[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry
is not needed, please add the `changelog-not-needed` label to the PR.
- [ ]
[`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md)
updated with experimental features.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches core query limiting and query streaming paths
(ingester/store-gateway/distributor) and changes how label memory is
accounted for; behavior should be equivalent but regressions could
affect query rejection limits or memory tracking accuracy.
>
> **Overview**
> Improves per-query labels memory accounting by introducing
`SeriesLabelsDeduplicator` (context-scoped, hash-collision aware) and
moving labels dedup + label-memory increases out of `QueryLimiter`.
>
> Query paths that build/stream series labels (distributor
`MetricsForLabelMatchers`, ingester streaming, and store-gateway
streaming) now deduplicate labels via the new deduplicator, then apply
`QueryLimiter.AddSeries()` strictly for max-series enforcement. A new
`querier.MemoryTrackingQueryable` wraps MQE queryables so `Select()`
installs the deduplicator and wraps results in
`MemoryTrackingSeriesSet`; several call sites drop older
`MemoryTrackingSeriesSet` wrappers to avoid double-wrapping, and ruler’s
dedicated “unlimited memory tracker queryable” is removed in favor of
injecting the memory tracker via the manager context.
>
> Tests/benchmarks are updated to ensure contexts include the new
deduplicator (and where needed a memory tracker), and `QueryLimiter`
internals/tests are simplified after removing
label-return/memory-tracking responsibilities.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
|
||
|---|---|---|
| .config/go | ||
| .github | ||
| cmd | ||
| development | ||
| docs | ||
| images | ||
| integration | ||
| mimir-build-image | ||
| operations | ||
| packaging | ||
| pkg | ||
| tools | ||
| vendor | ||
| .gitattributes | ||
| .gitconfig | ||
| .gitignore | ||
| .golangci.yml | ||
| .lintignore | ||
| .prettierignore | ||
| ADOPTERS.md | ||
| AGENTS.md | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| CODE_OF_CONDUCT.md | ||
| CODEOWNERS | ||
| CONTRIBUTING.md | ||
| go.mod | ||
| go.sum | ||
| GOVERNANCE.md | ||
| LICENSE | ||
| LICENSING.md | ||
| MAINTAINERS.md | ||
| Makefile | ||
| Makefile.local.example | ||
| README.md | ||
| RELEASE.md | ||
| renovate.json5 | ||
| VERSION | ||
Grafana Mimir
Grafana Mimir is an open source software project that provides a scalable long-term storage for Prometheus. Some of the core strengths of Grafana Mimir include:
- Easy to install and maintain: Grafana Mimir’s extensive documentation, tutorials, and deployment tooling make it quick to get started. Using its monolithic mode, you can get Grafana Mimir up and running with just one binary and no additional dependencies. Once deployed, the best-practice dashboards, alerts, and runbooks packaged with Grafana Mimir make it easy to monitor the health of the system.
- Massive scalability: You can run Grafana Mimir's horizontally-scalable architecture across multiple machines, resulting in the ability to process orders of magnitude more time series than a single Prometheus instance. Internal testing shows that Grafana Mimir handles up to 1 billion active time series.
- Global view of metrics: Grafana Mimir enables you to run queries that aggregate series from multiple Prometheus instances, giving you a global view of your systems. Its query engine extensively parallelizes query execution, so that even the highest-cardinality queries complete with blazing speed.
- Cheap, durable metric storage: Grafana Mimir uses object storage for long-term data storage, allowing it to take advantage of this ubiquitous, cost-effective, high-durability technology. It is compatible with multiple object store implementations, including AWS S3, Google Cloud Storage, Azure Blob Storage, OpenStack Swift, as well as any S3-compatible object storage.
- High availability: Grafana Mimir replicates incoming metrics, ensuring that no data is lost in the event of machine failure. Its horizontally scalable architecture also means that it can be restarted, upgraded, or downgraded with zero downtime, which means no interruptions to metrics ingestion or querying.
- Natively multi-tenant: Grafana Mimir’s multi-tenant architecture enables you to isolate data and queries from independent teams or business units, making it possible for these groups to share the same cluster. Advanced limits and quality-of-service controls ensure that capacity is shared fairly among tenants.
Migrating to Grafana Mimir
If you're migrating to Grafana Mimir, refer to the following documents:
Deploying Grafana Mimir
For information about how to deploy Grafana Mimir, refer to Deploy Grafana Mimir.
Getting started
If you’re new to Grafana Mimir, read the Get started guide.
Before deploying Grafana Mimir in a production environment, read:
Documentation
Refer to the following links to access Grafana Mimir documentation:
- Latest release
- Upcoming release, at the tip of the
mainbranch
Contributing
To contribute to Grafana Mimir, refer to Contributing to Grafana Mimir.
Join the Grafana Mimir discussion
If you have any questions or feedback regarding Grafana Mimir, join the Grafana Mimir Discussion. Alternatively, consider joining the monthly Grafana Mimir Community Call.
Your feedback is always welcome, and you can also share it via the #mimir Slack channel.
License
Grafana Mimir is distributed under AGPL-3.0-only.