Splunk Visual Handbook · lab
Your retention is 26 days, and nothing says 26
Ask a team how long they keep their data and you get the number in
frozenTimePeriodInSecs. Ask the index and you get a
different one, because data rolls to frozen when either limit is
reached — and Splunk's own documentation says it plainly: data will be
rolled to frozen before the configured time period has elapsed if the
size limit is reached first.
Below is one index with the four real defaults and 20 GB a day. The retention says six years. Watch the searchable window settle at twenty-six days — and then watch what happens to it when you move the volume slider, which is the one control on this page that is not in any configuration file.
Break it
- Double the volume. Nothing in the configuration changes, and the window halves. This is the whole argument: retention is an output, not an input. A new source type, a debug level left on after an incident, one chatty deployment — none of them look like a retention change, and all of them are one.
- Set the retention to 7 days. Now which limit froze the data flips to age, and the window follows the number you asked for — slightly above it, never below, because age is measured per bucket and one recent event keeps the whole bucket. This is the configuration doing what it says, and it is the only regime in which the number people quote is the number they have.
- Turn
coldToFrozenDiroff. Every number on the screen stays the same. The buckets keep leaving at exactly the same rate, the disk usage is identical, the window does not move — and the data is gone instead of archived. There is no counter anywhere that distinguishes the two, because deleting is the normal behaviour and archiving is a line somebody has to write.
The four limits, and what each one actually governs
| setting | real default | what it governs |
|---|---|---|
maxDataSize | auto, which is 750 MB | when one hot bucket closes. Granularity, not capacity |
maxWarmDBCount | 300 | how many warm buckets before the oldest rolls to cold. A count, and buckets do not change size when they move |
maxTotalDataSizeMB | 500000 | the size of the whole index. This is the one that decides your retention, in almost every installation |
frozenTimePeriodInSecs | 188697600 — about six years | the maximum age of a bucket. A ceiling, not a promise |
Three of the four are about size, and only one of those three is about how much the index holds. That is the shape of the mistake: the words warm, cold and frozen suggest a hierarchy of storage where data gradually stops costing, and there is no such hierarchy. Warm and cold are both the index — searchable, counted, and next in line.
Predict before you look
An index has frozenTimePeriodInSecs set to 90 days and maxTotalDataSizeMB left at its default of 500000. It receives 20 GB on disk per day. An auditor asks for events from 60 days ago. Are they there?
Commit to an answer first. You cannot change it afterwards.
Configure it
Three gaps from labs/splunk-buckets/idx/indexes.conf,
which is this model as something that boots. The correct block of each
one is lifted out of the file that runs, so it cannot drift from the
config.
All three are in the same stanza, and that is the point rather than convenience: in the lab of the previous phase the lesson was that the same file means different things on different machines. Here there is no wrong machine. There is a wrong limit, and the four of them are five lines apart.
This index receives about 20 GB on disk per day and the disk has 400 GB for it. The team agreed on “we keep 30 days”, and the line below this gap already says 30 days.
What is the line that will actually decide how far back you can search?
# O tamanho do índice INTEIRO. É este que, na prática, decide a retenção de
# quase toda instalação: 500 000 MB é o default, e o dado congela quando
# qualquer um dos dois limites chega primeiro.
Pick one. You cannot change it afterwards — and that is the point.
labs/splunk-buckets/idx/indexes.conf · Splunk Enterprise 10.6.0-rhel9
A compliance requirement: this index must hold 30 days and must not hold more. The disk is large enough for about 90 days at the current volume.
What expresses “thirty days, and then gone”?
# A idade máxima de um balde. O default são 188 697 600 segundos, aproximadamente
# seis anos — e a idade é por BALDE: todo evento dele tem de ser mais velho que
# isto para ele rolar.
Pick one. You cannot change it afterwards — and that is the point.
labs/splunk-buckets/idx/indexes.conf · Splunk Enterprise 10.6.0-rhel9
Same index, and an auditor who may ask for events from eight months ago — which is longer than this index keeps them.
What line keeps the data that leaves the index?
# Congelar é REMOVER do índice. Sem esta linha, o balde é apagado — e apagar é
# o comportamento normal, então não há aviso nenhum.
Pick one. You cannot change it afterwards — and that is the point.
labs/splunk-buckets/idx/indexes.conf · Splunk Enterprise 10.6.0-rhel9
What people get wrong
| What people believe | What the docs say |
|---|---|
| We keep 90 days, because that is what the retention setting says. | The retention setting is a maximum, not a promise. Data rolls to frozen when either limit is reached, and the documentation states it directly: data will be rolled to frozen before the configured time period has elapsed if the size limit is reached first. The number of days you actually have is volume ÷ size limit, and it appears in no configuration file. source |
| Frozen data is archived. That is what frozen means. | Freezing means the data is removed from the index. Archiving happens only if `coldToFrozenDir` or `coldToFrozenScript` is configured — and if neither is, the bucket is deleted. Nothing warns you, because deletion is the normal behaviour. source |
| Moving buckets from warm to cold frees space. | It moves them to a different path, which is usually cheaper storage — and bucket sizes do not change as data moves between states. `maxTotalDataSizeMB` is the limit of the whole index, so cold is not somewhere data goes to stop counting. It is where data waits to be frozen. source |
| Retention is per event, so yesterday's events leave tomorrow. | Age is tracked per bucket. The `indexes.conf` reference requires every event in a bucket to be older than the retention period before the bucket rolls — so a single recent event keeps the whole bucket, and the effective retention is always somewhat longer than the number configured. source |
| A bucket is a file. | It is a directory, and what is in it explains both the compression and the search speed: the compressed raw data journal, the index files that point into it, and some metadata files. The index files can be rebuilt from the journal, which is why the archived form of a bucket keeps only the raw data. source |
What this model does not claim
It evicts from cold only, and it tells you when that stops being
accurate. Splunk freezes the oldest bucket of the index wherever it
is, which with an empty cold tier means a warm one. A node in this
engine knows only its own state, so the size limit here is evaluated
where cold is — and when you configure
maxWarmDBCount × maxDataSize larger than the index limit,
the lab says so on the screen instead of drawing a window it cannot
compute. The real defaults are not in that region: 300 × 750 MB is
225 GB against 488 GB of index.
Hot is not in the size accounting. It is bounded by
maxHotBuckets × maxDataSize, which at the real defaults is
about 2.2 GB against 488 GB — and maxHotBuckets is not a
control here, because with a single stream one hot bucket is open at a
time. It matters when one index receives data from several epochs at
once, which is a different defect and deserves its own lab rather than
one more slider on this one.
It starts from bytes on disk, not from licensed volume. A bucket holds the compressed raw journal plus the index files that point into it, and the ratio between what you ingest and what you store depends on the data. This lab asserts no compression ratio — it begins at the number the size limits actually measure, which is the disk.
Time is compressed, and the control is visible. One tick is twelve hours by default. Lower it and the simulation runs more slowly in simulated days per second, but the window in days does not change — there is a test for exactly that, because the day it stops being true, the parameter stopped being a clock and became part of the model.