Splunk Visual Handbook · lab
The file was right. The machine was wrong.
Splunk's own documentation describes the product as four phases — input, parsing, indexing, search — and then states the rule this lab exists to make visible: each configuration belongs on only one component, the first one in the deployment that handles that phase. A universal forwarder handles input. It does not parse. So every line-breaking setting you put on it is valid syntax, deploys cleanly, and does nothing.
Below, the application writes a twelve-line stack trace every twentieth
record. Move props.conf between the three machines and
watch the verdict line change. Then put it on the indexer, where it
belongs, and tick the middle tier is a heavy forwarder —
nobody touched the file, and the stack trace goes back to being twelve
events.
Break it
Three moves, and the third one is the one that happens to teams who never touched their configuration at all.
-
props.confon the universal forwarder. The verdict line says no effect at all, and events per stack trace climbs to twelve. This is the most common version of this mistake, and it is easy to see why: the forwarder is the machine that reads the file, so it feels like the machine that should know how to read it. It reads bytes. Its only phase is input — cut the stream into 64 KiB blocks, annotate each one with host, source and source type, forward. Before the parsing phase runs, there is no such thing as an event yet. - Move it to the indexer. The verdict turns, stack traces · whole starts climbing and the twelve becomes one. Nothing else about the run changed: same records, same blocks, same bytes. The only difference is which machine holds a file.
- Leave it there, and add a heavy forwarder. Tick the switch.
The verdict goes back to no effect at all. Nobody edited
the file, nobody touched the application, nobody changed a source
type — somebody put a hop in the middle of the path, for a network
reason, in a different ticket. The parsing phase moved with it, and
the indexer now receives events that are already broken and does not
re-parse them. The
props.confon it is still correct and now has nothing left to act on.
The six combinations, and the two that work
Three machines where the file can be installed, two topologies. The table is not a list to memorise: it is one sentence applied six times — the line breaking takes effect only if the file is on the machine that runs the phase.
props.conf installed on | no heavy forwarder | heavy forwarder in the path |
|---|---|---|
| universal forwarder | no effect | no effect |
| intermediate tier | no effect — it forwards, it does not parse | works |
| indexer | works | no effect — the events arrived parsed |
Predict before you look
Answer first, then run it. Predicting and being wrong is what makes the answer stick.
Your stack traces are being indexed one line per event. You have a correct LINE_BREAKER in props.conf on every indexer, pushed by the deployment server, and the data flows through a heavy forwarder that does TLS termination. What fixes it?
Commit to an answer first. You cannot change it afterwards.
Configure it
Three lines from labs/splunk-ingest/, which is this model
as something that boots. The correct block of each one is lifted out of
the file that runs, so it cannot drift from the config.
Two of them are on the forwarder and one is on the indexer, and that is the point rather than variety: what is valid depends on the machine. The same file name exists on both, holding settings that do nothing on the other — and neither of them complains when it receives the neighbour's.
The universal forwarder on the application host. This file is the input phase: it opens the log file and annotates what it reads. The indexer has a props.conf with a `[app:json]` stanza in it, holding the line breaking for this application.
What has to be on this line for that stanza to apply?
Pick one. You cannot change it afterwards — and that is the point.
labs/splunk-ingest/uf/inputs.conf · Splunk Universal Forwarder 10.6.0-rhel9
The same forwarder, sending to one indexer. The indexer will be restarted tonight for patching, and the application keeps writing throughout.
What makes the forwarder resend what did not land?
Pick one. You cannot change it afterwards — and that is the point.
labs/splunk-ingest/uf/outputs.conf · Splunk Universal Forwarder 10.6.0-rhel9
The indexer, which in this topology is the first component capable of parsing. The application writes a twelve-line Java stack trace, every line of which starts with a tab except the first — and the first starts with an ISO 8601 timestamp.
What decides that those twelve lines are one event?
Pick one. You cannot change it afterwards — and that is the point.
labs/splunk-ingest/idx/props.conf · Splunk Enterprise 10.6.0-rhel9
What people get wrong
Every row below is a belief worth naming, because the model above is arranged to take it apart. The right column is the documentation, with the link.
| What people believe | What the docs say |
|---|---|
| The universal forwarder reads my props.conf, so that is where line breaking belongs. | The universal forwarder does not parse data except in a few limited situations, and what it forwards is unparsed. Line breaking, line merging, timestamp extraction and index-time field extraction all belong to the parsing phase, which runs on the first component capable of it — an indexer or a heavy forwarder. The file installed on the forwarder is syntactically valid, deploys cleanly, and changes nothing. source |
| Putting a heavy forwarder in the middle is just a hop. It cannot change my events. | It moves the parsing phase. Splunk's rule is that each configuration belongs on only one component — the first in the deployment that handles that phase — so with a heavy forwarder in the path, the input phase happens on the universal forwarders and the parsing phase happens on the heavy forwarder. The indexer then receives parsed events and the props.conf sitting on it has nothing left to break. source |
| An event is a line. Splunk figures the rest out. | The parsing phase is what decides where an event begins and ends: it breaks the stream into lines, identifies and sets the timestamp, and transforms the data according to regex rules. Before that phase runs, the concept of an event does not exist yet — what travels is a 64K block annotated with host, source and sourcetype. source |
| If the events were broken wrong, something would say so. | Nothing does. A twelve-line stack trace indexed as twelve events is indistinguishable, in the index, from twelve single-line log records: the index stores the events it was given, each with its own timestamp and its own default fields. The cost shows up twice, quietly — in the licence volume and in a search that returns the one line it matched, with no stack under it. source |
| A source type is a label for searching. | It is the handle the parsing phase uses: the source type identifies the data structure of an event, and it determines how the platform formats the data during indexing. Every line-breaking rule is attached to it, which is why data arriving with the wrong source type is broken by the wrong rules — and is still, by every counter, ingested successfully. source |
What this model does not claim
One indexer. The forwarder here sends to a single receiver, and
that is a deliberate boundary: the moment there are two, a forwarder
that has not broken the stream into events can switch destination in
the middle of one — and that is a different defect, with a different
setting (EVENT_BREAKER, which is valid only on universal
forwarders) and a different phase of this handbook. Teaching it here
would blur the one claim this lab makes.
The block is bytes, not time. The 64 KiB buffer is modelled at a fixed cost per line, so the block fills on volume alone. A real forwarder also flushes on other conditions, and a monitored file that stops being written to does not keep a block hostage forever. What the buffer is here for is the delay between written and searchable, which is real and which operators meet before they meet the explanation for it.
Breaking is all-or-nothing. A record is either one event or one
event per line. Real line breaking fails in more interesting ways —
a regex that matches too much, a timestamp recognised in the wrong
field, a truncated event at TRUNCATE. The lab models the
binary case because the question it answers is where, not
which regex.