Which data pipeline phase is the last opportunity for defining event boundaries?
Which data pipeline phase is the last opportunity for defining event boundaries?
The parsing phase is the correct stage for defining event boundaries. During this phase, data is broken up into events, and advanced processing can be performed to ensure that lines and events are appropriately segmented. This phase respects various line merging settings such as LINE_BREAKER, SHOULD_LINEMERGE, and BREAK_ONLY_BEFORE_DATE, making it the last opportunity to define event boundaries before data is indexed and written to disk.
Parsing phase respects LINE_BREAKER, SHOULD_LINEMERGE, BREAK_ONLY_BEFORE_DATE, and all other line merging settings in props.conf
1. Input phase: Handled at the source (usually a forwarder) – The data sources are being opened and read – Data is handled as streams; configuration settings are applied to the entire stream 2. Parsing phase: Handled by indexers (or heavy forwarders) – Data is broken up into events and advanced processing can be performed 3. Indexing phase: Handled by indexers – License meter runs as data is initially written to disk, prior to compression – After data is written to disk, it cannot be changed