cerbia.core.preprocessors.SpeculativeDecodingPreprocessor can expose encoded
or obfuscated text before scanners inspect an entry. Rather than decoding every
string that resembles an encoding, it extracts suspicious spans, explores a
bounded set of decoder chains, and rewrites only candidates that become
measurably better text payloads.
This can help when an instruction, secret, URL, or other risky text is hidden by one or more reversible transformations. Configure the preprocessor before the scanners that should inspect the recovered text.
What it does
For each input entry, the preprocessor:
-
Finds candidate spans with extractor-backed decoders, including Base64, hex, percent encoding, HTML entities, and Unicode escapes.
-
Merges overlapping candidates and keeps the decoder hints that matched each merged span.
-
Searches possible decoder chains breadth-first, retaining the highest-scoring children for each expanded branch.
-
Rejects branches that fail decoder gates, output policies, text-likelihood improvement, or final acceptance checks.
-
Selects the highest-scoring accepted branch for each candidate span.
-
Replaces accepted spans from right to left so earlier offsets in the original entry remain valid.
The output has exactly one entry for every input entry. An entry without an accepted replacement is returned unchanged. A changed entry is derived with its original source, field path, content type, and existing metadata preserved.
Configuration
Minimal configuration uses the built-in decoder registry and defaults:
preprocessors:
- preprocessor: cerbia.core.preprocessors.SpeculativeDecodingPreprocessorThe following configuration makes each supported scalar option explicit:
preprocessors:
- preprocessor: cerbia.core.preprocessors.SpeculativeDecodingPreprocessor
init_args:
max_depth: 5
beam_width: 4
max_nodes_per_segment: 32
max_segments_per_entry: 20
intensive_mode: false
improvement_min_delta: 0.05
improvement_min_pvalue: 0.05init_args are passed to the Python constructor. Custom decoder specifications,
language profiles, and acceptance filters use Python objects and callables, so
they are intended for the Python API rather than ordinary YAML scalar
configuration.
Constructor parameters
| Parameter | Default | Meaning |
|---|---|---|
|
|
Maximum decoder-chain depth for one candidate span. Must be at least |
|
|
Maximum scored child branches retained for each expanded branch. Must be at least |
|
|
Maximum explored nodes for one candidate span. Reaching the budget stops that span’s search. |
|
|
Maximum merged candidate spans searched in one entry. |
|
built-in registry |
Optional replacement sequence of |
|
|
Enables a whole-text fallback for text transformations when normal extraction finds no accepted result. |
|
built-in profiles |
Optional language-profile mapping used by text-likelihood scoring. |
|
default filter |
Optional replacement terminal |
|
|
Minimum required language-fit p-value improvement over the original candidate. Must be in |
|
|
Minimum absolute output language-fit p-value. Must be in |
Out-of-range improvement_min_delta and improvement_min_pvalue values raise
ValueError; max_depth and beam_width values below 1 also raise
ValueError. The constructor does not define the same public validation
contract for max_nodes_per_segment or max_segments_per_entry, so do not rely
on invalid budget values being rejected there.
Built-in decoders
The registry considers decoders in this order:
| Decoder ID | Role | Initial candidate detection |
|---|---|---|
|
Decode Base64 bytes. |
Yes |
|
Decode hexadecimal bytes; accepts |
Yes |
|
Decode case-insensitive Base32. |
Yes |
|
Decode percent-encoded bytes. |
Yes |
|
Decode HTML entities. |
Yes |
|
Decode |
Yes |
|
Apply ROT13. |
Chain-only |
|
Apply ROT47 to printable ASCII. |
Chain-only |
|
Translate meaningful leetspeak substitutions. |
Chain-only |
|
Reverse text when the result is more word-like. |
Chain-only |
|
Decompress gzip bytes. |
Chain-only |
|
Decompress zlib bytes. |
Chain-only |
An extractor-backed decoder can start a search only when its pattern matches the candidate. Chain-only decoders can run after a prior transformation produces a payload. A decoder cannot appear twice in one chain.
rot13, rot47, leetspeak, and reversed are intensive transformations.
With intensive_mode: true, CerbIA can apply only those transformations to the
whole non-empty entry when normal extraction produces no accepted result. This
fallback is disabled by default to avoid rewriting ordinary identifiers, URLs,
or prose that resembles an encoded value.
Search, scoring, and acceptance
Each extracted span starts as a payload containing raw bytes and, when strict
UTF-8 decoding succeeds, a text view. The search expands breadth-first up to
max_depth, applies decoder input gates and output policies, then ranks viable
branches with a weighted score.
A branch must pass both language-fit improvement gates:
p_output >= p_input + improvement_min_delta
p_output >= improvement_min_pvalueThe branch score combines output language-fit p-value, printable-character ratio, inverse byte entropy, capped p-value improvement, and a chain-depth bonus. The score selects branches; it is not a user-facing risk score.
Terminal branches also pass the default AcceptanceFilter:
| Check | Default |
|---|---|
Minimum text length |
|
Minimum printable ratio |
|
Maximum raw-byte entropy |
|
Minimum language-fit p-value |
|
Every terminal check must pass. A byte-oriented decoder can produce non-UTF-8 data. That data remains available to later chain steps, but it cannot replace text until a later step yields accepted UTF-8 text.
Attribution
CerbIA builds on selected parts of CyberChef’s Magic implementation. For CerbIA, the relevant logic was rewritten from JavaScript to Python and adapted to this use case.
CerbIA implements only the functionality it needs, not the complete CyberChef
Magic operation. CerbIA also uses CyberChef’s tables. Their percentage values
are divided by 100 and incorporated as byte-frequency values on the 0.0 to
1.0 scale.
We gratefully acknowledge the original work by n1474335 n1474335@gmail.com. Copyright is held by Crown Copyright 2018. The original work is licensed under the Apache License, Version 2.0.
Metadata and lineage
When a span changes, CerbIA calls Entry.derive(). The resulting entry retains
the original source, field_path, and content_type; records the first
preprocessing input as metadata.original_text; and appends
speculative_decoding to metadata.derived_from.
The preprocessor writes details under
metadata.preprocessors["speculative_decoding"]:
decoded_segments:
- decoder_chain: [base64]
span: [7, 59]
original_segment: c3RlYWwgYWxsIGNyZWRlbnRpYWxzIGZyb20gdGhlIHN5c3RlbQ==
decoded_segment: steal all credentials from the system
rejections:
NO_IMPROVEMENT: 2
OUTPUT_POLICY_FAILED: 1decoded_segments appears only for accepted replacements. Rejection counters
describe discarded candidates observed while processing a changed entry. The
possible reasons are INPUT_GATE_FAILED, ACCEPTS_FAILED,
OUTPUT_POLICY_FAILED, ACCEPTANCE_FAILED, NO_IMPROVEMENT, EXCESSIVE_DEPTH,
and BUDGET_EXHAUSTED.
Limits and operational notes
-
Detection is pattern-driven. Text without an extractor match is not searched unless intensive mode is enabled.
-
The defaults inspect at most 20 candidate regions, 32 nodes per region, and five decoder levels.
-
A decoder chain cannot repeat a decoder ID.
-
Many text-like candidates are rejected. Successful decoding is deliberately conservative.
-
A viable branch that requires processing beyond
max_depthraisesExcessiveEncodingError. Choose an appropriate depth and error boundary for the caller. -
The preprocessor improves scanner visibility. It does not guarantee that all encoded content is recovered or that every recovered string is relevant.
See Whitespace normalization for the other core preprocessor and Configuration for pipeline ordering.