CTEPM: Continuous-Time Event Process Memory for Long-Video Language Models¶

Conference: ECCV2026
Paper: Official page · PDF
Area: Video Understanding
Keywords: long-video question answering, continuous time, marked temporal point processes, event memory, temporal operators

TL;DR¶

CTEPM represents long videos as continuous-time event processes with semantic marks and conditional intensities, computes temporal evidence before generating an answer, and improves LongVideoBench accuracy from 46.0% to 49.2% under the same 64-frame budget.

Background & Motivation¶

Long-video question answering requires more than recognizing an action in a frame: a model may need to count drinks, compare action frequency between the two halves of a video, or estimate the delay between opening a door and sitting down. Uniform sampling can miss brief events, segment retrieval can locate relevant moments without collecting every occurrence required for counting, and free-form summaries can preserve meaning while losing precise timing. Counting, rate comparison, and interval estimation therefore remain largely implicit computations inside the language model.

Adding frames reduces observational gaps but does not automatically introduce a temporal computation mechanism. MA-LMM-style memory banks preserve more historical information, while pruning methods such as FastVID reduce visual token costs. These approaches mainly change how much evidence reaches the model. CTEPM instead changes the representation of that evidence: sparse events and their occurrence intensities become a memory that can be queried for counts, order, and waiting times without rereading a long sequence of segments for every question.

This representation also has a cost: inferred events are not necessarily all the real events, and process integrals are not deterministic observed counts. The goal is to provide more structured temporal evidence under constrained input budgets, not to recover every unseen frame. Core idea: turn video memory from retrievable event records into a continuous-time event process with executable temporal operators, separating numerical reasoning from language generation.

Method¶

Overall Architecture¶

The inputs are timestamped video features and a natural-language question; the output is a textual answer. Event proposal selects a small number of event centers and aggregates local semantics. Event process memory then models temporal trends and historical interactions between event types. A temporal query executor selects the appropriate operators and passes compact structured evidence to the answer model.

The paper's non-generative setting excludes video generation, not final text generation. By default, the visual input cap is \(F=64\) frames and the event budget is \(N=32\), controlling observation and event-level reasoning separately. The budget experiments explicitly state that CTEPM builds memory from the same \(F\) frames. Thus, the architecture's reference to dense features should not be interpreted as unrestricted access to every video frame at no additional cost.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Video and timestamped features"] --> B["Continuous-Time<br/>Event Proposal"]
    B --> C["Interaction-Aware<br/>Event Process Memory"]
    C --> D["Temporal Query Executor"]
    Q["Natural-language question"] --> D
    D --> E["Structured evidence and question"]
    E --> F["Language-model answer"]

Key Designs¶

1. Continuous-Time Event Proposal: replace a long token sequence with sparse events carrying time and semantics

The proposal module predicts an eventness score for each timestamped feature, constructs a selection distribution with a temperature-controlled softmax, and selects \(N\) centers using either a Gumbel-Top-N relaxation without replacement or a deterministic soft top-N approximation. Around each center, local temporal attention pooling and projection produce a semantic mark. A separate classification head predicts soft event-type assignments during training, with an argmax type available at inference. Each event consequently carries its occurrence time, an approximate type, and richer semantics than a discrete label alone.

The meaning of continuous time needs care: Eq. 5 obtains an event time by looking up the timestamp of its selected center, \(t_i=\tau_{c_i}\), rather than regressing a sub-frame offset. Event centers therefore remain limited by the sampling resolution; the subsequent intensity function and temporal queries are continuous, not the precision of visual localization. Local pooling reduces reliance on a single center feature, while sparsity and repulsion regularization discourage overly dense or clustered proposals. Neither mechanism guarantees that one peak corresponds to exactly one complete action.

2. Interaction-Aware Event Process Memory: let historical events change subsequent occurrence intensity

A type-specific conditional intensity describes the tendency for that event type to occur per unit time given the history. Time embeddings and an MLP represent global trends, historical events contribute triggering or inhibition over shorter and longer timescales, and softplus ensures a nonnegative final intensity. Global trends can capture an action becoming more common later in a video, whereas historical interactions represent changes in another event's tendency shortly after an occurrence. The cached extraction of Eq. 8 loses the operators between its terms, so this note does not invent a complete intensity formula.

The historical interaction term in Eq. 9 is explicit:

\[ h_k(\mathcal H_t)=\sum_{t_j<t}\sum_{k'=1}^{K}A_{kk'}\kappa_\theta(t-t_j)\pi_{k'}(\mathbf m_j). \]

Here \(A_{kk'}\) can be positive or negative, expressing triggering or inhibition, and \(\pi_{k'}(\mathbf m_j)\) is the soft probability that a historical event belongs to type \(k'\). Soft assignments avoid collapsing uncertain semantics into a hard label too early. The nonnegative temporal kernel uses a mixture of exponentials, with \(R=8\) components by default, combining different decay rates to represent short- and long-term effects while supporting incremental evaluation. These are learned temporal dependencies, not interventionally established causal relationships.

Semantic marks are additionally constrained by a history-conditioned Gaussian distribution. A small history encoder, such as a causal Transformer, predicts its mean and diagonal variance to encourage temporally consistent semantic evolution. The paper treats this term as a regularizer and allows a discriminative loss when type or mark supervision is strong. The memory is thus more than a timestamp list: it includes type-specific intensities, cross-type interactions, and a constraint on semantic consistency.

3. Temporal Query Executor: compute evidence before asking the language model to respond

A lightweight question parser, implemented as a small language-model head or classifier, identifies the operator family, target event types, and time ranges. The count operator integrates the selected type's conditional intensity; the rate operator divides that quantity by the interval length. By default, Monte Carlo quadrature uses \(J=64\) uniformly sampled times. The following retains the computational form of Eqs. 12–13, with \(\tilde t_j\) sampled uniformly within the query interval:

\[ z_{\mathrm{count}}(k;[t_a,t_b])=\int_{t_a}^{t_b}\lambda_k(t\mid\mathcal H_t)\,dt\approx\frac{t_b-t_a}{J}\sum_{j=1}^{J}\lambda_k(\tilde t_j\mid\mathcal H_{\tilde t_j}),\qquad z_{\mathrm{rate}}=\frac{z_{\mathrm{count}}}{t_b-t_a}. \]

The paper calls this integral an expected count. More precisely, integrating a history-dependent stochastic intensity along a given history yields a compensator; interpreting it as an unconditional expected count also requires expectation over the history distribution. It is therefore not a direct count of observed peaks and need not be an integer. This distinction matters for repeated actions: the process can express uncertainty, but its statistical estimate should not be presented as an exact observed fact.

The order operator samples the process and estimates which target type has the earlier first arrival, using \(S=32\) samples by default. The interval operator retains pairs where the second event follows the first and estimates their waiting time. Duration queries instead encode a state through entry and exit event types, measuring differences from sampled matches or nearest feasible proposal pairs. Which event happened first, how long until another event occurred, and how long a state lasted thus require different computations, not arbitrary subtraction of two timestamps.

Outputs are usually a few scalars or small vectors representing counts, probabilities, and time estimates. They are serialized as compact text or fixed embeddings and supplied with the question to the language model. The model handles expression while the executor performs inspectable temporal computation. This interface still depends on correct question parsing, alignment between question semantics and event types, and feasible entry/exit matching; the paper does not fully specify every exceptional case.

A Worked Example¶

Consider the paper's question about whether an action occurs more frequently in the second half of a video. This is a procedural illustration, not a fabricated experimental answer. After selecting \(32\) events from the default \(64\) input frames, semantic marks help identify the queried action type, and the process memory supplies its time-varying intensity based on the event history.

The executor divides the timeline into two halves, integrates the action intensity over each half, and divides each result by its interval length to obtain an average rate. It then passes both rates and their comparison to the answer model. Multiple semantically similar occurrences need not be expanded into prose and counted mentally by the model. If observations missed an action, the output remains an estimate supported by the learned process, not a recovery of the true number of unseen occurrences.

Loss & Training¶

The point-process negative log-likelihood has three components: rewarding the appropriate type intensity at observed event times, penalizing excessive total intensity across the entire timeline, and fitting the history-conditioned semantic mark distribution. The integral penalty prevents explaining events simply by raising intensity everywhere. Question answering uses standard token-level cross-entropy, while proposal regularization includes eventness sparsity and soft repulsion between very close events.

\[ \mathcal L=\mathcal L_{\mathrm{qa}}+\alpha\mathcal L_{\mathrm{pp}}+\beta\mathcal L_{\mathrm{prop}}. \]

The paper describes an end-to-end objective and uses a frozen backbone, lightweight LoRA adaptation, and AdamW in its experiments. The main backbone is labeled LLaVA-NeXT-Video-7B. The new modules are learned, but the cached text does not fully detail their optimization configuration, loss weights, or training-data recipe; learning rates, training epochs, and LoRA ranks should not be invented.

Key Experimental Results¶

Main Results¶

The following results come from Tables 1–3 under the same backbone and \(64\)-frame cap; Video-MME includes subtitles. LongVideoBench and Video-MME report Top-1 accuracy, while MLVU reports M-Avg and AO/AC task scores. Gains are percentage points or score points over uniform sampling, not relative percentages.

Dataset / metric	Uniform sampling	Segment retrieval	FastVID	CTEPM	Gain
LongVideoBench overall	46.0	46.8	46.5	49.2	+3.2
LongVideoBench 900–3600 seconds	41.0	41.8	41.3	45.6	+4.6
Video-MME overall, with subtitles	52.5	52.9	52.7	55.0	+2.5
Video-MME long videos	46.5	47.1	46.8	49.9	+3.4
MLVU M-Avg	34.0	34.6	34.3	37.8	+3.8
MLVU Action Order	16.0	16.6	16.2	21.5	+5.5
MLVU Action Count	21.0	22.0	21.6	27.8	+6.8

Ablation Study¶

The cache does not include supplementary Sec. C, so the sensitivity to \(N\) and \(R\), or component ablations removing triggering/inhibition, operators, or mark regularization, cannot be verified. The following is the real frame-budget analysis from Table 1 and memory comparison from Table 4, not a substitute claim of component-level ablations.

Configuration	Frames	LongVideoBench overall	900–3600 seconds	Over 30 minutes
Uniform sampling	64	46.0	41.0	40.8
More-Frames	128	47.1	42.0	Not reported in the corresponding table
Segment retrieval	64	46.8	41.8	41.6
MA-LMM-style Memory	64	47.5	Not reported in the corresponding table	42.4
CTEPM	64	49.2	45.6	45.9

Key Findings¶

Increasing sampled frames from 64 to 128 improves overall accuracy by only 1.1 points; CTEPM with 64 frames exceeds the 128-frame baseline by 2.1 points. This supports the importance of computation beyond input length but does not identify individual module contributions.
Against matched-budget MA-LMM-style memory, CTEPM gains 1.7 points overall and 3.5 points on videos longer than 30 minutes. This is the authors' same-backbone memory implementation, not necessarily a complete reproduction of the original MA-LMM system.
MLVU gains on counting and ordering exceed the overall gain, aligning with the purpose of explicit operators. These correlated improvements do not establish that an individual operator independently causes the entire gain.
The Fig. 3 discussion reports a +3.1 LongVideoBench gain at \(F=64\), whereas Table 1 gives 49.2−46.0 = +3.2. This note uses the directly checkable table values and explicitly retains the reporting inconsistency.

Highlights & Insights¶

Memory becomes a computational object. Event records describe what was stored; intensities and operators additionally specify how to aggregate it over a requested interval. This is more closely aligned with repeated-event questions than retrieving a single relevant segment.
Semantics and time have distinct roles. Continuous marks retain visual meaning, type-specific intensities expose temporal statistics, and the answer model need not carry the whole computation. The reusable idea is this interface, not the assumption that every task should use a point process.
Frame and event budgets are separated. The two budgets govern observation and reasoning representation respectively, enabling a comparison between added memory structure and simply adding frames. Deployment costs must still include proposal generation, quadrature, and sampling.

Limitations & Future Work¶

Reproducibility evidence: The main text refers complete training settings and hyperparameter ablations to supplementary material, but the cache ends with the references and supplies no usable code link. Training data, random seeds, learning rates, and component gains remain unverified and should be established before reproduction.
Localization resolution and missed events: Event times are timestamp lookups at sampled centers, so brief or densely repeated actions may disappear before proposal generation. Continuous intensity cannot remove this observation bottleneck; boundary supervision, proposal recall, and count calibration deserve separate evaluation.
Statistics are not observations: Order uses first-arrival probabilities, while interval and duration depend on event matching. Missing events, repeated alternation, and overlapping states are not fully specified. Uncertainty, missing valid pairs, and abstention conditions should be explicit outputs.
The comparison does not establish universal leadership: Table 2 reports 57.1 overall for the reference Long-LLaVA model, above CTEPM's 55.0; these different systems are not controlled backbone comparisons. The evidence supports improving the selected backbone under matched budgets, not superiority over every video model.
Efficiency evidence remains incomplete: Compact evidence reduces answer-context occupancy but does not prove lower end-to-end latency, memory consumption, or energy use. Full runtime accounting, repeated-sampling variance, and stratified evaluation of parsing errors would strengthen the case.

MA-LMM: Online memory banks preserve historical information; CTEPM additionally defines executable temporal statistics. They could be complementary, with a memory bank retaining semantic detail and an event process handling numerical queries, but that combination is not an established result of this paper.
Video-EM / EventMemAgent: Episodic or hierarchical event memories organize retrieval, whereas CTEPM represents events through a stochastic process with intensities. The distinction is not whether events exist in memory, but whether they support explicitly defined temporal computation.
FastVID / FitPrune: These methods optimize visual token budgets without generally replacing temporal logic in question answering. Combining them with CTEPM is a research direction, but the effect of pruned information on event recall would need checking.
Hawkes and neural temporal point processes: These provide history-conditioned intensities and long-range dependency models, which CTEPM applies to video question-answering memory. Further work should examine semantic-type identifiability and intensity calibration rather than treating a signed interaction matrix as a real causal graph.

Rating¶

These are subjective ratings based on the available main text, on a 5-point scale.

Novelty: 4/5. Connecting continuous-time point processes to video temporal operators goes beyond event retrieval, although the underlying process tools are well established.
Experimental Thoroughness: 3/5. Multiple datasets and matched-budget controls are useful, but component ablations, training details, and cost evidence are insufficient in the available cache.
Writing Quality: 3/5. The overall argument is clear, but expected-count semantics and gain reporting need greater precision; some cached equations also have extraction defects.
Value: 4/5. The method offers a reusable interface for repeated events and temporal aggregation, with practical value dependent on event recall and operator reliability.