Skip to content

Research log

  • in Corpus
  • 4 min read

RQ1 on the study's own windows

The first run using the train and dev windows rather than a random holdout from one month. Both organizations, base plus both adapters on both dev windows, equalized training, one seed. datasets/results/rq1-windows.json. Completed in 2:19:50 against a 4:30 request; all seven outcome-neutral checks pass and the apparatus holds.

The censoring confound, after two bugs in the inputs

scripts/censoring.py, datasets/results/censoring.json. Snapshots select changes by last update; windows assign them by creation. A change created inside a window whose last update falls after the final collected month is absent entirely, and what disappears is the slow reviews. The loss cannot be counted, because a truncated change leaves no record; the lag from creation to last update can be, for changes whose lag was short enough to be seen. That is right truncation, and Lynden-Bell (1971) gives its nonparametric MLE.

The temporal probe is weaker than the design assumed

Zhang et al., Test of Time: Rethinking Temporal Signal of Benchmark Contamination (arXiv:2509.00072, ACL 2026), argue that post-cutoff performance decay, the signal the time-partition probe reads, is not dependable evidence of contamination. Their result is that the construction of the items distorts the temporal pattern independently of the source material: model-transformed questions and cloze questions drawn from the very same documents produce markedly different temporal signals.

What the base model's own report says about its training data

The contamination battery compares a post-cutoff window against a pre-cutoff control, so the cutoff is load-bearing and had never been sourced. Search returns "June 2024" for Qwen2.5-Coder with some confidence. It does not survive the primary source: in QwenLM/Qwen3 discussion 1093 a user asks whether the cutoff is June 2024, a maintainer pings two team members, neither answers, and a third participant offers June as a personal opinion. No Qwen maintainer states a date. It is a community guess with a citation-shaped shadow, and the battery must not rest on it.

  • in Corpus
  • 3 min read

Collection: our own volume got connections to opendev dropped

Building the train and dev windows stalled twice against review.opendev.org. DNS was healthy throughout (16 timed lookups, none failed). Failed requests completed the lookup and never completed the TCP handshake, including with the address pinned. After hours of unpaced building, at about 20 requests a second, a quarter to a third of new connections were dropped. After 20 minutes of sending nothing, 0 of 30 were. Qt's server showed no such pattern.

  • in Corpus
  • 2 min read

The seal right-censors the collected windows: dev loses about a fifth

Snapshots select merged changes by last update (Gerrit after:/before:), windows assign them by creation, and the seal refuses every month from 2025-11 on. So a change created in a collected window whose last update falls on or after 2025-11-01 is in no snapshot. The loss is concentrated in the changes reviewed longest, near the boundary.