Skip to content

Half the "review comments" are not reviews

Measured on live OpenStack data, 62 code-file comments:

authored by the change owner 32 (52%)
carrying in_reply_to 34 (55%)
message is literally "Done" 15

These are the author acknowledging a fix, not an instruction to make one. Feeding them to the model pollutes the input and leaks the answer: "Done" says the edit was applied, which is what the model is supposed to produce. is_reviewer_comment drops comments whose author is the change owner.

Two bugs found while fixing this, both of the silent kind.

One example per comment. The spec pairs a hunk with the comments anchored inside it, plural. Emitting one example per comment produced identical before/after rows that dedup then discarded as exact duplicates, losing every comment but the first. The tell was a 64% exact-duplicate rate; after grouping by hunk it is 16%.

A filter that matched nothing. The author filter dropped exactly 0 comments on real data while 73 acknowledgements sailed through. The snapshot's owner had been scrubbed to a 12-hex string at fetch, while the comments endpoint returns a raw integer account id, so the comparison was str != int and always true. The unit tests passed because they used consistent ids on both sides. is_reviewer_comment now raises TypeError when the two sides disagree in type, which converts a silent no-op into a loud failure; verified by running it against the real pipeline and watching the guard fire.

Fixed, and measured. corpus/fetchers.py supplies a comment fetcher that scrubs with the same salt as the change fetch, so both sides are pseudonyms. On 584 real changes: 278 author comments dropped, against 0 before, and acknowledgements surviving into the model input fell from 73 to 5. The residual 5 are comments carrying no author object, which are kept deliberately rather than guessed at. The diff fetcher does not scrub: a diff carries no account objects, and running the identity sweep over it would rewrite anything in the source that merely looks like an email address, corrupting the code under study.