I paid someone to attack my own paper

The best table in my paper wasn't a measurement. I had written a sentence saying it was.

Last week I wrote a line that has been bothering me since.

Nobody was going to catch that for me, because from inside any one of those four-second choices, there is nothing to catch.

If that's true, and I think it is, then care doesn't help. Care comes from the same place the blindness lives.

So here is the most reliable thing I've found.

I paid someone to attack my paper before I submitted it to a journal.

The table

The mathematics doesn't matter much for this story. It was about how much information you actually gain from each test in a sieve. Not many people care about that. I'm one of them.

What matters is the table.

Twelve rows. Twelve moduli, each with a number beside it. It sat in the middle of the paper and did the paper's main job. It was the evidence.

It was also the thing I was proudest of. I want that on the record early, because it turns out to be the mechanism and not a detail.

I'd looked at that table a lot. Rearranged it. Thought about which column should come first. When I imagined someone reading the paper, I imagined them stopping there.

A first version was already public as a preprint. What I hadn't done was send it anywhere that would put it in front of referees.

What it found

So I bought the report myself. Not a friendly read. An attack. Find what's wrong with this before a stranger does.

What came back was that the twelve numbers had not been measured. They were the output of a model I'd built. And the paper said, in plain language, that they were direct measurements.

The thermometer and the forecast

Here's the distinction, without the mathematics.

A thermometer on your porch says it's 72 degrees. A weather model says it's going to be 72 degrees. Both give you the number 72. Both are useful. They are not the same kind of thing.

My table was twelve forecasts. The sentence underneath it called them thermometer readings.

Not because I decided to say that. Because nothing in the file said otherwise.

The numbers were made at one point and described later. By the time I wrote the description, the thing I was describing had become just "the table" — an object with a shape and a place in the argument, and nothing attached to it saying where it came from.

The reviewer had one advantage over me. They hadn't made it. To them it was just a table, so the obvious question — where did these come from? — was still available. I didn't ask it. I already had an answer, and the answer had stopped being a fact and become a feeling of familiarity.

Psychologists call this the curse of knowledge. Once you know a thing, you can't reconstruct not knowing it. I'd only ever met it as an explanation for bad writing. This is the first time it cost me a claim.

The part I'd rather not write

I didn't check where that table came from because I was proud of it.

That's the whole explanation. I've looked for a better one.

Checking is what you do to a claim you're unsure of. That table didn't feel unsure. It felt like the strongest thing I had. Verification goes where the doubt is, and I'd put my doubt everywhere else — the derivations, the constants, the edge cases. Those were the parts that worried me.

The thing I was proudest of became the thing I examined least. Pride had quietly displaced doubt.

I suspect that's a common shape of error. I can only tell you it was mine.

Then I nearly did it again

The correction sent me back to measure properly. I mostly couldn't.

At the sample size I was running, direct estimation only works where there are few enough candidates to estimate from. That ruled out most of the table. I managed four of the twelve.

Here is everything the corrected paper reports about how the model did.

Let me be exact, because this is the paragraph where I'd most like not to be. All four measurements are in the revised paper. What the paper states as an explicit model-versus-measurement comparison is two of them — the two above. Eight rows I have never checked against anything.

Now watch what I wanted to write instead.

I wanted to write: the model turned out to be right to within one percent.

There is a true result inside that sentence. And the sentence creates a false impression.

It is the same error, one floor up — quoting the number that flatters me, staying quiet about the one that doesn't, and saying nothing at all about the rows nobody has ever checked.

I caught it only because I'd spent the previous thousand words on exactly this.

The reviewer didn't show that my numbers were false. They showed that I hadn't earned the claim I made for them.

The numbers weren't the problem. The status I'd given them was.

Those two things feel identical when you're the one being corrected. Every instinct says: but the answer looks about right, so what's the problem?

The problem is that "about right" wasn't something I could know when I claimed it. I had a model's output and a sentence calling it a measurement. If the model had been badly wrong, the paper would have been wrong in exactly the same way, by the same process, with the same amount of care behind it.

A claim isn't good because it later turns out to be roughly true. It's good because of what stands behind it at the moment you make it.

What it bought

Fatal to the headline claim as I had stated it — that's the verdict in my own records, and I'll stand by the word. But the relabelling was small. One table, described honestly. One sentence of provenance saying what the numbers actually were.

What that small change forced was not small. Version 2 says what the table is, adds the direct measurements I was finally made to do, reports how closely the model tracks them where both exist, and works out where direct estimation is possible at all. It also carries a derivation I hadn't done. It's a bigger paper than the one I nearly submitted, and a sounder one.

The correction is in the abstract, not buried at the back. That was the point.

Buying the attack before a stranger delivers it free is the cheapest insurance in this line of work. It costs less than the reputation you spend defending a claim you should have withdrawn.

What survived

The paper, corrected. And two habits.

Label every number the moment it exists.

• Measured
• Derived
• Modelled

Not in the paper — in my own notes, at the instant the number is created. The confusion doesn't happen at publication. By then it's far too late, and the sentence you're writing describes something you no longer remember making. It happens weeks earlier, when a number goes into a file with nothing attached saying where it came from.

Provenance decays. Nobody tells you that.

And the second habit, which matters more: bring in someone who didn't make the thing.

Working alone, you can't fully manufacture the view of someone who didn't build the thing. Techniques help — the labelling rule above would have caught this one, and I use it now — but they run on the same equipment that produced the blind spot, and this one went straight past everything I had. Eventually you have to hand it to someone with no memory of making any of it. If that's hard to arrange, pay for it.

Then point them at the part you love. Everything else already has your attention.

So: which of your numbers is a forecast you've started treating as a thermometer?

It won't be the one you're worried about.