Back

@AeonCypher @paninid

"A p-value is an #estimate of p(Data | Null Hypothesis). " – not correct. A p-value is an estimate of

p(Data or other imagined data | Null Hypothesis)

so not even just of the actual data you have. Which is why p-values depend on your stopping rule (and do not satisfy the "likelihood principle"). In this regard, see Jeffreys's quote below.

Imagine you design an experiment this way: "I'll test 10 subjects, and in the meantime I apply for a grant. At the time the 10th subject is tested, I'll know my application's outcome. If the outcome is positive, I'll test 10 more subjects; if it isn't, I'll stop". Not an unrealistic situation.

With this stopping rule, your p-value will depend on the probability that you get the grant. This is not a joke.

"*What the use of P implies, therefore, is that a hypothesis that may be true may be rejected because it has not predicted observable results that have not occurred.* This seems a remarkable procedure. On the face of it the fact that such results have not occurred might more reasonably be taken as evidence for the law, not against it." – H. Jeffreys, "Theory of Probability" § VII.7.2 (emphasis in the original) <https://doi.org/10.1093/oso/9780198503682.001.0001>.