Can theories be tested? A cryptographic treatment of forecast testing

How do we test if a weather forecaster actually knows something about whether it will rain or not? Intuitively, a "good" forecast test should be complete - namely, a forecaster knowing the distribution of Nature should be able to pass the test with high probability, and sound - an uninformed forecaster should only be able to pass the test with small probability. We provide a comprehensive cryptographic study of the feasibility of complete and sound forecast testing, introducing various notions of both completeness and soundness, inspired by the literature on interactive proofs. Our main technical result is an incompleteness theorem for our most basic notion of computationally sound and complete forecast testing: If Nature is implemented by a polynomial-time algorithm, then every complete polynomial-time test can be passed by a completely uninformed polynomial-time forecaster (i.e., a computationally-bounded "charlatan") with high probability. We additionally study alternative notions of soundness and completeness and present both positive and negative results for these notions.