Author runs a business analytics company and is involved in economic/political thinktanks.
These notes are really sparse! There is a lot of interesting detail in the book.
Structure of science
Much in common with The Beginning of Infinity. Induction doesn’t work, competition between explanations etc.
Paradigm shift. Without any background assumptions it’s hard to even determine what has been falsified. Paradigm dictates how to test theories and how to interpret observations.
Branching hierarchy of paradigms, from broad trunks like falsification to narrow twigs like FMRI clustering.
Science itself is a paradigm which has competitors - common sense, tradition etc. Each embodies implicit learned/evolved knowledge about the world and has different mechanisms that generate, change and select ideas.
Rise of randomized experiments
Forces against MRA and similar post-hoc analysis:
- Missing variables
- Interaction effects - number of possible interactions quickly begins to dwarf data-points
- Crud factor
Meta-analysis by Ioannidis of 49 influential studies. 80% of non-randomized studies found weaker or no effect vs 10% of ‘large’ randomized studies.
Meritful criticisms of randomized trials in medicine:
- Encourages focus on quantitative outcomes over qualitative outcomes eg cured/not-cured vs livable/not-livable in mental illness.
- Holistic integration in human body - appropriate measure for any experiment should be entire life quality. Eg antibiotics might cure acute illnesses but increase long-term risk of chronic illness.
- Assumption of generalizable response vs eg genetic variation in response to medication.
Fall of randomized experiments
Causal density - density of direct causal relationships between variables. Crud factor = correlation. Causal density = causation. Can we distinguish them?
Holistic integration - everything interacts with everything else, so separate experiments cannot be performed. Eg in macro-economics we can’t run an experiment on one country and use another country as a control if those two countries interact heavily.
With high causal density but low holistic integration, we can still run experiments on some groups and use others as controls, but analysis is hard.
With high holistic integration but low causal density, we can get by with post-hoc statistical controls but we can’t create randomized controls and replication is hard.
Problems like social policy and macro-economics display high causal density and high holistic integration.
Internal validity - did the treatment work in the sample. External validity - will similar treatments work in similar samples ie to what extent can we generalize this result.
Problem of generalization: given high causal density, it’s very hard to determine the extent of external validity.
The experimental revolution is like a huge wave that has lost power as it moved uphill through topics of increasing complexity and holism. Physics was entirely transformed; therapeutic biology required statistical experimentation due to higher causal density but could often rely on the assumption of uniform biological response to reliably generalize findings from randomized trials; the yet-higher causal densities in social science make generalization from even properly randomized experiments hazardous, and integrated complexity of certain topics in social science appears to be fully impervious to experimentation.
Attack this on many fronts:
- Many replications, both direct and conceptual.
- Match trial conditions as much as possible to intended real-world conditions eg run medical trials under typical clinical conditions.
- Incorporate broader theories eg identify the mechanism for a given drug.
Lacking a powerful compact theory, we resort to many specialized micro-theories.
(Why does astronomy work? No ability to perform true experiments but: Low causal density. Low holistic integration. Universe is really big. Means that there are many good natural experiments. Also, much of the underlying laws are universal and can be separately tested in controlled experiments.)
Describes the rise of experimental methods of business.
Jam experiment - hazards of over-generalization. Single experiment on single day in single store. Many, many failed replications that never reached the public awareness.
In practice, businesses grapple with high causal density by repeated iterative testing (similar to pharma funnels):
- Explore (analyze data, generate hypotheses)
- Confirm (randomized controlled experiments)
- Generalize (test across wide range of background conditions)
- Implement (as an experiment eg hold out 5% of stores as control)
- Maintain (keep running tests in case environment changes eg new competition, different economic climate)
ie using experiments not to discover some global immutable truth, but as part of an optimization process.
- High-level management must publicly commit to the idea that experience and intuition are not enough, and promote epistemic humility.
- Create a distinct organization that designs and interprets experiments but does NOT generate theories. Reduces emotional attachment/bias.
- Build the experimental process into existing infrastructure. Reduces transaction costs, increases match between experiment and production envs, makes blinding easier.
Authors rule of thumb: experimental methods > experienced expert > non-experimental methods > inexperienced ‘expert’
Compared to medicine and business, social policy has a dearth of experiments. Discusses what decent work exists.
- The vast majority of proposed social policies do not work when tested under controlled conditions.
- Policies which attempt to change people fail more often that policies which attempt to change incentives / environments.
- There is no magic. High causal density can’t be modeled away. There will be no simple general theories.
Liberty-as-goal vs liberty-as-means. Former wants liberty as a fundamental value. Latter sees liberty as a useful mechanism for designing systems that learn and improve over time eg markets, science, experiments within business all rely on freedom to try new ideas combined with a framework for weeding out bad ideas.
Liberty-as-means implies that there are valid reasons for restricting liberty eg to leverage economies of scale of collective action (eg taxation), to reduce transaction overheads (eg trade standards, fraud laws) or to prevent gaming of selection mechanism (eg anti-monopoly / anti-collusion laws).
Last chapter gives a series of concrete suggestions for US policy.