2024-02-23
Lecture Friday: Intro to Empirical Software Engineering: What We Know We Don't Know
Here's a link to the talk webpage. This talk also apparently has a newer version but there's no video online.
I love calling out the logical fallacy, "Because we don't know how to do something, it can't be done."
This presentation has some useful results, but the real takeaway is how science actually works. You don't learn how the world works by reading a single paper. A paper is just one data point. The gold is in convergence: many studies, different designs and methodologies, all pointing to the same conclusion. Plenty of papers show contradictory results or suggest more complicated contextual factors. That's normal. You weigh the whole body of evidence.
Another critical point: a replication crisis doesn't mean the original findings are false. It means we no longer have strong evidence for them. This isn't pedantry. It's the difference between empiricism and theology. You never know something. You have a preponderance of evidence for or against it. A body of study that suggests something strongly or weakly. Non-replication is common. The original research wasn't necessarily bad, it was human research in a noisy universe. More study always improves confidence.
Beyond that, the big takeaways are that code reviews work, and sleep and stress have been shown to have tremendous impact on your abilities.