Analysis, Human Rights

Considerations of Risk and Methodology for North Korean Surveys

Commentary on Beyond Parallel’s “A View Inside North Korea”

It is no secret that getting anything meaningful in the way of data out of North Korea can be difficult. Analysts and experts rely on tidbits from regime broadcasts, defectors, and anecdotes from those who travel to the country. North Koreans living in the borderlands with China and those who have managed to escape further afield continue to form valuable resources. But actually talking to North Koreans living in North Korea seems to be nearly impossible. The interviews gathered by CSIS as a part of this report are a step in the right direction.

This is not to say that the work here is problem free. Working in circumstances as difficult as these requires that normal methodological rigor be put aside. The very serious risks assumed by everyone involved demand it.

The work here is clearly not a survey. The non-probability sample means that it is largely impossible to calculate a margin of error. It is also not a focus group. The interviews are done in a one-on-one setting between people who may or may not be familiar with one another. Reporting percentages—a hallmark of polling work—is also problematic. All of these issues will almost certainly mean the work will come under scrutiny for its shortcomings—and that scrutiny should be welcomed.

But such scrutiny should not lead to the dismissal of the work altogether. The efforts here are initial steps in a broader effort to help us better understand just what is happening in the everyday lives of those living under the harshest regime on the planet.

Instead, the responses offered here should be treated cautiously, taking care to place them within the greater body of knowledge that is slowly being established about life in North Korea. The findings in and of themselves—that North Koreans either complain or make jokes about the North Korean regime—may seem predictable. But a question such as this functions more akin to a control question. It helps to confirm that the responses being gathered during the interviews conform to expectations, and help those administering the interview establish a base level of confidence in the initial findings. The questions that come in subsequent waves will hopefully expand on these findings, adding another level of nuance to how we understand the personal experience of life in North Korea.