The Merits of Conducting Surveys Inside North Korea
Commentary on Beyond Parallel’s A View Inside North Korea
The survey conducted inside North Korea on behalf of CSIS Beyond Parallel is noteworthy because it employs standardized questionnaires for response consistency and draws samples from a wide cross-section of the North Korean population. While the number of individuals included in the survey sample is only 36, they come from 8 provinces (out of 9 provinces plus the capital district, Pyongyang) and the age ranges from 28 to 80. The sample is composed of 16 females and 20 males.
The main merits of the study are that the survey took place inside North Korea, respondents were to free to voice critical opinions about their own government unfettered, and these opinions are current. Of course, conducting a public opinion survey in a totalitarian society is highly risky, and the risk is reflected on the very small sample size and the limited number of items that the researchers were able to ask in the survey. But why should the researchers put up with such limitations? What is the ultimate usefulness of such a limited survey?
Before answering these questions, let us first examine what would be the prerequisites of an ideal public opinion survey. The most important condition is that respondents are to be selected at random, i.e., through random sampling. What is required to randomly sample survey respondents from a population of interest is an index list of the population from which survey administrators can take random samples. Such an index list is often a phone book, or more fashionably these days, a computerized system that randomly dials phone numbers called Random Digital Dialing (RDD). Of course, this sort of randomization is impossible to carry out in North Korea, where even a phone book is apparently classified as “state secret.”
But the impossibility of carrying out random sampling is a fundamental challenge. The reason is because the lack of randomization can lead to all kinds of issues that can potentially prevent researchers from inferring objectively about what is happening on the ground. In practical terms, it means that the respondents in the survey may be acquaintances who share similar socio-economic backgrounds and political attitudes. Or, the respondents could be sampled from urban areas at the expense of rural areas because in most countries the bulk of the population is concentrated in urban areas. The beauty of random sampling is that when it is carried out correctly it reduces or even gets rid of the biases.
[T]his sort of randomization is impossible to carry out in North Korea, where even a phone book is apparently classified as “state secret.”
Going back to the survey conducted for CSIS, readers may see that the survey addresses some of the most apparent biases, such as the gender gap as well as geographic origins of the respondents. But was the apparent balancing a product of random sampling? The answer is very likely no, as it is apparent that respondents were not chosen at random and interviewers probably knew the respondents in some capacity. And the personal bond between the respondent and the interviewer, which would be anathema to public opinion surveys under more normal circumstances, is ironically the best guarantee of confidentiality and objectivity in the case of North Korea: The respondents can trust that the interviewers will protect the confidentiality of their responses, which in turn allows them to respond truthfully to the survey.
In a way, the survey designers have decided to trade objectivity for lower risk and cost of conducting the survey in North Korea. This approach is essentially “convenience sampling” as opposed to random or probability sampling. While the use of convenience sampling is often frowned upon because it does not address bias inherent in non-random selection of respondents the use of convenience sampling in North Korea is understandable given the risks involved. It does not mean that the current survey can be improved upon. The small sample size of only 36 individuals lowers the confidence that the inferences from the survey can be trusted to represent the true state of the North Korea society. While it does not render the survey result irrelevant, it is nonetheless a serious flaw that should be addressed in the next iteration. The current project’s use of convenience sampling cannot address the biases and the small sample size is a major issue. But it was expedient, current, and most importantly, ensured the safety of survey administrators and participants.
Representativeness Dilemma: Refugees vs. Convenience Sample
But the key question remains: does the survey result reflect the current mood in the North Korean society? Or, is the survey simply a set of opinions of 36 individuals? As discussed above, the use of convenience sampling does not reduce the inherent bias in the sample. What is more typically done by researchers is to survey the population of North Korean refugees in South Korea. The merits and demerits of the current survey of convenience sampling become clearer when compared to surveys of the refugee population.
One definite advantage of refugee population surveys is sample size. As of last August the refugee population in South Korea stood at 29,688, and is projected to surpass 30,000 mark before the end of the year. Theoretically speaking the entire population of North Korean refugees can be sampled, and researchers and government agencies in South Korea conduct regular surveys of the refugees. The sheer size of the population is extremely valuable resource for investigation. For instance, it is well known that the North Korean state did not allow the UN Commission of Inquiry on Human Rights in DPRK (COI-DPRK) to enter the country to conduct investigations on the ground. The intention behind the ban was probably to discredit COI-DPRK’s findings by preventing UN investigators from gathering facts from the ground. But COI-DPRK was still able to accomplish its aim by interviewing more than a hundred refugees residing in South Korea. It is safe to say that the COI-DPRK would not have been able to give the recommendation to take Kim Jong-un to the International Criminal Court had it not been for the North Korean refugees in South Korea.
But the sampling refugee population is also plagued with problems. The refugee population in South Korea is not geographically and socially representative of the general population in North Korea. According to the latest statistics from South Korean Ministry of Unification, 76% of the refugees in South Korea hailed from just two provinces in North Korea: North Hamgyong and Ryanggang, both of which border China. According to the 2008 census reported to the United Nations by North Korea, the population shares of these two provinces were only 10% and 3.1%, respectively. So there is clearly a very strong geographic bias in the refugee population.
Conducting a public opinion survey in a totalitarian society is highly risky, and the risk is reflected on the very small sample size and the limited number of items that the researchers were able to ask in the survey.
Still, one may wonder whether geography matters in a small country like North Korea. The answer is yes, mainly because in North Korea there is a large regional gap in economic development. Like the fictional place “Panem” in the movie “the Hunger Games”, the biggest gap is between the core regions and the periphery: in North Korea power and wealth are heavily concentrated in the capital, Pyongyang and its environs, and the rest suffer from permanent economic stagnation and shortages. For instance, nutrition surveys conducted by UNICEF showed that North Hamgyong and Ryanggang had the worst child malnutrition levels in the country during the 90s while in Pyongyang the situation was never acute even during the worst of the famine. This shows that the breakdown of the public distribution system (PDS) must have been more severe in these two provinces compared to the rest of the country. This explains not only why refugees leave their homes in the first place, but also why many of them report very high levels of market penetration in North Korea, which was probably in response to the breakdown of PDS. Overall, the collapse of the PDS and food shortages may have been especially severe in the Northeastern part of North Korea, which in turn may have negatively influenced the public perception of the regime in the region more so than the rest of the country.
Another issue with the refugee population is the gender ratio. More than 70% of the refugees in South Korea are female, due to the fact many female North Korean refugees are victims of human trafficking rings that either snatch them on the Chinese side of the border or smuggle the women across the border to be sold into forced marriages with Chinese men. Once out of North Korea, these women try desperately to reach South Korea. The gender imbalance is one of the consequences of this horrific dynamic.
These are serious challenges, but there are standard sampling techniques that address these issues. The most standard approach is to use statistical weights. By considering the refugee population in South Korea as “samples” of the general North Korean population, one can use the information in North Korea’s 2008 census to over/underweight geographic and gender biases in the refugee population. Just like the presidential polls in the US, where polling numbers are weighted by geography and demographic categories, refugees hailing from North Hamgyeong and Ryanggang provinces could be weighted down and overweight the rest in proportion to their share in the general population. Gender ratio can be adjusted as well using the census information.
What statistical weighting cannot do, however, is to address the recall bias inherent in the refugees. It is rare for refugees to arrive in South Korea within weeks of escaping from North Korea. Some spend years in China- and while in transit the live under constant threats of forced repatriation by the Chinese authorities. Once they arrive in South Korea their recollection of North Korea may no longer be current and could be biased. Stephen Haggard and Marcus Noland tried to address the recall bias by employing distance effects; they hypothesized that the closer the refugees are to North Korea, the less likely would the refugees be biased. As if to confirm their suspicion, their research on refugee population in China and South Korea showed significant differences between the two populations: for instance, North Korean refugees in South Korea were more likely to report that their reason for leaving was politically motivated than their counterparts in China (Chapter 2, Table 2.4): 94.7% of the refugees surveyed in China reported that their primary reason for leaving North Korea was economic, while only around 2% reported that political freedom was the main motivation. The corresponding figures for the refugees in South Korea were 56.7% and 27%. As refugees spend more time outside of North Korea, they become more aware of the exploitation and oppression prevailing back in their home country. In a word, North Korean refugees develop political consciousness and awareness of their human rights: a desirable outcome, but not for survey purposes.
Conclusion
Even after weighing for the socio-demographic factors, surveys of North Korean refugees are not exactly unbiased. Both convenience sampling (for example, survey within North Korea) and refugee surveys have respective advantages and disadvantages. The current survey result should not be regarded as a complete description of the public opinion in North Korea. It has too many caveats for it to be considered as such. Instead, the results from the current survey should be used as data points alongside other survey results, most of them based on samples of refugees. Fortunately, the current results do not contradict other survey results regarding market penetration and breakdown of the PDS in North Korea. While hampered by the small sample size, the current survey result gives some confidence that the recall bias inherent in refugee samples is not too severe. Therefore, a judicious combination of convenience sampling within North Korea and surveys of refugees in South Korea should provide the least distorted picture of the situation within North Korea that is available to researchers today.