3 Problems with The Culture Test Results

I developed The Culture Test in 2013 to help people better understand cultures. The test is generally helpful in that regard—in five minutes people can get a snapshot of any culture. To help people learn about cultural differences, it is a good conversation start. 

However, The Culture Test, just like all other assessment tools, is not perfect. In this post I explain three issues with the results that I have noticed. As you view your personal results and use the Interactive Map, keep these in mind.

1. Quantifying “Culture”

Problem: The Culture Test purports to quantify a culture with three categories—guilt-innocence, shame-honor, and fear-power. While the simple framework offers some practical insights, we must acknowledge that “culture” in reality is far more complex and nuanced than, say, “62% guilt, 28% shame, 10% fear.” This three-number summary of a particular culture must be taken with a grain of salt. Any attempt to “quantify culture” easily becomes overly simplistic, much like personality tests. For example, my Myers-Briggs profile is INTJ. Those four letters may describe aspects of my personality, but they do not define me as a person. The same applies to The Culture Test.

Solution: Read results from The Culture Test with fair expectations. The results for individual countries should not be woodenly interpreted as absolute values. So if your country is 70% guilt, that does not mean you make seven out of ten decisions based on guilt values. Rather, the results should be interpreted relative to other countries. So, for example, Germany is 20% shame, while Albania is 66%. So we can conclude from these scores that Albania, compared to Germany, is strongly influenced my shame.

2. Accuracy of Answers

Problem: Nearly 30,000 people have answered the 32 questions (25 cultural + 7 personal) on The Culture Test. Of these nearly 1,000,000 total answers, I have not stood over the shoulder of a single person to verify the accuracy of their answers. Though I tried to make the test as simple and clear as possible for people to take by themselves, surely some people have misunderstood a question or entered an incorrect answer. Here are some examples of how answers might be incorrect:

• There are several results for North Korea, but I wonder if people simply clicked the wrong option on the dropdown menu.
• For the final question about the respondent’s age, I wonder, “What percentage of people enter their correct age on online forms?” I know it is less than 100%.
• I suspect people have misread or misinterpreted some of the cultural questions.
• Even if people answered the question “correctly” it may still be “wrong” simply because the test in in English. For example, a twenty-year-old Japanese person may answer the questions based how they act in their English classroom (because the English-language questions evoke that context), not how they interact with their Japanese relatives in the rest of their life. (Note: the The Culture Test is also available in Spanish, so this may not apply as much in Latin contexts). 

Solution: Though I have not calculated the margin of error, it certainly does exist. The best solution to this problem is to have a global team translate and personally administer The Culture Test in all global cultures. I personally do not have a budget for that, so let me know if you do! Until then, assume a margin of error.

3. The Test-Takers Are Not Random, So The Results Are Not Representative

Problem: The Global Map of Culture Types visualizes the primary culture type of every country in the world. However, the respondents were not randomly selected, so they may not fairly represent the entire demographic of that country. Three examples illustrate this point:

1. Zimbabwe: The average score of 44 results from Zimbabwe is 52 guilt/38 shame/9 fear. Compared to other African countries, this score is strongly guilt-oriented. But a closer look explains the reason. Because of Zimbabwe colonial history, there are effectively two cultures in the country. Zimbabweans who self-identified as “African,” or “Shano,” or “colored” (20 results) averaged 37/51/12—a score much closer to other African countries. However, Zimbabweans who self-identified as “White” or “Anglo” (22 results) scored on average 66/28/6—a score close to Europeans and Western countries. Since about the same number of people from each group took The Culture Test, the average score that is visualized on the map for Zimbabwe splits the difference between these two cultural groups.

However, white/Anglo Africans are only 0.2% of Zimbabwe’s population; yet they represent over half of the results in the data. The 44 people who took The Culture Test for Zimbabwe do not accurately represent the countries racial demographics.

2. China: For Chinese culture, the average score from people born and raised in China (“Culture Type: Primary”) is 47 guilt/ 37 shame/ 16 fear. But among expatriates who have lived in China for 10+ years, Chinese culture was 27 guilt/ 55 shame/ 18 fear. This is a swing of ~20 percentage points from between guilt and shame.

Here is my hypothesis for this discrepancy. The Chinese nationals who took The Culture Test obviously knew English. From this, we could assume they are educated, and most likely urban, younger, and from the East coast. Moreover, they probably learned about this resource from another Christian (likely a Western Christian), suggesting they themselves are Christian and connected with Western Christians. If these deductions are generally true, then the Chinese nationals who took The Culture Test represent a narrow segment of the overall population, and are therefore not an accurate snapshot of “Chinese culture” as a whole.

However, I suspect that the responses from expatriates who have lived in China for 10+ years might better represent China as a whole. This is not because they know the culture better, but because they are more dispersed throughout Chinese society, so their answers are based on a broader set of cultural experiences and social groups.

3. UAE: Of the 30 people in United Arab Emirates who took The Culture Test, only nine answered the questions for “Arab” culture. The other twenty-one are Western, African, or Asian expatriates. This result obviously slants the average score for UAE.

The un-random and un-representative nature of the results is perhaps the biggest shortcoming of The Culture Test. Most people have learned about The Culture Test in one of three ways: reading The 3D Gospel, visiting HonorShame.com, or attending a class where it was assigned. This suggests the majority of people who have taken The Culture Test are Christians engaged in ministry or interested in global cultures. Such a particular social group hardly represents the whole of a society.

Conclusion

These issues, IMO, do not render all the results invalid or useless, but helps us understand how much weight we can rightly attribute to the results. The results and the data visualization map are not perfect, but hopefully they serve as first steps towards helping us think through the issues. By making the anonymous data available for others to analyze, I hope further insights will emerge. 

Read more posts in this series “Guilt-Shame-Fear: Revisited“.