3 Problems with The Culture Test Results

I developed The Culture Test in 2013 to help people better understand cultures. The test is generally helpful in that regard—in five minutes people can get a snapshot of any culture. To help people learn about cultural differences, it is a good conversation start. 

However, The Culture Test, just like all other assessment tools, is not perfect. In this post I explain three issues with the results that I have noticed. As you view your personal results and use the Interactive Map, keep these in mind.

1. Quantifying “Culture”

Problem: The Culture Test purports to quantify a culture with three categories—guilt-innocence, shame-honor, and fear-power. While the simple framework offers some practical insights, we must acknowledge that “culture” in reality is far more complex and nuanced than, say, “62% guilt, 28% shame, 10% fear.” This three-number summary of a particular culture must be taken with a grain of salt. Any attempt to “quantify culture” easily becomes overly simplistic, much like personality tests. For example, my Myers-Briggs profile is INTJ. Those four letters may describe aspects of my personality, but they do not define me as a person. The same applies to The Culture Test.

Solution: Read results from The Culture Test with fair expectations. The results for individual countries should not be woodenly interpreted as absolute values. So if your country is 70% guilt, that does not mean you make seven out of ten decisions based on guilt values. Rather, the results should be interpreted relative to other countries. So, for example, Germany is 20% shame, while Albania is 66%. So we can conclude from these scores that Albania, compared to Germany, is strongly influenced my shame.

2. Accuracy of Answers

Problem: Nearly 30,000 people have answered the 32 questions (25 cultural + 7 personal) on The Culture Test. Of these nearly 1,000,000 total answers, I have not stood over the shoulder of a single person to verify the accuracy of their answers. Though I tried to make the test as simple and clear as possible for people to take by themselves, surely some people have misunderstood a question or entered an incorrect answer. Here are some examples of how answers might be incorrect:

  • There are several results for North Korea, but I wonder if people simply clicked the wrong option on the dropdown menu.
  • For the final question about the respondent’s age, I wonder, “What percentage of people enter their correct age on online forms?” I know it is less than 100%.
  • I suspect people have misread or misinterpreted some of the cultural questions.
  • Even if people answered the question “correctly” it may still be “wrong” simply because the test in in English. For example, a twenty-year-old Japanese person may answer the questions based how they act in their English classroom (because the English-language questions evoke that context), not how they interact with their Japanese relatives in the rest of their life. (Note: the The Culture Test is also available in Spanish, so this may not apply as much in Latin contexts). 

Solution: Though I have not calculated the margin of error, it certainly does exist. The best solution to this problem is to have a global team translate and personally administer The Culture Test in all global cultures. I personally do not have a budget for that, so let me know if you do! Until then, assume a margin of error.

3. The Test-Takers Are Not Random, So The Results Are Not Representative

Problem: The Global Map of Culture Types visualizes the primary culture type of every country in the world. However, the respondents were not randomly selected, so they may not fairly represent the entire demographic of that country. Three examples illustrate this point:

1. Zimbabwe: The average score of 44 results from Zimbabwe is 52 guilt/38 shame/9 fear. Compared to other African countries, this score is strongly guilt-oriented. But a closer look explains the reason. Because of Zimbabwe colonial history, there are effectively two cultures in the country. Zimbabweans who self-identified as “African,” or “Shano,” or “colored” (20 results) averaged 37/51/12—a score much closer to other African countries. However, Zimbabweans who self-identified as “White” or “Anglo” (22 results) scored on average 66/28/6—a score close to Europeans and Western countries. Since about the same number of people from each group took The Culture Test, the average score that is visualized on the map for Zimbabwe splits the difference between these two cultural groups.

However, white/Anglo Africans are only 0.2% of Zimbabwe’s population; yet they represent over half of the results in the data. The 44 people who took The Culture Test for Zimbabwe do not accurately represent the countries racial demographics.

2. China: For Chinese culture, the average score from people born and raised in China (“Culture Type: Primary”) is 47 guilt/ 37 shame/ 16 fear. But among expatriates who have lived in China for 10+ years, Chinese culture was 27 guilt/ 55 shame/ 18 fear. This is a swing of ~20 percentage points from between guilt and shame.

Here is my hypothesis for this discrepancy. The Chinese nationals who took The Culture Test obviously knew English. From this, we could assume they are educated, and most likely urban, younger, and from the East coast. Moreover, they probably learned about this resource from another Christian (likely a Western Christian), suggesting they themselves are Christian and connected with Western Christians. If these deductions are generally true, then the Chinese nationals who took The Culture Test represent a narrow segment of the overall population, and are therefore not an accurate snapshot of “Chinese culture” as a whole.

However, I suspect that the responses from expatriates who have lived in China for 10+ years might better represent China as a whole. This is not because they know the culture better, but because they are more dispersed throughout Chinese society, so their answers are based on a broader set of cultural experiences and social groups.

3. UAE: Of the 30 people in United Arab Emirates who took The Culture Test, only nine answered the questions for “Arab” culture. The other twenty-one are Western, African, or Asian expatriates. This result obviously slants the average score for UAE.

The un-random and un-representative nature of the results is perhaps the biggest shortcoming of The Culture Test. Most people have learned about The Culture Test in one of three ways: reading The 3D Gospel, visiting HonorShame.com, or attending a class where it was assigned. This suggests the majority of people who have taken The Culture Test are Christians engaged in ministry or interested in global cultures. Such a particular social group hardly represents the whole of a society.

Conclusion

These issues, IMO, do not render all the results invalid or useless, but helps us understand how much weight we can rightly attribute to the results. The results and the data visualization map are not perfect, but hopefully they serve as first steps towards helping us think through the issues. By making the anonymous data available for others to analyze, I hope further insights will emerge. 

Read more posts in this series “Guilt-Shame-Fear: Revisited“.

 

resources for Majority World ministry

7 Comments on “3 Problems with The Culture Test Results

  1. Thanks for this post – living and serving in Africa, and having done TheCultureTest with quite a number of people, I wasn’t entirely surprised to see the results that came out of Zimbabwe and also South Africa indicating a ‘guilt’ predominance – Even though it doesn’t reflect the entire population it does indicate the reality of the cultural tensions.
    As in Zimbabwean Shona ethnicity results which show a shame emphasis, if you do a search for Zulu ethnicity results in South Africa, a shame emphasis will emerge.
    But both countries have been highly colonised and I have found that even non ‘whites’ from these two countries can return a significant ‘guilt’ percentage (though not necessarily a majority%), especially if they are from an urban or educated background. This doesn’t necessarily mean they function with an even balanced approach to life all-round, but that they can be more ‘guilt’ oriented in certain urban/business contexts and more ‘shame’ oriented in certain family/home/village and day-to-day life contexts.
    Despite the shortcomings of TheCultureTest, I have found it immensely helpful and a super tool in usefulness of both the test itself and the results. It has opened up amazing conversation as I have shared the results with others on this continent – in helping both expats and locals understand cultural emphasis differences.
    THANK YOU SO MUCH

  2. Jayson, I really appreciate you delineating the limits of something that you have worked so hard to develop and promote. When I used TheCultureTest in my Haitian Bible School classroom, the results from my students and from their own surveys were highly skewed toward guilt, partly because my students are more educated in Scripture and science and partly because the people they surveyed tended to give them the answers they knew they wanted to hear. Although Haiti is culturally homogenous, there are significant differences in mindset based on religion (Catholic, Protestant, or Voodoo). Someday, I’d like to do a more thorough study since it affects evangelistic and discipleship approaches.

    All that being said, Jayson, my students found TheCultureTest not only interesting, but usefull and one used it in his degree thesis about effective children’s ministry in a local slum. Kudos!

  3. In regard to the comments on Zimbabwe, we also had some confusion here in Mozambique when a group took the test. Should they have answered for the traditional culture or should they have answered for their “culture” as Christians? The test took a long time to take because of the tension they felt between the two.

  4. Thanks for acknowledging the limitations of your tool. In the end, I think this makes it even more useful because people have a fair understanding of what the results do or do not represent.

  5. This post is very helpful and important. You have done an excellent job in discerning the outcomes of the test results. And you have been very thorough in seeking out why the problems occured.
    Together with the observations of this article, The Culture Test is giving us some valuable insight.
    Thanks a lot, Jayson!

  6. One limitation that you possibly should highlight as a fourth point is that your dataset is actually quite small. Can we really draw conclusions from 11 people allocated to Iran? For the larger dataset of 70 in Afghanistan (which has high levels of illiteracy) can we really conclude that 69% of the population went to University? Doubtful. Which makes the results doubtful too, sadly.
    A great exercise though.

    • Jason,

      Good points, thanks for sharing those numbers. I agree that dataset for many countries is insufficient. About 500 people take The Culture Test per month, so hopefully overtime those gaps will get filled in.

Leave a Reply

Your email address will not be published.

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.