So you've run your Exchange, survey or Interview with ThoughtExchange? Fantastic! What's next? You've got a treasure trove of participant data waiting to be explored!
However, navigating survey results can be daunting for many leaders, who often have to grapple with the challenge of interpreting these findings with impartiality and a basic level of statistical integrity. The good news is that we offer simple statistical principles that can bolster a leader's confidence in their results.
In this guide, we'll explore three critical facets of survey interpretation, equipping you with the tools you need to extract maximum value from data insights and turn them into actionable strategies.
- Crafting effective questions
- Understanding sample representativeness
- Evaluating statistical significance
- Additional: ThoughtExchange’s Favorability Matrix
Crafting Effective Questions
Confidence in results begins well before the engagement is launched. Well-thought-out questions have the power to yield reliable data, while inadequately formulated ones run the risk of introducing bias and ambiguity. Here are some tips for crafting effective questions:
Including demographic questions: Asking demographic questions is essential for understanding the surveyed sample’s characteristics and representativeness, which is discussed further in the next section. It is also necessary to compare responses across different groups to identify differences and commonalities between them. Common demographic questions include gender, ethnicity, role, department, school, and/or grade level.
Clarity and Neutrality: Craft clear, unbiased questions to elicit meaningful responses. Avoid leading or loaded questions that predispose respondents towards particular answers. Objectivity in question formulation maintains survey integrity.
Research-backed questions: Consider leveraging validated question templates or psychometric tests to ensure question validity and reliability for the highest level of confidence. For instance, ThoughtExchange offers a research-backed template for School Climate surveys, which was produced via a three-year study spearheaded by the American Institutes for Research, the U.S. Department of Education, and the National Center for Education Statistics.
Understanding Sample Representativeness
The participant sample plays a crucial role in the reliability of results. While larger samples provide more robust data, smaller samples can yield skewed or unreliable results. Here's how to navigate sample size considerations:
Significance of Sample Size: Understand the significance of sample size in interpreting survey results. Larger samples offer more representative data. Smaller samples pose a risk of unreliable outcomes.
Guidelines for Adequate Samples: While there's no one-size-fits-all answer, a commonly recognized guideline suggests aiming for at least 30 responses per group being evaluated to achieve a reasonable level of confidence in the findings. Furthermore, evaluating how well the sample’s size and distribution compares to the population’s can provide valuable context. For the highest level of confidence, a number of statistical tests can be performed using advanced software to ensure statistical significance. This is discussed further in the next section.
Example 1: A school climate survey’s results show only 10 responses from students in grades 9-12. While this might provide some insights, it's essential to recognize that such a small sample size is very unlikely to be fully representative of 9-12th graders’ opinions. Aim for a larger sample size to ensure more accurate and representative data.
Example 2: A school climate survey’s results show 200 responses from high school students. This number appears significant at first; however upon filtering further, it becomes apparent that nearly all of those students came from one of the district’s five high schools. This puts into question how representative the survey’s results would be for the district’s four other high schools.
Evaluating Statistical Significance
Statistical significance determines whether observed differences in survey responses are meaningful or due to chance. While advanced statistical tests provide the highest level of confidence, visual analysis, and external validation can also enhance interpretation:
Look for Clear Differences: Start by visually examining the charts for noticeable differences between groups or categories. Are there substantial variations in the heights of the bars or the proportions of different segments? Clear discrepancies may indicate potential areas of statistical significance.
Example: A survey includes several questions about peer-to-peer relationships. The majority of participants rate their relationships positively, except for the question, “On a scale of 1 to 10, how much do you feel your peers understand your personal interests and hobbies?” The results for this question are 30% lower than those of other questions, which suggests a potential area of statistical significance.
Look for Consistency: Compare the differences or the patterns observed across different demographics, topics, or survey questions. Consistent trends and patterns strengthen the evidence for statistical significance, whereas contradictory or erratic patterns may raise questions about the reliability of the findings.
Example: A survey’s results indicate a significant difference in the perception of safety between males and females. By visually examining the data across 5 safety related survey questions, it is apparent that male students consistently report higher scores in comparison with their female counterparts. This clear difference suggests a potential area of statistical significance worth further investigation.
Cross-sectional comparisons: Contrasting responses between demographic groups or segments of the surveyed population uncovers disparities or disparities in opinions, preferences, or experiences. Such insights inform targeted interventions or initiatives to address specific needs or concerns. As part of this analysis, it's important to consider the statistical significance of observed differences.
Time-based comparisons: Analyzing data from multiple survey administrations allows for the identification of evolving trends or changes in attitudes or behaviors over time. This facilitates proactive decision-making based on shifting dynamics.
Cross-Check with External Data: Where possible, cross-reference the survey results with external information, such as previous survey results and feedback, existing literature, or benchmark data. Consistency between your findings and established trends or norms lends credibility to your results and enhances confidence in their statistical significance.
Example: A school climate survey’s results indicate a significant increase in reports of bullying compared to previous years. You review incident reports from the school's disciplinary office and find a corresponding uptick in reported bullying incidents. This consistency between survey results and disciplinary records strengthens the evidence for the statistical significance of the observed increase in bullying.
Advanced tests: significance tests can be conducted using advanced statistics software to provide the highest level of confidence. The tests enable us to evaluate the likelihood that differences between groups or responses are statistically meaningful. This ensures that conclusions drawn from survey data are grounded in statistical rigor rather than random variation. These tests typically take into account factors such as the variation within results, the size of the sample, and the magnitude of the observed differences. By analyzing these inputs, statistical tests determine whether the observed differences are likely to occur by chance or represent genuine effects. This information helps researchers and analysts make informed decisions based on the strength and reliability of the evidence presented by the survey data.
ThoughtExchange’s Favorability Matrix
ThoughtExchange’s favorability matrix is an example of a tool that uses both cross-sectional comparison and advanced tests to provide statistical insights. This matrix measures whether a demographic group’s results are statistically different than that of the overall, not only via the actual score, which is based on a calculation of averages, but also via factors such as distribution of responses, their variance, and sample sizes. The exact calculations are proprietary and part of ThoughtExchange’s “secret sauce.”
The following scenario illustrates how this is useful: An organization’s HR team decides to gather feedback about safety in the workplace.
The overall favorability score across all teams is 75%, based on the distribution of answers below.
As the different teams delved into their individual results, the Finance team emerged with the same score as the overall (75%). However, a closer look revealed that the distribution of their responses was far different than that of the overall. There seemed to be a significant amount of polarity, with most of the answers awarded to “Strongly Agree” and “Strongly Disagree”. The takeaway is that while their favorability score was the same as the overall, the Finance team’s distribution of responses was statistically different than that of the Overall group.
The IT team, on the other hand, boasted a higher overall score of 79%. Yet, the results indicated that this higher average was not statistically different from the town's overall score. The reason? The IT team’s responses had a nearly identical span as the overall. They couldn't necessarily say with confidence that the team felt safer than the overall.
Finally, the marketing team celebrated an even higher overall score of 90%. In this case, the results indicated that the narrow range of responses—mostly “strongly agree”—suggested that we could say with far more certainty that the marketing team had a more consistent experience.
In the end, the teams understood that statistical significance went beyond the proximity of averages. This shed light on the importance of understanding the range of responses, helping each team dig deeper into their data and make better-informed decisions.
It's essential to acknowledge that while the above example illustrates the complexity of statistical comparison, there are a myriad of factors that can contribute to the differences between two samples. This tool's ultimate takeaway is that leaders can feel more confident that variations in favorability scores are statistically significant rather than mere chance.