Note: Not all features are available on all accounts. If you would like to learn more about any of these features, please reach out to your ThoughtExchange representative.
Compare survey results across multiple engagements
The Advanced survey question comparison tool lets leaders compare responses to questions from an engagement (or, depending on the features available in your account package, across multiple engagements). The ability to display multiple question results in one display makes it easier for leaders to perform advanced qualitative analysis on survey question responses and identify outliers or responses that are significant.
This tool is accessible via the 'Compare' tab of Survey analysis in your Results Dashboard and provides an easy way to visualize:
- Satisfaction scores by demographic answers
- Favorability scores by demographic answers
- Demographic group sizes
- Comparing survey results across multiple engagements
Choosing the Options
The Advanced Survey Question Comparison Tool is currently available under the Compare tab in the Survey Analysis section for customers with Engagement360 access. There is a 'Compare' dropdown menu that allows leaders to customize comparisons.
Use the ‘Compare’ menu to select the type of questions you want to compare with your demographic questions.
- Favorability score questions ask participants to respond to a question or statement using a 5-point Likert scale often indicating agreement or alignment (ex. strongly disagree, disagree, neither agree nor disagree, agree, strongly agree). This comparison will show how groups differed from one another in their levels of agreement.
- Satisfaction score questions ask participants to rate their satisfaction on a 10 point scale. This comparison will display how demographics groups compared to one another in their degrees of satisfaction.
- Demographic group sizes provide a simple comparison of the number of participants who responded with each possible option to closed-ended survey questions other than Satisfaction score or Likert-type questions.
- If you chose Satisfaction or Favorability scores in the ‘Compare’ menu, the ‘And these Likert Scale/Satisfaction score questions’ dropdown will appear. This is where you can select the relevant questions whose scores you will be comparing across demographic groups.
- The optional ‘Filtered by’ menu allows you to focus your analysis on participants who:
- responded in a given way to any of your demographic questions.
- left the survey before completing all the questions (partial responses)
Reading the Results
The results of your chosen comparisons are displayed according to the type of comparison being conducted (each is described below). The visualization of each comparison can be exported as a .png file using the ‘Export’ button above and to the right of the visualization.
Favorability Scores
Favorability score comparisons show the percentage of participants in each category of the demographic question who responded with the options indicated as ‘favorable’ in the ‘Configure Favorability’ menu. The tool compares these proportions to the ‘Overall’ favorability score (which combines the responses from all participants). Groups with a green or dark green treatment demonstrated favorability scores that were higher than the overall group and groups with a red or dark red treatment demonstrated favorability scores that were lower than the overall group. Results with an asterisk (*) indicate a statistically significant difference from the overall score (see section below about tests of statistical significance for more information).
Note: Results are hidden for demographic groups with fewer than 5 participants to protect participant identities.
-
If you chose Favorability scores:
- You will also see an option to include data from other engagements you have hosted. If you would like to compare data from another engagement, set the toggle for 'Include another engagement' to ON and choose the engagement in the 'Select another engagement' dropdown menu.
- If you have also set up Survey Topics (groups of related questions) in your engagements, you will also select whether you want to compare individual questions or the topics (which will display the favourability scores for all questions in the topic).
-
Once "Other engagement" has been selected, the question dropdowns will only include questions that match. Questions have to be worded the same way across both engagements.
Using the 'Other engagements' dropdown to compare favorability scores across multiple engagements provides a clear visual of how participants’ responses have changed over time. This tool is particularly useful for tracking trends, as shown in the School Climate Engagement Survey example in the screenshot above.
The tables in the picture show that the overall favorability score in 2024 is, on average, 3 points higher than in 2023. However, responses to the question, 'All students are treated the same, regardless of whether their parents are rich or poor,' show that we score 5 points lower than in 2023, indicating that participants feel more negative sentiments around this topic in 2024 compared to 2023.
In this vein, you can explore a detailed breakdown of responses by demographics across different years.
You can choose to compare favorability by Individual Questions or Survey Topics:
- Going back to Survey Responses tab to ‘Configure Favorability’ allows you to select the responses to each of your Likert-type questions that you consider favorable or positive for this comparison (sometimes, based on the way questions are posed, ‘disagree’ could be considered a favorable response rather than ‘agree’). Simply click the checkboxes next to the favorable responses for each question and click Save.
Survey Topics
If you chose to compare responses to Survey Topics rather than individual questions, you will see the topic heading as well as the favorability scores assigned by the various demographic groups in the questions you selected. You can also click the topic(s) name to see the scores for each individual question within the topic(s).
Satisfaction Scores
Satisfaction score comparisons show the percentage of participants in each category of the demographic question who indicated they were dissatisfied, passive, or satisfied in response to the chosen satisfaction score question. Participants who responded with a score of 0 through 6 are sorted into the dissatisfied category, those who responded 7 or 8 are passive, and 9 and 10 are satisfied. The ‘Satisfaction Score’ is the percentage of participants who were satisfied minus the percentage of participants who were dissatisfied.
Note: Results are hidden for demographic groups with fewer than 5 participants to protect participant identities.
Demographic Group Sizes
Demographic group sizes display the number of participants who provided each possible answer to the selected demographic question(s).
Compare survey results across multiple engagements
If you have the Advanced Survey Comparison Tool enabled for your account (for accounts with the Engagement360 package), you can now compare results from multiple surveys, Exchanges, and Interviews in a single, streamlined view. This comparison matrix provides a comprehensive analysis across different engagements, allowing you to track overall changes, favorability scores, and satisfaction scores. This tool is ideal for gaining a broader perspective on how different engagements perform relative to each other.
This feature is currently available under the “Compare” tab in the Survey Analysis section for customers with Engagement360 access.
You can choose up to 5 engagements at once for a side-by-side comparison.
You have the option to compare either individual questions or overall topic favorability across the selected engagements.
We have designed the tool to allow you to easily select questions that appear in all chosen engagements for a clear, apples-to-apples comparison. The matrix will automatically organize and display matched data for quick review. It's important that the questions are worded the same way across both engagements to be able to compare results from different surveys. Once the engagement or engagements you want to compare have been selected, the question dropdowns will only include questions that match.
For example, if you're looking to compare responses regarding how teachers respect cultural differences from a School Climate 2024 survey to the School Climate survey from the previous years, a question structure like "Teachers respect cultural differences" must be included in the 2022, 2023, and 2024 surveys as well. Only then will you be able to compare how students responded to that question over time.
Once your comparisons are set up, a matrix will appear, showing engagement titles, questions, and key metrics like participant counts and satisfaction scores. The matrix is divided into sections based on the type of question selected, such as:
- Single-select/Multiple-select questions: Shows the percentage of participants who selected specific answer choices.
- Favorability Scores: Displays scores from Likert scale questions.
- Satisfaction Scores: Shows satisfaction scores based on participant feedback.
- Engagement Type Display: Depending on the engagement type, you'll see either survey titles, Exchange questions, or Interview titles as column headers.
Time-Based Comparison
The Time-based comparison feature, found under 'Compare Multiple Engagements,' allows you to enhance your engagement comparisons.
To use this feature, simply toggle ON 'Time-based comparison' after selecting the engagements and questions you want to compare.
This feature provides a visual representation of the differences (deltas) between responses and the overall change across engagements. It also includes clickable charts that can be expanded to provide a clearer view of how participants’ responses change across engagements or over time.
You can choose to compare favorability by Individual Questions or Survey Topics: