6 Best Practices in Creating Employee Engagement Surveys
In the VIDEO:6 Best Practices in Creating Employee Engagement Surveys I cover six ways to form questions in order to create a survey that will obtain the best and most accurate responses possible. By creating a powerful employee engagement survey you will accurately measure your organizations levels of engagement like never before.
Use a subset of anchor questions to measure overall engagement.
The most accurate way to measure employee engagement is to use the average score from a subset of validated anchor questions. We then use the score from the anchor questions to place employees in one of four different groups from Fully Engaged to Fully Disengaged.
Use a variety of questions to measure employee perceptions regarding their work, their team, their boss, and the organization overall.
These items will be used to determine unique perceptions among the different engagement groups of employees. Also include items to measure the five major drivers of employee engagement, which are: Meaning, Autonomy, Growth, Impact, and Connection.
Use proper question structure and formatting:
Good questions are actionable, meaning that it is obvious what actions to take based on the item. So avoid vague or general statements that can be interpreted in a variety of ways. Also, don’t use double-barreled questions which are items that contain two ideas in one question. Finally, use questions that are positively worded with a consistent rating scale.
Customize the survey so that it is relevant to your organization.
Every organization is unique, so make sure to include items that evaluate recent changes, alignment with company values, or other pertinent issues. But don’t go too far. If you customize too much, you won’t be able to utilize industry benchmarks on similar questions.
Use two or maybe three open-ended questions.
We recommend asking: “What are the areas that need the most improvement in our organization?” and “What are the greatest strengths of our organization?” These two items will generate good qualitative feedback that provides context to the scores on the other survey questions. Don’t use too many open-ended questions because it makes the survey run long and it doesn’t provide any additional feedback.
Use between 40-60 questions total.
A survey with 50 questions takes about 8 minutes to complete. If your survey is too short, then it doesn’t provide enough information to make specific conclusions and it becomes in actionable. If it is too long, employees will not finish it and you could also end up with too much information with leads to data paralysis.
Let me share an example that will help clarify what I mean. The other day my seven-year-old came to me and told me he was sick and couldn’t go to school. Now, I love my kid, but let’s just say that when it comes to school attendance, he hasn’t earned my unadulterated trust just yet. So I check his throat and, wouldn’t you know it, it’s bright red. Flaming red. So, (a)I felt bad for questioning him, and (b) I immediately went to get a thermometer. His temperature was higher than normal, but nothing I would normally worry about. But coupled with how bad his throat looked, it was enough to put us on the road to an urgent care facility and the doctor running some strep tests.
So, back to your engagement survey. There are three basic steps when analyzing results of any kind (we’ll focus this article on differences in employee engagement scores). First, identify a difference. Second, determine if it’s a real difference. And third, if it is real, determine how meaningful it is.
In the example, my son came to me with a difference (he felt different from when he was healthy). I could have taken these results at face value, but I’m sure you can come up with a number of reasons why that would be inadequate parenting. So my next step was to determine whether or not that was a real difference. A visual check and a temperature check indicated that perhaps there was a difference from his normal functioning. At this point, it seemed likely that he should stay home from school. (OK, he’s starting to earn a bit of that academic trust back)
Likewise, most organizations stop at step one or two and make the mistake of drawing conclusions and taking actions without understanding whether their engagement results are real and meaningful.
Let’s look at each step in more detail.
Step 1: Identify Changes in Scores “Differences”
A typical employee engagement survey consists of participants answering questions along some sort of scale. This is your basic quantitative survey. Reporting generally falls along the lines of taking the percentage of respondents who answered a certain way, and ranking those numbers against the other questions or comparing them to past years or to industry norms. “We see a 5% increase in item X as compared to last year and we have lost 3.2% in item Y.” Does this sound familiar at all? Likely. We seldom see a company that doesn’t make this comparison.
This method of interpreting results is intuitive; it makes sense to almost any audience. The problem here is in the fundamental nature of measuring humans. We’re not atomic particles. You can ask us a question one day and it might be a completely different answer from when you asked us last week. We’re fickle and capricious and easily swayed. And even when we try to be consistent, we’re not. So any survey of employee attitudes and perceptions is going to have some natural fluctuations one way or the other.
Flip a coin 100 times and I’ll bet you $100 that it doesn’t end up 50-50. It will more likely be off balance, something like 53-47 or 45-55. If you then said “looks like we lost 8 percentage points from our first flip,” technically you’re correct. But does that loss mean anything?
So when we say we “lost 3.2% in item Y from last year” not only are we being overly precise on something that shouldn’t be measured with such granularity, we’re not even sure if that’s a real difference or if it’s just random chance. This method is not robust enough to draw any conclusions yet. We need to know if the difference is real and significant.
Step 2: Test the Significance of the Difference
Fortunately there are statistical analyses that we can use to analyze whether those differences are likely due to chance or to some sort of meaningful difference. Data are compared, and a score is spat out. That score corresponds with a percentage likelihood that the difference is due to chance—or not. The typical cutoff used by statisticians and sociologists when dealing with people is 5%, which for our purposes here means there is only a 5% chance that the difference between two things is due to random variation.
This level of rigorous testing is critical for interpreting data meaningfully. We have to know if a real difference exists before we start investing in something that was never actually a problem to begin with. However, statistical testing is not without its concerns.
Statisticians are quite good at what they do. As such, they have developed tests that are incredibly sensitive in picking up differences between groups, even when the groups are quite small. In industry, we often are testing large groups of employees with relatively sensitive tests, which can lead to complications.
Let’s look at this in real life. A 2013 Business Insider poll asked which American states were the most arrogant. New York won, although followed closely by Texas. Let’s say I want to examine whether New Yorkers have a reason to be more arrogant than Texans, and I choose “average IQ” as my measure. I want to make sure I capture the effect, so I test 4,000 people from each state—a statistically representative sample of the overall population. Lo and behold, I find that New Yorkers do, in fact, have a reason to be arrogant, as they have significantly higher IQs than Texans (corroborated by a 2006 study). After all, as Babe Ruth said, “It ain’t braggin’ if you can back it up.”
A deeper dive into the data, however, reveals that New Yorkers averaged a 100.7 on the test, and Texans an even 100. That means in a 75-minute test, with hundreds of activities and tasks, the average Texan possibly defined one fewer word, or could only recite six numbers backwards instead of seven. In other words, despite the statistical test indicating there was a significant difference, that difference is meaningless. Texans redeemed.
Using significance can work if the environment is just right. But, there are enough potential obstacles that could necessitate something a bit more powerful.
Step 3: Measure the Effect Size
This is where effect size calculations come in. Effect size measures the magnitude of the difference between two groups. This simple procedure can shed a vast amount of light onto the true nature of the differences and allow for more meaning to be drawn from the results. The interpretation of effect size numbers is actually very straightforward.
A score between 0 and .2 is a trivial difference.
A score between .2 and .5 is a moderate difference, and at that point where you want to start paying attention.
Any score above .5 is substantial and should be either a cause for concern a cause for celebration.
For the IQ example, the effect size was .02, i.e. it shouldn’t merit a second thought.
Effect size can always be measured, and is independent of significance, meaning every significant result has an effect size, but not every effect size is significant.
Sometimes, when looking at the difference between two groups, it is readily apparent that the difference is so small as to be negligible (as in the IQ example). However, often it is not. Statistics take into account both the average of a group and also the variation of scores around that average. This can reveal insights that might go undetected.
Recently, I completed an analysis for one of our employee engagement client partners. When comparing their survey items to our national norms, several items jumped off the page as needing immediate attention. But, one that was lurking in the wings was a question that asked if they “…[understood] how [their] work contributes to the overall goals of the organization.” This particular item typically receives fairly high scores across industries, and while this organization came in lower than the industry benchmark, the scores were still relatively high. Under normal circumstances, this would have been treated as a non-factor.
However, the effect size was unusually large. Further analysis revealed that the scores on this question are relatively tight around the average score (there wasn’t much variance between employee opinions). When this company scored lower than the benchmark or average, that was a more meaningful difference than a question with a greater differential, but for which answers range all over the scale. In other words, it wasn’t just a few employees telling the company they were having a “below average employee experience,” it was most employees. Dealing with this issue ended up being one of this company’s highest priorities to come out of the survey. It wouldn’t have even registered without checking for effect size.
The reality is that any question you ask is going to have a distribution of answers. To ignore the distribution, in favor of reporting simple comparisons such as those we typically see from most surveys is not only inadequate, it’s potentially misleading. Bad data result in bad decisions. The effect size is a straightforward calculation that resolves any problems arising from sample size or comparing differently shaped distributions. Further, it has the added bonus of simplifying ordering priorities.
The next time you analyze results, ask yourself if you’d rather:
Make your job harder and get poor conclusions, but at least do it the way it’s always been done, OR
Expend the same amount of effort in calculating, less effort in interpreting, and draw more accurate conclusions.
Without answering each question in the three-step process, organizations can be distracted by focusing on the wrong issue or miss an issue all-together. Remember:
Is there a difference?
Is that difference real?
Is that difference meaningful?
The end of my story is that when the doctor swabbed my son’s throat for a strep test, the redness came off on the swab. It turns out my wonderful progeny had decided he didn’t want to go to school. So at breakfast, he ate around all the red marshmallows from his cereal until he was done and then swallowed them last, hence the bright red throat. I was too impressed to be angry. And more germane to this article, had I not followed the three-step process (i.e. seeing if the apparent difference was meaningful), my conclusion would have been incorrect. Maybe we all need to quit faking it and get back to school.
Employee engagement is a two-way street. It’s a 50-50 proposition. It’s the organization’s responsibility to provide a working environment where employees want to engage, but employees need to choose to be engaged. So what’s in it for employees you ask? Here are Five Personal Employee Engagement Wins that employees receive by being engaged.