Want more expert insights on the topic of pay equity? Access our entire Pay Equity Deep Dive Series.
Our first blog in the series covered the topic of Pay Analysis Groups (PAGs). As you’ll recall, one of the first steps in conducting a pay equity analysis is deciding how to segment your workforce into meaningful pools for comparison purposes. We call these segments PAGs. We also discussed the trade-offs between creating many differentiated PAGs (e.g., segmenting by job) and creating fewer pooled PAGs (e.g., segmenting by job function).
One of the downsides of creating many differentiated PAGs is the possibility of limited statistical power, which will make it more difficult to detect pay disparities. Here we delve into this topic more deeply.
The Role of Statistical Significance
As noted in an earlier blog on developing a remediation strategy, the gold standard in pay equity reviews is to use statistical significance to identify pay disparities. The statistical significance of a pay disparity is typically measured using a 5% significance level (i.e., p-value ≤ 0.05). This is a commonly used significance threshold and indicates that there is a 5% chance or less that the pay disparity we’re seeing is due to chance. A 5% significance level does not refer to the size of the pay disparity. This is a common misunderstanding. It tells us whether the pay disparity we’ve identified, which could be of any magnitude, is statistically meaningful and unlikely to be due to chance.
On-Demand Webinar: Pay Equity Deep Dive: Setting the Right Foundation
When conducting a pay equity review, most will rely on a 5% significance level as the primary signal that a pay disparity warrants attention and potential remediation. With that in mind, we think it’s important to look under the hood and see what influences the statistical significance of a pay disparity. Four factors are particularly important in influencing statistical significance.
- Size of the Disparity. The first, and most intuitive, is the size of the disparity. All else being equal, a larger pay disparity (e.g., 6%) is more likely to be statistically significant than a smaller pay disparity (e.g., 2%). By “all else being equal,” we mean that in comparing these two scenarios, the only difference is the size of the disparity; otherwise, the scenarios are identical.
- Number of employees. The second factor is the number of employees in the PAG. All else being equal, a disparity in a PAG with more employees is more likely to be statistically significant than a disparity in a PAG with fewer employees. This is because as the PAG size grows, statistical power grows, which means we’re more likely to detect a pay disparity if one exists.
- Representation Balance. The third factor — the representation balance for the demographic groups examined — may be a bit less intuitive. All else being equal, a disparity in a PAG with more evenly balanced representation (e.g. 50% Female/50% Male) is more likely to be statistically significant than a disparity in a PAG with less evenly balanced representation (e.g., 10% Female/90% Male). Intuitively, we’re less certain that a pay disparity is statistically meaningful when its measurement is based on a small number of cases, such as when a PAG includes relatively few people in a particular demographic group.
- Unexplained Pay Variability. The fourth factor is the unexplained variability in pay. To understand what we mean by this, recall our earlier blog on the topic of Wage Influencing Factors and Reliability Testing, where we noted that in developing pay models you should aim for a model adjusted R-squared of at least 70%. While this threshold is a rule of thumb, a low value of adjusted R-squared tells us that the model is not doing an adequate job of explaining the observed variation in pay, resulting in high unexplained pay variability. High unexplained pay variability indicates more “noise” in the data, which makes our pay disparity measurements less precise.
- There are several possible causes of high unexplained pay variability. One relates to the WIFs in the model. A model that includes an insufficient number of relevant WIFs may lead to high unexplained pay variability. Moreover, insufficient variation in the values of the WIFs may also create high unexplained pay variability. For example, if Tenure is one of your WIFs and all the employees in the model have between one and two years of tenure, it will be difficult to detect a relationship between tenure and pay.
- Higher variability in the pay outcome you are examining can also create high unexplained pay variability. Intuitively, it’s more difficult to explain variation in pay with a set of WIFs if the pay outcome itself is highly variable.
Some Illustrative Examples
To make the factors we outlined above more concrete, we thought it would be helpful to share some illustrative examples. In all these examples, we’re assuming that everyone in a PAG is in the same job and location. Moreover, the models include only two WIFs: Tenure and Prior Experience (as proxied by age at hire minus 22). The model also includes an indicator or dummy variable reflecting the employee’s gender. For simplicity, we’re assuming only two genders: Female/Non-Binary and Male. Throughout these examples, Tenure and Prior Experience are unchanged from one example to the next. Thus, high and low unexplained pay variability situations are created by changing the variability in pay.
Example 1: Small PAG with Low Demographic Balance & High Unexplained Pay Variability
In our first example, the PAG includes only 30 employees, so it’s small. The PAG also has low demographic balance with 5 Female/Non-Binary employees and 25 Male employees. Lastly, pay within the PAG is highly variable, with a minimum value of $75,000, a maximum value of $150,000, and a standard deviation of $21,960.
When we run the regression analysis, we find a pay disparity for Female/Non-Binary employees of 6.5%. That is, after accounting for tenure and prior experience, Female/Non-Binary employees are paid 6.5% less than Male employees. The p-value for this pay disparity is 0.40, indicating that it is not statistically significant at the 5% level. Despite the pay disparity being relatively large, it’s not even close to being statistically significant.
What if the PAG were larger? To see what would happen, we replicated the 30 employees in the example above three times to create a PAG with 90 employees. Because we’re simply replicating the same 30 employees, the demographic balance and pay variability are unchanged. The pay disparity is also the same at 6.5%. This time, however, the p-value is 0.13. While the pay disparity is not statistically significant at the 5% level, it’s considerably closer.
How large does the PAG have to be in this case for the 6.5% pay disparity to be statistically significant at the 5% level? Turns out the answer is 150, with a p-value of 0.05.
This example illustrates the difficulty of identifying even a relatively large pay disparity as statistically significant when the PAG is small, has low demographic balance, and high unexplained pay variability. In this case, one option is to pool this job with other similar jobs to increase the size of the PAG (and add Job as a WIF in the model). Another option is to consider practical significance. Practical significance indicates whether a pay disparity is meaningful from a business perspective and warrants attention, regardless of statistical significance.
Example 2: Small PAG with Low Demographic Balance & Low Unexplained Pay Variability
For our second example, we start with the same 30 employees we used in Example 1. The only change we’re making to the data is that pay now has lower variability. The minimum value is $95,000, the maximum value is $125,000, and the standard deviation is $7,624. When we run the regression analysis, we again find a gender pay disparity of 6.5% disadvantaging Female/Non-Binary employees. However, in this example the p-value of this disparity is 0.01, indicating that it is statistically significant at the 5% level (in fact, it’s statistically significant at the 1% level). This second example, coupled with the first, illustrates very clearly the important role pay variability plays in the statistical significance of a pay disparity.
Example 3: Small PAG with Even Demographic Balance & High Unexplained Pay Variability
In this last example, we’re again starting with the same 30 employees as in Example 1. We’re making two changes this time. First, we’re creating an even gender balance with 15 Female/Non-Binary employees and 15 Male employees. We’re also changing pay values, to return to the highly variable pay of the first example, with a minimum of $75,000, a maximum of $150,000, and a standard deviation of $18,424. The changes we made were designed to maintain a pay disparity of 6.5% to match the two prior examples. In this case, the p-value associated with the disparity is 0.19, so it is not statistically significant at the 5% level. However, this p-value is less than half the p-value of 0.40 in the first example with low demographic balance.
When we replicate these 30 employees to create a PAG with 60 people, the p-value associated with the 6.5% disparity falls to 0.05 and becomes statistically significant. In the first example where the demographic balance was low, the PAG size needed to increase to 150 before the disparity was statistically significant at the 5% level. In this final example, with an even demographic balance, it only needed to increase to 60.
Implications
If you’re looking for a way to help ensure that your pay disparities are not statistically significant, then create small PAGs with unbalanced demographic representation and high unexplained pay variability. Voila!
Identify, Understand and Resolve Pay Disparities
However, you may be asking yourself, doesn’t this approach defeat the purpose of a pay equity review? Shouldn’t we structure our pay equity review so that we can detect pay disparities and take corrective action? We think so.
When designing a pay equity analysis, we recommend structuring your analysis to ensure you can detect pay disparities. The easiest way of doing this is to ensure that your PAGs are sufficiently large — we recommend 50+ employees in a PAG. This might require pooling smaller PAGs into larger PAGs. As we illustrated here, you might require even more pooling if your PAGs have low demographic balance and high unexplained pay variability.
Related Posts
