How your company’s ESG rating may diverge from one rater to another

– The rating score that a company receives from different ESG raters can vary widely.
– Even when one adjusts for the different measurement methodologies that raters use, it’s difficult to find consistency across ratings. One reason for this is the quality of the data.

Here is an excerpt from this Paul Weiss memo about how your various ESG ratings may diverge from one rater to another, a frustrating thing for sure:

Divergences in ratings are commonplace. One study by researchers from the MIT Sloan School of Management found the correlations between six of the major ESG ratings were on average 0.54, and ranged between 0.38 and 0.71. Divergences in scope of analysis (e.g., analyzing different segments of the value chain, such as supply chain analysis vs. internal operations) and specific issues measured (e.g., employee training vs. internal promotion used to measure human capital development) have been shown to explain the majority of variation across ratings.

Moreover, ESG ratings can be challenging to interpret by their nature of aggregating a variety of issues. Combined ESG ratings (i.e., the final ratings reported by ESG rating agencies) also take into account social and governance factors, which are equally broad topics that similarly lend themselves to varying interpretations of scope and measurement categories. For example, a company might excel in worker health and safety but demonstrate poor performance with respect to the board’s risk management oversight or diversity and inclusion.

Critics (including former SEC Chairman Jay Clayton) suggest combined E, S and G scores may create “aggregate confusion” or may be over-inclusive and imprecise. Positive performance in one area does not negate poor performance in another – a point that combined ratings might obscure.

Even when one adjusts for differing measurement methodologies, it is difficult to find consistency across ratings. Commentators believe that the cause is not only the approaches to ratings, but also the quality of the underlying data used.

Due to the lack of mandatory ESG reporting standards, providers of ESG ratings use not only data disclosed voluntarily by the company, but also data from third-party sources to evaluate ESG issues identified as relevant by the ESG rating agency. For example, if a company does not report water use data, an agency may use data from water utilities near the company’s known operational sites to estimate the water used by the company at those locations.

And here is another excerpt:

What else is causing these problems with ESG data? Research suggests a few answers:

• Inconsistencies in the form of arguably similar data are commonplace. A random sample of 50 Fortune 500 companies’ employee health and safety related disclosures found these companies reported employee health and safety metrics more than 20 different ways in their sustainability reports (e.g., number of accidents with fatal consequences, occupational injury rate-related fatalities, and/or lost-time incident frequency rate).[14] These metrics each attempt to assess a company’s health and safety performance, yet their differences make determining which metrics best capture good company performance challenging. It is unclear if differing sets of metrics can be used to produce comparable assessments of companies’ employee health and safety performance. Those who prepare ESG ratings are, therefore, left to determine for themselves which metrics best capture good employee health and safety performance and how to aggregate this data. Their conclusions may vary.

• ESG data is unaudited. Significant data omissions, unsubstantiated claims and inaccurate figures can be difficult to identify within sustainability reports.

• Actual company disclosure does not always match what is reported by ESG data providers, suggesting they may impute ESG data.
Different imputation methods can produce wildly different results, threatening the validity of any analysis that uses the data. As such, Harvard Business School researchers found systemic disagreements and frequent errors in environmental data sourced from two providers.

• ESG metrics predominantly measure whether organizations have specific policies or engage in certain activities, rather than the impacts of such policies or activities. A study of 1,700 “S” indicators found 92% assessed whether the organization conducted audits, risk assessments or training; participated in membership organizations or other collaborations; or engaged stakeholders (e.g., consider the metric “does the company have a Diversity & Inclusion Policy?”). Only 8% measured the actual effects or impacts of these policies and activities (e.g., consider the metric “percent women in management positions at the company”).
Ratings based on binary responses as to whether a given company policy exists may be practical, since this information is easy to measure and generally available, but may not offer investors a clear picture of company performance.