4/29/11: In response to a national grievance filed by Chapter 293 in February, the SEC last week delivered to the Union a summary data set for the performance ratings that were distributed to employees in connection with the recently completed pilot of the SEC’s new, so-called “Evidence-Based Performance Management System” (the “EBP System”). Although some managers have released top-level ratings averages to some of their employees, the Union has compiled more detailed EBP System ratings data here, so that employees may place their own performance ratings into a more meaningful context by perusing the average ratings in various groups and categories. The ratings data demonstrate that, although the SEC promised to use “standardized, validated performance objectives and standards” in the EBP System and to emphasize “fairness and transparency,” the new system has missed this mark by a wide margin.

For years, the Union has consistently supported the creation of a robust performance management system at the SEC, provided that the system is fair, objective, transparent and successful in making meaningful distinctions that employees will understand. The stated goal for the EBP System, with which the Union strongly agrees, is to create a system that helps the SEC develop an even more robust, performance oriented culture – a system that rewards high performers and identifies and holds accountable those who are failing. We continue to believe that to succeed in this goal it will be necessary for the SEC’s performance management system to make it clear to each employee what is expected of him or her to improve his or her performance. The pilot run of the EBP System demonstrates that, as currently configured, it does not accomplish this basic task.

How It Works

The pilot, or “test run,” of the EBP System covered the performance appraisals for employees in the Division of Enforcement and the Office of Compliance, Inspections and Examinations for the brief rating period from July 1 through October 31, 2011. These 1,309 employees constitute approximately half of the bargaining unit employees who currently work at the SEC. They worked in six positions, Attorney Adviser, General Attorney, Paralegal Specialist, Securities Compliance Examiner, Staff Accountant and Trial Attorney.

Although these employees were rated under the new EBP System during the test run, their pay increases continued to be made under the existing system that has been in place for the past few years. Their test run appraisals under the EBP System also will not be retained in their personnel files. Commencing with the rating period ending September 30, 2011, however, EBP System ratings will dictate employees’ performance based pay raise at the end of this year. 

Under the EBP System, managers rated various aspects of employees’ work based upon a rating scale of 1 through 5. The average of these ratings became each employee’s individual EBP System rating. In the future, for purposes of determining merit pay salary increases, an EBP System rating of 0 - 1.4 will convert to a 1 or “unacceptable” merit pay rating. A 1.5 - 2.4 will convert to a 2 or “needs improvement” merit pay rating. A 2.5 - 3.4 will convert to a 3 or “meets expectations” merit pay rating. A 3.5 - 4.4 will convert to a 4 or “exceeds expectations” merit pay rating and 4.5 - 5 will convert to a 5 or “greatly exceeds expectations” merit pay rating. In the future, employees who receive a 3, 4 or 5 will receive merit pay salary increases.

Highly Subjective Ratings Varied Dramatically from Office to Office

Even a cursory review of these data demonstrates that the ratings varied dramatically from office to office – a result that should never have occurred if the system was working properly. Indeed, the ratings differences between the various offices appear to demonstrate that different managers assigned ratings with very different understandings regarding the levels of performance necessary to earn particular ratings. Such a profoundly subjective system is simply unacceptable, because it could never be viewed as providing a credible roadmap for what an employee needs to accomplish to succeed at the SEC.

Consider, for example, the range of employees rated at the highest level for merit pay purposes, the “5” rating for greatly exceeding expectations. 15.2% of employees nationwide received a “5.” But that percentage varied wildly depending upon the office in which the ratings were assigned. In two regional offices, 0% received a “5,” whereas in another regional office the number was 34%. In others, the percentage could fall anywhere in between, with 1% receiving a “5” in one office and 23% in another.

Similarly, while it is true that most employees nationally  (61%) received a level “4” merit pay rating, the percentages receiving those awards ranged from 41% in one regional office to 83% in another. Indeed, the percentage of employees awarded the level “4” merit pay rating included offices with percentages between 40% and 50% (1), between 50% and 60% (5), between 60% and 70% (3), between 70% and 80% (2) and between 80% and 90% (1).

The range was equally wide in connection with the percentages of employees receiving a level “3” for merit pay purposes. These awards ranged from offices between 0% and 10% (2), between 10% and 20% (4), between 20% and 30% (3), between 30% and 40% (2), between 40% and 50% (1) and between 50% and 60% (1).

SEC employees may ask themselves, should the single most important factor driving the performance rating that one receives be the geographic location of the office in which one happens to work? Obviously, the answer to this question must be “no.”

Union officials believe that, as we have predicted for some time (review an article about this last year here), the striking lack of consistency in the ratings is a direct result of the EBP System’s failure to clearly define performance at the various performance levels and to make meaningful distinctions between those levels. The pilot run of the EBP System largely covered Enforcement and OCIE, where there is considerable consistency with respect to the type of work performed by employees. This makes the inconsistency in the results all the more troubling. 

Merely calling a system “evidence-based” does not make it so. Clearly, differing management expectations about the level of performance required to earn a particular rating are continuing to drive profoundly different ratings. The data reflects that these differences exist from office to office, but the Union is also receiving a substantial volume of information indicating that these differences exist from manager to manager. This undercuts the central goal of the EBP System, which is to help create an organization in which employees “know what is expected” of them. Expectations should attach to a particular job and what that job is designed to accomplish – and should not be dependent upon the subjective perceptions and biases of individual managers. 

Problems from the Old Performance Management System Continue to Surface

In addition to the issues outlined above, the EBP System pilot data also reveals further potential problems as the SEC continues to move forward. Indeed, some of the same problems that led to the demise of the SEC’s last five-level performance management system are already appearing within the new system.

SEC employees will recall that an independent arbitrator ultimately ruled that the SEC’s old performance management system violated federal antidiscrimination statutes because it had an impermissible disparate impact upon African American professionals and employees over the age of forty. The new EBP System appears to have precisely the same problems.

Indeed, the pilot data show that white employees were approximately two and one half times more likely to receive the top rating, “5,” than African American employees. Similarly, African American employees were twice as likely to receive a “3” as white employees. Statistically significant differences also appear in the top level data based upon age, with many more employees who are over forty receiving a “3” than those under forty. Other biases also seem to appear in the data.

Of course, when a rating system is highly subjective, the biases of the raters will tend to seep inexorably into the ratings process. It cannot be gainsaid that it is very difficult to understand why the SEC would continue to run headlong toward the perpetuation of a profoundly subjective rating system that appears to be leading towards obvious violations of the federal antidiscrimination statutes. We have seen this movie before.

Moving Forward

Although Union officials were disappointed by the pilot results, they were not at all surprised. James Fay, a senior counsel in the BRO Enforcement program and a Union representative heavily involved in the Union’s discussions with the SEC in connection with the development of the EBP System recently said, “We have been predicting this outcome for the past two years. As the SEC’s in-house experts developed the EBP system, it became apparent to us that the performance levels they created were essentially undefined. They could not provide clear examples of what performance at different levels looked like with reference to the work that we actually do here at the SEC. We have pointed this out in meeting after meeting, in our written communications and in employee focus groups. Unfortunately, our input was disregarded. We have for some time predicted that the lack of clarity in the system would drive a lack of consistency in the results – and that is exactly what happened.” 

Chapter 293 President Greg Gilman recently noted, “The Union still strongly supports the creation of a robust and fair performance management system at the SEC. But h.ow can the agency expect employees to achieve success if its performance management system does not lead to a common understanding as to what success actually means? This always has seemed to be a basic, but necessary, component to any successful performance management system. We continue to have trouble understanding why it is so obviously missing.”

The Union is continuing to talk to SEC management about how to move the performance management system forward. It will be critical for the agency to make a serious effort to listen to the employees’ Union representatives, because the EBP System is now covering all agency employees and later this year will dictate each employee’s merit pay salary increase. The Union also recently signed an agreement with the SEC which requires an independent third-party study of the EBP System, with recommendations to be issued after the 2011 data is gathered. 

We will continue our efforts to engage SEC management to address the many concerns that have been raised by this new EBP System. As you participate in the system, we urge you to provide your comments and suggestions to the Union.