Duke is carefully moving into the world of "faculty productivity assessment" with a new tool that academic leaders believe will provide useful data to help shape departments' decisions on strategic planning, faculty hires and budgets.
Quantifying faculty research has always been a problematic goal, and Provost Peter Lange told the Academic Council this past Thursday that previous efforts were rejected by Duke's leadership for not providing reliable comparative data.
But Lange said, a new tool by Academic Analytics, the Faculty Scholarly Productivity Index (FSPI), provides data that, when used with good judgment, can be valuable in making decisions that now are being made without strong data.
One valuable use will be comparing Duke departments to those of its peer institutions. "We want to get at this information. Better data is better than impressions and rumor," Lange said at the meeting. "Institutional comparisons are useful for gaining a sense of research strengths."
Such data, in turn, will be highly relevant for external departmental reviews, Lange said, as well as for their strategic planning, particularly in faculty hiring. It also will support departments' budget planning.
FSPI's latest data, which comes from 2010, ranks Duke faculty in fourth place in overall faculty productivity behind Harvard, Stanford and MIT. While overall rankings aren't the main purpose of the data, the fact that Duke faculty in every discipline continued to score one standard deviation above the average does give evidence to the broad strength of Duke faculty, Lange said. Data from all schools other than the clinical sciences were included.
Here's how FSPI works:
- Academic Analytics -- a private for-profit company specializing in higher education research -- draws information from independent databases, web sources, and government agency reports. The only data it gets from Duke and other institutions are about faculty appointments in departments. A scholar whose primary appointment is teaching would not be included in the survey, which seeks to fairly compare research faculty.
- The FPSI calculates metrics per faculty member, so the size of the institution doesn't skew its results. The index also emphasizes recent data rather than old data, with the exception of certain awards and honors that carry value for longer periods.
- The index assesses 13 metrics in five areas: books; journal articles; citations; grants; and honors/awards. The metrics are weighed according to the different publishing traditions of the fields; for example, books in the humanities are scored higher, as are journal articles in the natural sciences.
- The data are then aggregated in a database including 382 institutions, 172 disciplines and more than 5,300 programs.
- The results are available for universities to review across a secure website accessible only to Duke faculty and administrators. Faculty and administrators can run comparisons controlling for a large number of factors.
The bottom line, says Lange, is the tool provides useful information to help Duke assess the productivity of its faculty compared with those at other schools, However, FPSI also comes with caveats and limits, and the administration will be careful in its use.
"One question is whether the data will be used in determining individual faculty salaries, and the answer is no," Lange said. "The data doesn't cover the full research activity of a faculty member, does not provide metrics of research quality, says nothing about teaching or university service. It is primarily designed as a constant basis for inter-institutional departmental comparisons."
Lange said he was concerned about the "fallacy of concreteness," a proclivity to rely on hard data and undervalue less measurable aspects of being a faculty member.
"People have a tendency to flow to the most solid piece of data and other things will flow away," he said. "I'm concerned about it and I will discuss it with the deans. We want to use self-conscious judgment in this process rather than just relying on the intuitive use of the data."
In rare cases, the data could be useful if a faculty salary equity study indicates an individual faculty member is being paid significantly less than what would be expected for someone in that position. "In that case, we could use the data to see if the faculty member has been [underpaid]. The data won't determine the decision, but it will be an additional tool we can use."
Lange said he expects Academic Analytics to continue to improve the tool to make the information more useful. Duke will do extensive random checks to ensure faculty are being assessed accurately -- for example ensuring that faculty members who have changed their names get credit for research published under their old names, or that research is properly attributed when several faculty members at different institutions have similar names.
At the council meeting, faculty members expressed concerns about whether the FPSI data captured interdisciplinary work and whether it would later include the clinical sciences. But faculty members also affirmed the usefulness of the data if used for specific purposes and with judgment.
"The metrics from Academic Analytics can give a useful measure of institutional, departmental and disciplinary research productivity, but we should be careful not to use this metric as the measure of an individual's intellectual contribution to the university's research mission," said Susan Lozier, chair of the Academic Council and professor of earth and ocean sciences in the Nicholas School.
Academic Analytics makes the data available to Duke and other universities for a fee. It started producing faculty productivity databases in 2005, and the FPSI is one of its most recent and most sophisticated products, Lange said. Duke reviewed previous Academic Analytics products and wasn’t satisfied, but Lange said the new FPSI is "more broad, flexible and accurate."