Reputation, authority and incentives. Or: How to get rid of the Impact Factor
This session is moderated by Peter Binfield and Björn Brembs:
Discussion:
Historically, there has been much use and misuse of Thomson Scientific’s (Thomson Reuters) Impact Factor (IF). Originally devised to rank journals according to the citations its articles draw, it has subsequently been misused to rank the authors publishing in these journals.
In this respect, the first question to be addressed would be as to whether we will need to rank journals in the future. In other words, should where something is published matter at all? If this question is answered with ‘YES’, what could be better criteria for objective journal rank?
Irrespective of how the initial question is answered, the next question is whether or not we need a per-publication assessment tool (which could then be aggregated for each scientist). In this discussion, the main message to be kept in mind is that there is no replacement for actually reading a scientist’s contributions. Should this be practically impossible or other important reasons preclude reading all relevant contributions, what new criteria would make the most sense for evaluating research and researchers?
Once we know what we want to replace the IF with, how would we go about replacing the de facto stranglehold the IF has on the major decision-making bodies in science? In other words, if we can agree on technically feasible, meaningful alternatives, what is the best way to popularize these methods and push the IF out of the marketplace?
Some further notes towards a discussion:
- Can we kill off the IF? How? It’s been roundly and soundly criticized in prominent places by prominent voices for decades — and yet it endures, even gains in relative importance.
- One of the easiest ways — some will say the only way — to break a bad habit is to substitute a good one. So perhaps the IF will die a natural death as Open Access takes over, enabling much richer and more fine-grained metrics?
- In addition to ranking individuals with a journal-level metric, the IF is often used to rank papers — that is, given a handful of papers too large to digest at a sitting, a lot of researchers will triage by impact factor. Does this work? That is, do you really find that Nature papers are better than, say, J Virol or Mol Cell Biochem papers? Furthermore, is it really necessary, or should you as a professional scientist be capable of crafting a search query that will return a readable (or at least skimmable, that is, manually triage-able) number of results? If Clay Shirky’s right that filter failure is more important than information overload, where should we be building our filters — pre or post publication — and what role might alternative, non-IF metrics play in such filtering?
Some notes towards an IF-centric bibliography: see this Google Document, which I’ll make public as soon as I figure out how. If you want access n the meantime, email me (cwhookerATfastmailDOTfm) with a gmail address I can use to share it with you.
