What would happen if we gave major federal funding bodies like the Australian Research Council a “memory”, that persists over multiple years and multiple rounds of a funding scheme?

That’s the idea I explore in today’s video: the concept of providing some of the funding schemes with a form of memory that, in certain circumstances, takes into account the history of an individual’s, proposal’s (and combinations of them) or discipline’s history with an application scheme.

This is already the backbone of academic publishing pipelines where submissions go through many rounds of revisions, sometimes over years, but is somewhat absent from the bread and butter project schemes in funding bodies like the ARC.

After every funding round outcome, two refrains you often hear are a) from an individual researcher who’s just had their 3rd or 4th attempt rejected, and b) from research disciplines who feel like their area has been neglected for many years.

Even in a “perfectly” run system, both of these occurrences are a statistical certainty to happen, when funding is tight, and acceptance rates are low.

The idea of the memory-based approach is that, where the proposal and individual, combined, has consistently been assessed as superb but just missed out, that history is taken into account in assessing their later repeat applications.

Good researchers with great ideas who get the chance to submit multiple times and are always highly rated but miss out, often eventually get over the line: but some, at least anecdotally, don’t – I imagine we can find data for this.

The idea of this scheme is that it catches some of those, and perhaps even more importantly, signals to researchers that if they’re consistently excellent but miss out repeatedly, there is a mitigation mechanism to try and catch at least some of them – a “safety” net so to speak. Unis already do this locally to some extent with “near-miss” schemes.

The idea is also that you could do this on a discipline level as well: reducing the incidents of certain disciplines receiving no funding for many years on end.

Everything has a trade off: however implemented, this would likely have the effect of first time applicants having a slightly smaller pool of resources to pull from (being especially sensitive to ECRs), and we’d need to consider the various gamification responses and whether they could be addressed.

This is not a new idea: it could be a good, bad, even horrible, or ineffective / un-realizable in practice. But admidst much research system reform discussion, it’s one I haven’t seen get much airplay that I think it would be deserving of, even if it’s to decide not to do it!

So, what does everyone think?

Thoughts are my own and not representative of my organization.

Full Video Notes

All right, it’s Friday afternoon, and I thought I would put to video an idea I’ve been thinking about in the back of my mind for how we could possibly make a small but meaningful improvement to the research ecosystem in Australia. So, when you submit a paper to a conference or especially a journal, it will go through a round of revisions. You get some feedback from the reviewers, the editor may have a look as well, and often, if successful, you’ll get a chance to revise and resubmit the paper. Sometimes this takes a couple of rounds, sometimes this can take many, many rounds of revisions, feedback, improvements, until eventually, hopefully, you submit a vastly improved research paper that is accepted and published in the journal. What is key to that process is the concept of memory. So, each time you submit the paper, what has been previously submitted and the reviews that were previously submitted does not get thrown out; it’s all part of the historical record and, depending on the nature of the journal, is considered to varying degrees at each round of the assessment. And this history is really important because it means you’re not throwing away information and signal from what has happened to that paper previously in that journal.

My proposal for improving the research funding ecosystem in Australia, especially funding bodies like the ARC, is to give the funding body an explicit concept of memory from one year to the next. As I understand it, the way that most of the major funding processes in the ARC work from one year to the next is largely independent, and that is why you often get perplexed academics submitting one year, getting amazing reviews, submitting what they genuinely think and someone objectively would think is an improved version the next year, and getting absolutely destroyed.

The idea of a memory would be to catch the rare but, I think, situations that actually do exist where either an individual, or a project idea associated with an individual consistently gets exemplary scores that don’t quite get it funded for several years or applications in a row, and to take that into account in considering the merits of whether the grant or proposal should get funded after a certain number of very highly rated trials. Now, nothing is for free, and if you consider it, it’s a horrible way to think of things, but if you consider it a zero-sum landscape at the moment, what you would be doing here is ensuring that people who submitted consistently extremely highly rated proposals over a sustained period of time got a slight advantage in terms of getting their proposals funded. And that would come at the cost of people who were submitting their grants for the first time. But that cost would be relatively minor. It wouldn’t be a situation where if you submit a grant three times in a year, by default, you get it funded on the fourth year. That just wouldn’t work in the current very tight funding ecosystem. But the idea would be for a net to catch those people who have submitted amazing ideas that have just missed out on a very consistent basis.

Now, how many people would this affect? That is a good question, and you’d have to look at the data. I would suspect that some people, through university schemes like near-miss schemes who submit highly rated unsuccessful proposals, do eventually get their proposals submitted. But I bet it’s not all of them. Now, you might argue, well, even if it’s not all of them, the majority of people do get their grants funded eventually. This isn’t going to affect very many people. Now, normally, the number of people you’re affecting is a valid argument to consider, but I think in this case, it doesn’t have as much weight because what you would be doing here is making sure a very small number of individuals with fantastic ideas do not, over a period of multiple years, fall through the cracks of the research funding ecosystem. And I think the morale effect that would have on the system would be not insignificant because people would have a little bit more faith that if they really have amazing track records and have amazing ideas for what they’re submitting, and do consistently great submissions, that there is somewhat of a signal from year to year that increases the chances that eventually, their research will get funded. And I think the slightly reduced funding pool for first-time submitters, that might be an acceptable trade-off to make if you consider the overall amount of funding not increasing.

I also want to propose this as an idea at a discipline level. So, one of the refrains you hear every year when grant funding outcomes come out is that a certain discipline feels hard done by; they’ve had no funding outcomes in that year, or perhaps for multiple years. And one of the other things that this memory effect, if you chose to implement it, would do, would give you the ability to ensure that disciplines weren’t systematically, well not systematically, weren’t neglected for long periods of time, for long sequences of multi-year attempts at funding. Again, this would be like a little bit of a socialist effect in that super successful disciplines might get their funding