At SC19 I had the privilege of serving on a panel about fostering career paths in HPC. In the days prior, I spent some time preparing for the panel by identifying what I felt were the current gaps and problems facing the career path for RSEs. Because of the response (very positive) I took the ideas from my panel introduction and turned it into this post.

To be honest, I was not expecting the sheer volume of questions and discussion surrounding RSEs and RSE careers during the panel session. I had assumed that the both the term “Research Software Engineer” and the role were too new in the community to generate much interest. I was glad to be wrong. It was encouraging to see how much the landscape had changed, especially in the year since SC18, and given the current momentum and interest, it seems the perfect time to earnestly increase our efforts.

The RSE career, while improving with every passing year, continues to face a handful of major challenges. Call them gaps in the landscape, barriers to progression, or simply problems – it doesn’t matter. The profession is young, growing, and the sooner we as a community identify and address these issues the better. This list is my attempt at pulling out the top 5, however, they are unquestionably biased by my background and institutional environment. So if your list would look significantly different, that doesn’t mean either is wrong or less valid, just different.

There are lots of definitions out there, but for the sake of this post, I’m using a fairly broad and inclusive a definition of an RSE. See my post about “What is an RSE?” for the two definitions/categories I think about the most. There are more (see the us-rse, uk rse, de rse, etc.). Why haven’t we settled on a single definition? Because it’s hard. So let’s leave that discussion for another day and consider this from the perspective that an RSE refers to someone who advances research software (often someone else’s) through programming and software development. People can spend a fraction of their time as RSEs, and this applies to them, just as much as positions that are 100% RSE. For the most part, these challenges are not geared towards researchers who practice research software engineering. That’s another topic for another day, and to be honest, a topic I’m just not as versed in the current landscape of issues (I’d guess issues like credit for software is a top priority).

Keep in mind that the RES landscape is changing fairly rapidly, and this list (and my opinion) is likely to change. My hope is this list starts a discussion and I’m happy to be proven wrong, convinced that the list is wrong/incomplete, or work to polish it into something more refined. More than anything I hope this starts people thinking about ways to improve the career path. Reach out on twitter (@iancosden) or join the US-RSE Association Slack workspace (usrse.slack.com) and let me know what you think.

On to the challenges and then how to fix them.

Table of Contents

Challenge 1: Pipeline
Challenge 2: Organizational Understanding of the RSE Role
Challenge 3: Term Contracts and Soft Money
Challenge 4: Career advancement for single RSEs
Challenge 5: Salary
Final Thoughts

Challenge #1: Pipeline

Anyone who has had the privilege of trying to hire an RSE position will know it’s a challenge. The more research experience and domain knowledge that is required, on top of the prerequisite software engineering experience, the harder it is to find qualified, interested, and diverse candidates. At times as a hiring manager it can feel impossible. All professions need a pipeline of interested early-career applicants. Without such a pipeline, we run the risk of having sub-par candidates fill roles and underperform or having organizations take away funding or positions because they aren’t filled.

The solution: Raise awareness

Most people assume that the problem is that good candidates are “diamonds in the rough” or “unicorns.” While I think it’s true that they don’t grow on trees, I fundamentally do not believe that the problem is a lack of qualified candidates. Instead, the problem is that many, many candidates who would make excellent RSEs are simply unaware that such a position exists.

I’ve run out of fingers to count how many people have talked to me at various events, or reached out after they’ve found out my story. They say things like “I feel like you were talking directly to me.” “I had never heard of this before, but this is exactly what I want to do!” “Do you have any advice for how to become an RSE?” The problem is simple and so is the solution. We need to raise awareness. We as a community need to publicize and market the profession. We need to put the RSE career on the map in the research ecosystem. There are lots of reasons graduate students and postdocs decide not to pursue a traditional academic career and some of those reasons make them good candidates for being an RSE. Software engineers in industry looking for a new challenge need to know that there is a market for their skills in the research community. While an RSE career obviously isn’t for everyone, it’s for no one if they don’t even know it’s an option.

I view raising awareness of the RSE role as a vital role of the US-RSE Association and the members of the community. For the long term success of the career, we need to continually foster diverse groups of new RSEs. Let’s rally around the term “Research Software Engineer” and put it on young people’s aspirational career possibilities. Let’s promote the profession with stories and examples. Let’s give talks and write blogs. Let’s make time for mentoring and counseling those at the very beginning of their career. Things are unquestionably moving in the right direction, but we have a long way to go.

back to top

Challenge #2: Organizational Understanding of the RSE Role

The concept of an RSE can be new to many people including HR, senior administrators, faculty, PIs, research computing professionals (sysadmins, facilitators, etc), and researchers. In the end, each of these groups have some influence over both the short term specifics of an RSE’s job and the long term career prospects of the role. So, where do RSEs belong in an organization? Are they researchers, are they support staff, or somewhere in between? Should they be in a central RSE team or in research groups/departments?
How should RSEs be reviewed, evaluated, and compensated? These are all questions that institutions have to answer before hiring an RSE. So, if an institution doesn’t understand the RSE role, the likelihood that an RSE will end up in a situation that isn’t conducive to long term success increases significantly.

Perhaps not as tangibly obvious as management, administrators, and HR, researchers at all levels need to understand the RSE role for the career path to become legitimized in the research ecosystem. Researcher demand can be a powerful motivator and impotence for hiring RSEs, expanding RSE groups, and improving the stature of the profession in the community.

I’m not convinced that assigning appropriate credit for software contributions is a top 5 challenge to the RSE career path, but it is an issue, and I’m going to lump it in here as a cause of organizational mis-understanding. Research (to which RSEs are by definition associated) has a long history of providing credit in the form of peer reviewed publication. It’s been accepted by the research community and therefore any contribution that isn’t in this traditional form, is viewed as substandard. Because of the proximity to research, it’s only natural to want to look for an RSE’s work output in the form of publications. A clear and accepted method to track software contributions would go a long way towards demonstrating research impact to non-software people, namely academic decision makers. If RSEs are classified as researchers, whether by choice, or organization design, and there are no other alternatives, then improving software credit and placing such contributions on par with other academic publications might be the only option to improve the long term career path of such RSEs.

The first-step solution: Raise Awareness

Yes, I’m repeating myself. I almost lumped the first two challenges together, but I think the information that needs to be conveyed in this situation is different. I say “first step” because I don’t think the community has come to a consensus as to the right answers to all of these questions. However, the community is in a much better place to answer them than people or decision makers who have no concept of the role.

In this case, we as a community need to provide details about the profession that speak to the decision makers and researchers. This means highlighting the impact on research. This means making a business justification to administrators. This means doing some marketing. Take a look at my blog post “why RSEs?” It’s over the top because sometimes we need to do a bit of selling.

A longer-term solution: make software contributions count

I don’t know how to do this, and others have thought MUCH more about this than me. It doesn’t keep me up at night, probably because I’m lucky enough to work in a supportive administrative environment where there is an understanding that RSEs will not be producing the same scholarly output as researchers. RSEs are not evaluated based on the same criteria as researchers and therefore improved metrics for software contributions aren’t one of the primary challenges we are facing. That said, if the community can solve the credit for software problem, or at least make substantial headway, it could have a significant impact on the career path of RSEs by providing a clear, non-anecdotal, indication of research impact.

back to top

Challenge #3: Term Contracts and Soft Money

Not all RSEs are on term contracts and soft money. Some are on 100% hard funding, and some have a combination of permanent funding and soft money. I understand that soft money is a consequence of the current research funding landscape and term contracts are a result of finite research awards. However, if we are going to attract and retain talented, motivated, and diverse RSEs we can’t expect them to tolerate term contracts. Or we’ll only attract and retain the mediocre RSEs who can’t get a better position elsewhere. Soft money isn’t necessarily the problem, especially in large RSE groups. Once established, large groups can insulate the individual RSEs from gaps and lapses in project funding by having a portfolio of funded projects that cumulatively provide a stable baseline of funding. For better or worse, it’s my impression that RSEs who belong to large groups that operate on a model like this (fully cost recovered from soft money grants) are still very much a minority in the overall US RSE landscape.

The solution: RSE positions should be permanent

I’m not going to propose that all RSEs should be on hard money. That’s unreasonable and unlikely to happen. Instead I am going to argue that the profession needs to insist on permanent contracts.

How do we do this? It’s definitely easier said than done, but here are some ideas.

1. Peer pressure – point out other organizations/institutions that have permanent contract RSEs. In order to do this, we have to be sharing details internally within the community. And it has to be easy. US-RSE provides a perfect platform for this kind of sharing.
2. Get support from the researchers – PIs typically have some influence on the administration. Vocal support for RSEs and RSE programs can go a long way toward providing some flexibility.
3. Encourage the formation of RSE groups. Remember, large groups are much better at absorbing the natural ebbs and flows of grant funding.

back to top

Challenge #4: Career advancement for single RSEs or small RSE groups

Congratulations, you’ve been hired as a Research Software Engineer in a computational lab! You were hired by the PI, and now you’re the RSE in that lab. Things are great, but what options do you have for advancement? There is no obvious (or even non-obvious) ladder to climb and things are unlikely to change. If you are lucky, perhaps your PI will convince HR to give you “senior” in front of your title after a while along with a higher than normal raise, but your role and responsibilities won’t appreciably change. The problem is that the role simply can’t change; it’s the only position that this lab requires. There just aren’t opportunities for taking on any leadership/management responsibilities. So what options do you have when you are looking for something new, when you are ready for a new challenge, or when you are feeling ambitious? Unfortunately, you might only have two options: leave or stay put.

Large RSE groups, again, don’t have this problem, at least not as glaringly. Large groups can support junior, mid, and senior level RSEs. They can support technical leads, managers, and directors. Career progression and upward mobility options exist (even if they might be limited) and are visible from the beginning. But this is not the case in very small groups or with single, embedded RSEs. If advancement and growth is not possible and visible, you don’t have a career, you have a job.

The solution: I don’t know (maybe remote work)

This problem is the hardest to identify a solution because it is inherently structural. It’s not a matter of knowledge, understanding, or policy - all of which can change quickly. Remote work seems to be the only viable option right now. This still means that RSEs would be required to leave small groups or lone RSE positions, thereby perennially leaving these roles filled by early career RSEs. This may or may not be a bad thing for research and the smaller groups, but it’s clearly a better option for those seeking advancement. Remote work would allow progression without physical relocation. Many types of RSE wok could be done remotely, but I’m unconvinced that all of it can. Proximity breeds collaboration, and unless many people in the group are remote, and the organizational/group culture is inherently supportive of remote employees, it’s wrought with dangers and limitations. Still, as of now, it seems like the best option.

back to top

Challenge #5: Salary

RSEs have very marketable skills. If they so desire, they have options to make significantly more money in certain private industries. If we are being honest, research (specifically academic research) is never going to compete with industry salaries. However, if we are still being honest, huge inequities between people with similar skill sets causes problems. There are real benefits to working in research and for academic institutions, and, as long as these benefits outweigh the salary difference, some people will eschew the larger salaries for the academic/research benefits.
The balance between monetary and non-monetary benefits is individual, but I’ve yet to hear someone say “no, I don’t want that raise.”
Like it or not, it matters. As people progress in their lives and have families, so does their need for income. People have left RSE roles for this reason.

Even if it doesn’t cause someone to leave, the inequity, or perceived inequity, can cause resentment and disdain for the profession. We aren’t helping the career path if current RSEs complain at every chance they get that they aren’t paid enough and could “make so much more in industry.”

Let’s move on from the wish that we’ll ever get to Facebook/Google/Netflix salaries. It’s not happening. And, remember everyone is going to view money and equitable salaries in their own unique way so if someone is primarily motivated by a paycheck, they are not going to be an RSE, or they aren’t going to be one for long. Alternatively, if someone is sufficiently motivated by the intellectual challenge and academic setting, they probably aren’t going to complain, but that’s not to say that they might not be happier in an industry research discovery environment.

Anecdotally, most RSEs find their jobs fulfilling, appropriately challenging, and have a high level of job satisfaction. And that goes a long way. In the best survey of RSEs to date, when asked to rate their job satisfaction, respondents reported a mean value of 8 of 10, where 0 was “not at all satisfied” and 10 was completely satisfied (read a summary of the study here here).
Add in the typical benefits of working in higher education, and the non-salary benefits of the position are tangible and by no means insignificant. But all things being equal, money can be a significant motivator, and can be a carrot that lures otherwise content people away from jobs.
If we want to make RSE a lifelong career, we need to start having honest discussions about the pay scale and the necessary compensation to retain mid-career professionals.

The solutions: (1) Make a business case for higher salaries, (2) improve career path

There are lots of reasons that hiring and retaining high-quality RSEs, even if they cost more, makes good financial sense. It’s fairly well accepted within the software engineering discipline that the best programmers are on the order of ten times more effective than average (google “10x programmer” and see what I mean). And good RSEs cost more than average ones. It costs a lot of money to hire and onboard new employees. Every time an RSE leaves she takes with her knowledge and expertise. In some cases (especially in smaller groups or individuals) she takes with her the life of the project. As we make the case of the value of RSEs to research, it should become clear that losing an experienced RSE can be devastating. In the end, we likely will need to make a strong economic argument to really have success. We’ll need to show that RSEs are significantly more effective than graduate students or post docs, thus dollar for dollar, the research output is better. We’ll also need to demonstrate that RSEs bring in grant money, both directly and indirectly by enabling new research. And finally, HR will need to understand clearly the other employment options RSEs have when they do their comps.

Because the salaries of most RSEs in academia or national labs will never match that of their industry peers, at the same time we need to improve the academic RSE career path as much as possible. If we can solve the preceding 4 challenges, there is a good chance that the salary issue will simply resolve itself. If we succeed at making an RSE career rewarding, valued, held in high regard, and offering a clear path for continued growth, the salary issue may become less and less important.

back to top

Final Thoughts

It wouldn’t surprise me at all if we end up with a handful of different career paths within the RSE umbrella. In other words, there is likely not going to be a one-size-fits-all solution. Instead we’ll need to define a few tracks within the profession to fit the variety of roles and individuals. RSE is a broad term that can mean a number of things and the associated career path(s) should reflect this. For example a “Super RSE” with a PhD probably has a different path than an RSE with a computer science bachelors degree who leans towards pure software development. Both are valid RSE roles and we need to find a way to make sure neither is marginalized and each sees a clear path for success and growth. And those are just two examples, there are others.

Finally, something else came out at SC19: anything that helps the RSE community, helps individual RSEs. We are in this together and every little contribution helps. Our best chance for establishing and improving the career path for RSEs is to work together as a broad community.
In this country, the US-RSE Association looks to have the potential to have a significant impact, and this is one of the reasons I’m so heavily involved in the organization. But it’s not just a US issue, nor just a pure-RSE issue. We are more alike than we are different so we should be connecting and coordinating with other groups and initiatives that are also trying to make progress in this space (e.g. UK RSE, NL-RSE, DE-RSE, CASC, CaRCC, URSSI, etc..).

As I said at the beginning, let’s discuss and refine the challenges and the solutions as a community. Reach out on twitter (@iancosden) or in the US-RSE Association Slack workspace (usrse.slack.com), or simply write a blog post with your take on the situation. The more ideas on the table, the better.