Open Source Software Policy Options for NASA Earth and Space Sciences (2018)

National Academies Press: OpenBook

Chapter: 5 Policy Options and Recommendations

Visit NAP.edu/10766 to get more information about this book, to buy it in print, or to download it as a free PDF.

Suggested Citation:"5 Policy Options and Recommendations." National Academies of Sciences, Engineering, and Medicine. 2018. Open Source Software Policy Options for NASA Earth and Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/25217.

The committee outlines a selection of policy options below, including both incentives and mandates for NASA’s Science Mission Directorate (SMD) to consider. Based on the charge to the committee and discussions with NASA officials, the committee operated under the assumption that SMD will transition to a greater level of openness in accordance with federal policy (see Chapter 1). It is important, therefore, that NASA ensures that the transition helps advance science, foster collaboration, and generally advance NASA goals (see Section 1.2). The committee believes that the best way to achieve this is to work toward a cultural norm of robust, open source software (OSS) development and maintenance. This will not happen overnight and will require ongoing strategic investment.

SMD does not currently have division-wide policy regarding software publication, distribution, or licensing. As described in previous chapters, each software type may have its own legal requirements, and raise different policy issues and concerns from the community on the value and practicality of openness. Correspondingly, existing NASA software varies across a spectrum of openness. Each division may have its own policies, and the chief information officer (CIO) and Technology Transfer Office may differ on matters of OSS release. SMD may choose an overall level of openness for new software produced by and for the directorate in the long term, but individual programs and software types will advance toward greater openness at their own rate and through different means. The options below can be considered a sort of toolbox to draw from and help move the community toward greater openness while recognizing that different disciplines and code types will have different requirements and transition at different rates. Incentives will help to move the community norms toward greater openness regardless of whether mandates are eventually implemented. Overall, the committee believes that there will need to be a combination of different incentives in place and transition to mandates only as appropriate.

Recommendation: NASA Science Mission Directorate should consider a variety of policy options depending on discipline and software type and transition to greater openness over time.

With all that in mind, the committee presents the following three open source policy options at a basic level:

Option A: Continue status quo.

Option B: Incentivize openness.

Option C: Mandate openness.

Continuing the status quo (Option A), allows individual SMD programs to determine whether they and their research communities are interested in moving toward open source. Ideally, regardless of what NASA decides, as accepted research metrics begin to routinely include a measure related to software impact, OSS will gradually become more common at least in some disciplines. Option A could eventually lead to OSS being required or a de facto norm in some areas, but in others it would remain unusual. This is the least strategic option and is unlikely to bring much change or fully realize the value of OSS.

SMD could accelerate the move toward openness, especially in programs less inclined to it, by offering specific incentives to investigators (Option B). This could be the long-term policy of SMD or could be a step on the way to some level of SMD-mandated openness (Option C). These incentives can also contribute to fostering an OSS culture and norm of sharing. Section 5.4 describes how incentives and mandates can be combined for different types of software.

5.1 POLICY OPTION A: CONTINUE STATUS QUO

As stated earlier, SMD does not currently have division-wide policy regarding software publishing, distribution, or licensing. Option A continues to allow individual NASA programs to determine whether they and their research communities are interested in moving toward open source. Some programs and modeling centers have taken steps toward openness, but there are no SMD coordinated open-source requirements, education efforts, or trainings. Several examples of existing policies for parts of SMD are discussed in Chapter 3. The community concerns raised in Chapter 4 would mostly remain unaddressed. This option could lead (and may have already led) to inequalities in access to results if some, but not all, programs mandated OSS. Without guidance from NASA, others (including publishers who are increasingly requiring OSS) will determine the course of OSS policies.

Option A also leads to approaches customized to certain communities. For example, some large modeling centers, such as NASA Community Coordinated Modeling Center (CCMC), have what might be called an “open-to-run” approach. They host software packages and run them on in-house hardware, possibly on thousands of CPUs for months at a time. This allows outsiders to run software at the center with cases and configurations of their choice, but under various restrictions. Usually, the software cannot be taken elsewhere. Sometimes the source code is viewable; sometimes not. This ad hoc approach may be a pragmatic compromise, because the software cannot be practically run on other systems. It may also be a way to provide some protection for the investigators developing the software while satisfying some level of research transparency. Investigators may have some ability to reproduce results and learn from the software depending on the restrictions, but this is not OSS.

5.2 POLICY OPTION B: INCENTIVIZE OPENNESS TO ACCELERATE THE CHANGE

Option B uses incentives that preserve community interests while moving to OSS. The goal of this option is to build trust while working toward making openness a community norm. With mandates absent or delayed, community pressure toward openness would naturally increase as investigators compete for the incentives.

As described in Chapters 3 and 4, some policies, such as the compulsory inclusion of a data management plan, was initially not well understood and was seen as a burden to most investigators, even those who do not produce any data. As education efforts and online resources increased, data management plans became a normal section to all proposals. The archive of widely used data at NASA archive centers, rather than individual investigator’s websites, has improved the quality, availability, and usability of NASA investigator-produced data. NASA’s Living with a Star (LWS) program offered researchers funds to make widely used data available to the general community and was better received than unfunded mandates. The implementation of an OSS policy could learn from these experiences, through implementation of options given below.

Success with Option B depends on the allocation of adequate resources. Incentives within the current budget that lead to reduction of research funds will be less accepted by the community. There may be a delay in the scientific return from research funding. Over the long term, however, motivation to move toward more openness is likely to provide a net benefit to science, as more researchers take advantage of the opened software. Careful consideration and guidance to proposal review panels would be required to award incentives to those proposals

with software that is more likely to be reused. This option also recognizes the full cost of community software development.

Because openness incentives may be applied at different rates and perhaps be absent at other governmental agencies, some researchers may not participate, possibly gaining a research or career advantage over those who devote time and resources to opening their software. For example, if one program manager or agency called for OSS and another did not, a researcher might target proposals to programs that allowed them to delay releasing their software.

The five incentives identified by the committee are the following:

B1. Funding for new proposals specifically addressing an OSS need

B2. Funding augmentations or components of proposals

B3. Piloting the use of software management plans (SMPs) in some programs

B4. Supporting open source libraries and infrastructure software

B5. Offering a prize for exemplary contributions to OSS in the NASA science community.

One or more of these elements could be adopted as part of Option B. Table 5.1 summarizes how these may apply to the different software types presented in Chapter 2.

5.2.1 Option B1—Funding for Full Open Source Software Proposals

With Option B1, SMD or its divisions allocate funding, either in existing or new grant programs, to open existing software with community reuse potential or replace it with functionally equivalent OSS, to develop new OSS, to maintain existing OSS, or to extend community open source libraries and frameworks. In this case, the proposal provides a software management plan (see Box 5.1) that describes what the software does, its value and user community, the needed software and documentation updates, test suites, licensing, any other legal issues, and a plan for long-term sustainability. The funding could be delivered as contracts or cooperative agreements to allow full oversight with milestones and deliverables.

Similar to the implementation of NASA’s data management plans, a corresponding NASA website could explain what is required for SMPs and provide suggestions and examples to ensure less-experienced researchers are not disadvantaged. Online tools (e.g., https://dmptool.org) could be developed to further help investigators.

Pros: Option B1 provides direct funding for scientists to develop OSS solutions. It allows for a prioritized approach, as evaluation panels of community experts decide which software to support. It recognizes the cost of community software development and it creates or builds scientific software projects that other researchers can reuse, accelerating science.

Cons: Option B1 could delay scientific returns within programs implementing it, as scientists spend time to gain experience and familiarity releasing software. It opens only some software, and it may provide a disincentive for groups who do not win funding to open their software. It may not fully recognize the long-term costs of maintaining the software.

5.2.2 Option B2—Optional Proposal Open Source Add-On

With Option B2, SMD provides an opportunity to add additional pages to scientific research proposals in existing grants programs to justify additional effort and funding to open software from the project and to provide an SMP. The review panel and program manager evaluate add-on proposals and allocate funds to the best of them. Unlike Option B1, the OSS management plan and funding is an augmentation to the scientific proposal.

The pros and cons are similar to those for Option B1, with the difference that there may be situations where the scientific merit of the proposal is not rated high enough for support, but the open source add-on is seen as of significant value.

TABLE 5.1 Policy Recommendation Summary by Software Type

A: Continue Existing Policy	B1: Incentivize with New Opportunity	B2: Incentivize with Proposal Add-on	B3: Pilot SMP for New Development	B4: Support Existing Open Infrastructure	B5: Prize	C: Mandate Open Source	Remarks
Libraries	x	✓ a	✓	✓	✓	✓	Immediate for new software or additions to existing OSS
Single-use software	✓	x	✓	* b	x	✓	Last priority; most difficult to implement	A greater priority for journals implementing reproducible research standards.
Analysis software	x	*	✓	✓	*	✓	New tools ASAP ~ 5 years
M&S software	x	*	✓	✓	*	✓	Variable
Frameworks	x	✓	✓	✓	x	✓	Immediate for new; as possible for old
Sensor software	x	✓ c	x	✓	x	✓	Immediate	Include in new AOs as soon as possible.
Infrastructure software	*	x	*	*	*	✓	When necessary for science and to ensure competition and good practice

a Yes, for maintaining current libraries and creating major new libraries.

b Items with an asterisk (*) indicate that the policy may be applied in some programs before others.

c Yes, for high priority, general-use legacy pipeline.

NOTE: General approach of moving toward a full SMD-wide OSS policy based on software types described in Chapter 2 (see Table 2.1). The table indicates which policy options may be appropriate for each software type and the recommended time frame for moving to mandated openness (Option C).✓ means that an option can be implemented right now; * means that it might be implemented in some programs before others, and likely with lower priority; and x means that the option is not appropriate for that code type. An x might appear because going straight to a mandate is recommended for that software type (e.g., sensor software), or because a more comprehensive solution involving OSS and other requirements would be necessary (e.g., single use). More detailed descriptions and recommendations are given below.

BOX 5.1
Software Management Plans

Software management plans (SMPs) with specific requirements are already required as part of new NASA mission and supporting infrastructure proposals (e.g., SWE-102 1 ). Within this report, an SMP is a document that concisely describes all new software produced during a research project; intellectual property or legal issues that might arise; and how the software will be developed, shared (or not shared), and archived during the project.

Most of the policy options described will require SMPs, which would include the following:

A description of software that will be produced during the funded research and other components used (OSS, COTS, SaaS, etc.). It may be appropriate to separate software into categories for this discussion and following information.
Version control processes.
The license to be used (if released).
Any legal issues involved in the software components (e.g., export control and ITAR restrictions).
Long-term sustainability plan (if appropriate).

SMPs that involve releasing software would also address the following:

Analysis of risks in releasing software (e.g., security, intellectual property, and licensing).
A software release schedule.
Testing and integration processes.
Software distribution and archiving.
Documentation provided to the community.
Methods for incorporating contributions from the community into the software and communicating those contributions.
Evidence of understanding and commitment to established open source development practices.
Budget description for software development, documentation, distribution, support, publication, and maintenance.

5.2.3 Option B3—Pilot Software Management Plans

With Option B3, specific programs within SMD begin to require SMPs for scientific proposals containing substantial new software development. Requiring an SMP would not mandate openness, but it could gradually expand existing policy and impose more specific requirements over time. The goal is to gradually develop an effective policy by identifying different approaches to making software more open and responding to community feedback.

Where researchers receive NASA funds to deliver OSS, they also need to provide evidence of OSS practices. The SMP could be added as an evaluation criterion during proposal reviews.

Pros: An SMP would provide an opportunity for the research community to document and be evaluated on their software practices and for NASA to learn from the experience. It will allow reviewers and program managers to understand how knowledgeable and experienced the proposers are at OSS development.

Cons: Option B3 imposes an additional requirement that researchers must adhere to and evaluators must consider. If implemented rapidly in scientific communities unfamiliar with OSS, innovation and science could suffer due to either inexperience making software open or the additional time spent by researchers on software.

It is unclear what implications successful proposers would face if their stated SMP goals were unmet, except by evaluating previous practices, which would only apply to previous OSS funding from NASA.

5.2.4 Option B4—Support Open Source Libraries and Infrastructure Software

With Option B4, SMD uses existing funding mechanisms or allocates SMD employee time to support and adopt open source libraries and infrastructure software that are widely used in NASA-funded research. NASA support of software will demonstrate its commitment and promote the careers of scientists who spend time developing and improving these libraries. Because it is focused on broadly useful software, share-in-savings contracts might be one useful approach in this option. 1

Pros: Option B4 could improve community software quality and generate savings for NASA as a whole. With agency support, community software will increase in visibility and value, which will help the careers of researchers who develop them.

Cons: Some software of this type currently exists without dedicated NASA funding.

5.2.5 Option B5—Create an Annual Prize for the “Advancement of OSS Development and Impact”

With Option B5, greater recognition for scientists who create quality OSS would enhance their career advancement. A NASA SMD award or prize could provide some recognition and visibility for the importance of OSS. This might be similar to NASA’s current “Software of the Year” award, 2 but the focus would be on open source. The prize would recognize how OSS provides value to NASA. The prize could be judged on a number of criteria including, but not limited to, code contributions, software reuse or extension, training, and advocacy, as documented by application materials or testimonials and letters of nomination. NASA could consider partnering with scientific professional societies to present and administer the prize.

Pros: The prize provides an incentive and recognition for scientists who create quality OSS, while highlighting the importance of NASA SMD OSS resources.

Cons: The prize will require resources and take time away from other activities, as SMD needs to create the prize, publicize it, organize a review committee, review applications, and make a selection. Awards and prizes are not nearly as effective at advancing scientific careers as funded proposals unless they are extremely prestigious.

5.3 POLICY OPTION C: MANDATE OPENNESS

Under Option C, SMD decides that, by a certain date, software created through SMD funding will be open source, with only few, strongly justified exceptions. Mitigating the concerns raised in the prior chapters requires a period of transitional activities that may vary in timing and implementation by program and software type (Table 5.1) and sustained strategic investment.

Pros: An OSS mandate would be the surest and quickest way to increase the transparency of NASA science and to satisfy the recent Office of Management and Budget (OMB) memorandum “Federal Source Code Policy: Achieving Efficiency, Transparency, and Innovation through Reusable and Open Source Software” 3 and NASA CIO response to “encourage vendors to use open source technology wherever possible.” 4 It could potentially enhance NASA’s national and international reputation as a leader in open science. Experience with open data policies suggest than an OSS mandate could drive other agencies, both nationally and internationally, to enact similar policies, resulting in more OSS, thereby benefiting NASA researchers.

1 Share-in-savings contracts are where “the Government awards a contract to improve mission-related or administrative processes or to accelerate the achievement of its mission and share with the contractor in savings achieved through contract performance,” https://www.law.cornell.edu/uscode/text/10/2332.

Allowing flexibility in the timing and implementation of a mandate for different software types or programs could improve buy-in and understanding from the NASA research community, especially if NASA uses the time to implement training programs and some of the incentives under Option B. This flexibility in implementation would also allow costs to be spread over time and offers more flexibility in budgeting.

Cons: A rapid mandate would likely cause backlash from NASA-funded investigators. It would likely put a financial strain on NASA-funded investigators, especially those on soft money and could lead to a drain of people from the field. The response from the community may be to follow the letter but not the spirit of the policy. A mandate may also be the costliest option, requiring major enabling and sustaining infrastructure to enforce the mandate. The cost of implementing mandates could exceed the benefit for some software types. A mandate might also hinder collaboration with other agencies in creating OSS, notably the Department of Defense, which would have to provide permission and may have additional security concerns to protect controlled unclassified information and export-controlled information (WP 31 5 ). Last, behavioral research suggests that positive incentives can effectively drive both individual and group behavior, so mandates are more effective once incentives have been established. 6

Conclusion: Immediately mandating open source across all software types and in all of SMD could damage the NASA science enterprise.

If mandates are implemented, a transition period toward openness is necessary with specific activities to help level the playing field, provide training and resources, and ensure research continuity. In many cases, incentives will need to be in place before firm mandates can be implemented. The transition schedule may need to be modified based on the response to incentives as they are developed. Program managers will play an important role by tailoring their policy execution to their understanding of communities and culture across SMD, to incremental needs depending on software types, and to community experience with open science practices.

Conclusion: An incentive-driven transition period is needed before a comprehensive SMD open source software policy is implemented. Incentives and timelines will vary by software type and community experience.

5.4 TRANSITIONING TOWARD OPENNESS

As mentioned, SMD should consider a variety of policy options depending on discipline and software type and transition to greater openness over time. In Box 5.2, the committee outlines one example of a path to openness for a program or software type that moves to an OSS requirement in 3 years. The committee considers 3 years the minimum transition time applicable to only some software types or communities (e.g., new instrument code or pilot efforts such as NASA’s Earth Science Directorate’s ACCESS). Many programs or software types will transition more slowly because of different grant cycles, infrastructure availability, and general community readiness; but they will follow the same general path. Achieving SMD-wide openness means implementing this path for every program and for all software types. There will, however, be limitations. Some software cannot legally be open source; it may simply be too expensive to convert some legacy software; and different software will have different levels of maintenance (sometimes none). SMD will need to continually balance trade-offs and priorities while continually assessing how policies are meeting their goals. Early assessments are critical in policy implementation and establish a baseline for long-term assessments (e.g., reuse, publications, citations). This transition to a desired level of openness requires time and resources for training, software support and maintenance, and contributions to the overall software infrastructure. Introducing OSS requirements without strategic investment in software development and maintenance may not advance innovation and discovery and other goals.

5 WP is used to reference the white papers submitted to the committee. See Appendix C for a full listing.

6 E.E. Smith, S. Nolen-Hoeksema, B.L. Fredrickson, G.L. Loftus, D.J. Bem, and S. Maren, 2003, Atkinson and Hilgard’s Introduction to Psychology, Wadsworth/Thomson Learning, Belmont, CA.