Developers Guide & Project Plan

This document includes some details relevant for developers working on any of the Python Prognostics Packages Tools (prog_models, prog_algs, prog_server)

Installing from a Branch

To install the package package from a specific branch. First clone the repository and checkout the branch. Then navigate into the repository directory and use the following command:

pip install -e .

This command installs the package using the checked-out version.

Contributing

New external (non-NASA or NASA contractor) developers must complete either the organizational or individual Contributor License Agreement (CLA).

Curious about what needs to be done? Have an idea for a new feature? Find a bug? Check out open issues prog_models, prog_algs, prog_server.

Notes for Developers

  • Configuration-controlled items: source code, tests, test workflow, issues (requirements, non-conformances, bugs, etc.), milestones, examples, tutorial, templates, project plan (this document), and documentation

  • The package itself is stored in the src directory.

  • This is a research tool, so when making a design decision between operational efficiency and usability, generally choose the more usable option

  • When supplied by or to the user, values with names (e.g., inputs, states, outputs, event_states, event occurance, etc.) should be supplied as dictionaries (or dict-like objects) where they can be referred to by name.

  • Utils, Visualize, and metrics subpackages shall be independent (i.e., not have any dependencies with the wider package or other subpackages)

  • Whenever possible Models, UncertainData types, State Estimators, and Predictors should be interchangable with any other class of the same type (e.g., any model should be interchangable with any other model)

  • Demonstrate common use cases as an example.

  • Every feature should be demonstrated in an example * The most commonly used features should be demonstrated in the tutorial

  • Except in the most extreme cases, maintain backwards compatibility for the convenience of existing users * If a feature is to be removed, mark it as depreciated for at least 2 releases before removing

  • Examples are included in the examples/ directory.
    • Examples should cover the major use cases and features. If a major new feature is added, make sure there’s an example demonstrating the feature.

    • For new examples- add to examples __all__ and example tests (tests/test_examples).

    • Also for new examples- add to getting started page in sphinx_config.

    • Examples should include comments explicitly describing each step.

  • Tests are included in the tests/ directory.
    • Each new feature should have a test. Check this in each PR review.

    • Check test coverage to improve completeness, automatically reported by bot in each PR.

    • For tests- make sure test are quality. They should cover expected input ranges, error handling.

  • Documentation
  • GitHub Actions Test Workflow
    • Automated tests are defined in the .github/ directory.

    • The repository administrator can add tests to the set required to pass for each PR must be done by .

  • Template
    • An empty template of a prognostics model is maintained at prog_models/prog_model_template.py.

    • An empty template of a state estimator and predictor is maintained at prog_algs/state_estimator_template.py and prog_algs/predictor_template.py.

    • Any changes to the basic model setup should be documented there.

  • A tutorial is included in tutorial.ipynb. This required Juypter Notebooks. All major features should be illustrated here.

Project Roles

Branching Strategy

Our project is following the git strategy described here. Release branches are not required. Details specific to each branch are described below.

master: Every merge into the master branch is done using a pull request (never commiting directly), is assigned a release number, and must comply with the release checklist. The release checklist is a software assurance tool.

dev: Every commit on the dev branch should be functional. All unit tests must function before commiting to dev or merging another branch. Merging into dev should only be done through a PR, unless only documentation has been updated.

Feature Branches: These branches include changes specific to a new feature. Before merging into dev unit tests should all run, tests should be added for the feature, and documentation should be updated as appropriate.

PR Checklist

  • Each PR into dev or master must be checked by someone on the team other than the author.
    • Reviewer should look for bugs, efficiency, readability, testing, and coverage in examples (if relevant).

  • Each PR adding a new feature should include a test verifying that feature.

  • All tests must be passing.

  • All errors from static analysis must be resolved.

  • Any added dependencies are included in requirements.txt

  • All new files have the following notice on them:
    “Copyright © 2021 United States Government as represented by the Administrator of the National Aeronautics and Space Administration. All Rights Reserved.”
  • All warnings from static analysis must be reviewed and resolved - if deemed appropriate.

  • Final merging can only be performed by the software assurance officer.

  • For merging into master branch - see the Release Checklist below.

Release Checklist

  • [Complete - done in PRs to dev] Code review - all software must be checked by someone other than the author

  • Check that each new feature has corresponding tests

  • [Complete - checked automatically in PRs to dev] Confirm that every page has the copyright notice

  • Confirm dependencies in requirements.txt, setup.py

  • Confirm that all issues associated with the release have been closed (i.e., requirements have been met) or assigned to another release

  • Run unit tests python -m tests

  • For prog_models: run manual tests python -m tests.test_manual

  • Review the template(s)

  • Review static-analysis/linter results

  • Test the tutorial

  • Run the examples

  • Check that all examples are tested

  • Check new files in PR for any accidentally added

  • Check documents
    • Check that all desired examples are in docs

    • General review: see if any updates are required

  • Rebuild sphinx documents: sphinx-build sphinx_config/ docs/

  • Write release notes

  • Update version number in src/*/__init__.py and setup.py

  • For releases adding new features- ensure that NASA release process has been followed.

Post-Release Checklist

  • For prog_server: Update openapi specs on SwaggerHub

  • Send notes to Software Release Office (SRO) of updated version number

  • Publish to PyPi

  • Send Highlight

Updating Documentation

Use the following command to update documentation (requires sphinx): .. code-block: bash

sphinx-build sphinx_config docs

Uploading new version to PyPI

New versions are uploaded upon release (i.e., merging into master branch). The Release Checklist must be complete prior to release

See here

NPR 7150

NASA software must comply with the requirements enumarated in NPR 7150. See the NASA Software Engineering and Assurance Handbook for more information.

  • Software Classification: Class-E (Research Software)

  • Safety Criticality: Not Safety Critical

  • Assessment done by Christopher Teubert in March 2021

Intention is to expand the scope of use to also include its inclusion in Ground software tools that support mission planning or formulation; Ground software that operates a research, development, test, or evaluation laboratory (i.e., not a major engineering/research facility); or Ground software that provides decision support for non-mission-critical situations and airborn software whose anomalous behavior would cause or contribute to a failure of system function resulting in a minor failure condition for the airborne vehicle, or whose anomalous behavior would cause or contribute to a failure of system function with no effect on airborne vehicle operational capability or pilot workload. The classification for this increased scope is described below

  • Effective version: v1.4 (Est. Fall 2022)

  • [Future] Software Classification: Class-D (Basic Research and Technology Software)

  • [Future] Safety Criticality: Not Safety Critical

  • Assessment done by Christopher Teubert in April 26 2022

The safety criticality assessment was made based on the responses to the following questions:

  • Does the software reside in a safety-critical system? NO

  • The software causes or contributes to a hazard? NO

  • The software provides control or mitigation for hazards? NO

  • The software controls safety-critical functions? NO

  • The software processes safety-critical commands or data? NO

  • The software detects and reports, or takes corrective action, if the system reaches a specific hazardous state? NO

  • The software mitigates damage if a hazard occurs? NO

  • The software resides on the same system (processor) as safety-critical software? NO

  • Does the software process data or analyze trends that lead directly to safety decisions (e.g., determining when to turn power off to a wind tunnel to prevent system destruction)? NO

Compliance Notation Legend

  • FC: Fully Compliant

  • T: Tailored (Specific tailoring described in mitigation) SWE-121

  • PC: Partially Compliant

  • NC: Not Compliant

  • NA: Not Applicable

Compliance Matrix

This matrix tracks the compliance of this software and the project’s software engineering practices with the requirements from 7150.2.

Note: for requirements with different evidence for the different softwares, they have a capital M (for ProgModels), A (for ProgAlgs) or S (for ProgServer) following the requirement.

Life Cycle Management

SWE #

Description

Compliance

Evidence

033

Assess Aquisiton Options

FC

See section below

013

Maintain Software Plans

FC

This document

024a

Conformance with Project Plan

NC

Not required for Class-E

024b

Project Plan Configuration Mgmt

FC

Project plan is a tracked object in github

034

Software Acceptance Criteria

NC

Not required for Class-E

036

Software Processes

FC

See notes below

037M

Document Milestones

FC

Milestones

037A

Document Milestones

FC

Milestones

037S

Document Milestones

FC

Milestones

039a

Monitor integration

FC

Integration status visible on github pull requests

039b

Review Verification Activities

FC

Verification status and results visible on github actions

039c

Review Trade Studies

FC

Trade studies will be documented in github issues

039d

Audit Development Processes

FC

Development processes are visible in pull requests and commits

039e

Software Reviews

FC

Software Assurance Officer gives final approval after reviews

040aM

Products

FC

Kept in Repo

040aA

Products

FC

Kept in Repo

040aS

Products

FC

Kept in Repo

040b

Tracability

FC

Maintained in PR and issue documentation

040c

Non-conformances

FC

See github issues

040dM

Change tracking

FC

See Commits

040dA

Change tracking

FC

See Commits

040dS

Change tracking

FC

See Commits

042

Electronic Accesss to Source

FC

See respective github repository

139

Comply with 7150

FC

This document

121

Tailored Reqs

NA

No tailoring

125

Compliance Matrix

FC

This document

029

Software Classification

FC

This document

027

COTS/GOTS/MOTS/OSS

NC

Not Required for Class-E

Note on software processes [SWE 034]: The project manager shall establish and maintain the software processes, software documentation plans, list of developed electronic products, deliverables, and list of tasks for the software development that are required for the project’s software developers, as well as the action required (e.g., approval, review) of the Government upon receipt of each of the deliverables. * Processes are tracked in this document. * Documentation, electronic products, and deliverables are tracked in GitHub * Tasks are tracked in GitHub issues * Actions required are listed in the checklists above

Cost Estimation

SWE #

Description

Compliance

Evidence

015

Maintain 1 cost estimate

NA

See note below

151

Cost Estimate Requirements

NA

See note Below

174

Submit Planning Parameters

NA

Specified Center measurement repo does not exist

Software Cost Estimation Note:

The Python Prognostics Packages are a collaborative product of multiple projects. As projects use the software they will implement features and fix bugs to accomplish the goals of their projects, with some input from the Project Manager as the chief software architect. These improvements are incorporated into the shared product for the all participating projects to benefit from.

Responsibility for cost accounting for these contributions is delegated to the project(s) conducting them. Cost of contributing to the Prognostics Python Packages should be represented in their project plans and other documents.

Schedules

SWE #

Description

Compliance

Evidence

016M

Schedule Requirements

FC

Milestones

016A

Schedule Requirements

FC

Milestones

016S

Schedule Requirements

FC

Milestones

046

Maintain Schedule

FC

See Milestones (from SWE016, above)

Classification

SWE #

Description

Compliance

Evidence

020

Software Classification

FC

This document

176

Software Classification

FC

This document

Software Assurance

SWE #

Description

Compliance

Evidence

022

Software Assurance

FC

This document

Safety Critical Software

SWE #

Description

Compliance

Evidence

205

Safety Cricial Software

FC

See above

023

Safety Critical Reqs

NA

Not safety critical

134

Safety Critical Reqs

NA

Not safety critical

219

Safety Critical Reqs

NA

Not safety critical

220

Safety Critical Reqs

NA

Not safety critical

Automatic Generation of Source Code

SWE #

Description

Compliance

Evidence

206

Autogen Software

NA

No autogen

Reuse

SWE #

Description

Compliance

Evidence

147

Reusability Requirements

NC

Not required for Class-E

148M

Software Catalog

FC

prog_models

148A

Software Catalog

FC

prog_algs

148S

Software Catalog

FC

Will be posted soon

Cybersecurity

SWE #

Description

Compliance

Evidence

156

Perform CyberSecurity Assessment

FC

See below

154

Perform CyberSecurity Risks

NC

Not Required for Class-E

157

Protect Against Unauthorized Access

NC

Not Required for Class-E

159

Test CyberSecurity Mitigation

NC

Not Required for Class-E

207

Secure Coding Practices

NC

Not Required for Class-E

185

Static Analysis

FC

See Static Analysis Notes under Implementation

Cybersecurity risks were assessed, the identified cybersecurity threats and our mitigations are described below:
  • Code injection * Risk: insertion of hazardous code into an open-source project by malicious actor * Mitigation: Strict code review requirements in the repository. Static analysis/security alerts. Vetting for contributors.

  • Dependencies * Risk: Dependencies could introduce cybersecurity vulnerabilities * Mitigation: GitHub “dependabot” alerts will identify any known issues with package decencies. Also, the project is actively trying to limit the number of dependencies, and only use well-known packages from trusted developers.

  • Language * Risk: Python itself may introduce cybersecurity vulnerabilities * Mitigation: Python is a well-known language, this risk is low. To mitigate this we only support actively maintained versions.

Bi-Directional Traceability

SWE #

Description

Compliance

Evidence

052

Tracability

FC

See Tracability Notes, below

Requirements

SWE #

Description

Compliance

Evidence

050M

Software Requirements

FC

Enhancement Issues

050A

Software Requirements

FC

Enhancement Issues

050S

Software Requirements

FC

Enhancement Issues

053

Requirement Change Tracking

FC

Tracked in enhancement issues, see comment from SWE050, above

054

Track Inconsistencies

FC

Tracked in enhancement issues, see comment from SWE050, above

055

Requirement Validation

NC

Not Required for Class-E

Implementation

SWE #

Description

Compliance

Evidence

061

Coding Standards

NC

Not Required for Class-E

135

Static Analysis

FC

See list of static analysis tools, below.

062

Unit Testing

FC

Unit tests are created with each enhancement, run automatically with each PR.

186

Unit Test Repeatability

FC

Unit tests are created with each enhancement, run automatically with each PR.

063M

Software Version Description

FC

See here

063A

Software Version Description

FC

See here

063S

Software Version Description

FC

See here

Static Analysis Methods Used: * CodeFactor.io (prog_models, prog_algs, prog_server): Runs automatically in each PR. If issues are detected, they are noted in the PR chat. * LGTM (prog_models, prog_algs, prog_server): Runs automatically in each PR. If issues are detected, they are noted in the PR chat. * Codecov (prog_models, prog_algs, prog_server): Runs automatically in each PR. If issues are detected, they are noted in the PR chat. * CodeQL Scanning: Runs automatically in each PR. If issues are detected, they are noted in the PR chat. * Github Dependabot Alerts: Tracks dependencies, alerts of any issues.

Testing

SWE #

Description

Compliance

Evidence

065a

Test Plan

FC

See this document.

065bM

Test Procedures

FC

See GitHub Actions Workflows.

065bA

Test Procedures

FC

See GitHub Actions Workflows.

065bS

Test Procedures

FC

See GitHub Actions Workflows.

065cM

Tests

FC

See tests directory.

065cA

Tests

FC

See tests directory.

065cS

Tests

FC

See tests directory.

065dM

Test Reports

FC

See Github Actions Results.

065dA

Test Reports

FC

See Github Actions Results.

065dS

Test Reports

FC

See Github Actions Results.

066

Verification

FC

Each requirement has verification tests created before closing. Tests run using GitHub actions

068

Evaluate Test Results

NC

Not required for Class-E

071

Update Test Plans

FC

Workflow, tests, and this document are updated as requirements change

186M

Code Coverage

FC

See Codecov

186A

Code Coverage

FC

See Codecov

186S

Code Coverage

FC

See Codecov

192

Reqs that Trace to Hazard

Not required for Class-E

Operations, Maintenance, and Retirement

SWE #

Description

Compliance

Evidence

075

Ops, Maintenance, and Retirement

NC

Not required for Class-E

077

Record-Keeping

FC

Records maintained in repository and Sharepoint folder

194

Delivery Verification

FC

See Release Checklist and automated unit and verification tests

195

Maintainance Standards

FC

See this document, GitHub records

196

Retirement Plan

NC

Not required for Class-E

Configuration Management

SWE #

Description

Compliance

Evidence

079

Configuration Management Plan

FC

See this document

080M

Evaluate Sotware Product Changes

FC

See PRs

080A

Evaluate Sotware Product Changes

FC

See PRs

080S

Evaluate Sotware Product Changes

FC

See PRs

081

Identify Configuration Items

FC

See this document

082a

Levels of Control

FC

See this document

082b

Authorization Authority

FC

See this document

082c

Authorization Authority

FC

See this document

083M

Configuration Status

FC

See Branches

083A

Configuration Status

FC

See Branches

083S

Configuration Status

FC

See Branches

084

Configuration Audits

NC

Not Required for Class-E

085

Release Procedures

FC

See this document

Non-Conformances

SWE #

Description

Compliance

Evidence

201M

Track non-conformances

FC

See Github Issues

201A

Track non-conformances

FC

See Github Issues

201S

Track non-conformances

FC

See Github Issues

Transition to a Higher Class

SWE #

Description

Compliance

Evidence

021

Transition to a higher class

FC

Plans have been updated to reflect the updated classification

Aquisition Options

Assessed, there are some existing prognostics tools but no general Python package that can support model-based prognostics like we need and no general python package that can support model-based prognostics is an Service Oriented Architecture (SOA) like we need (prog_server).

Requirement Tracking

Requirements are tracked as issues with the “Enhancement” label (See prog_models, prog_algs, prog_server Enhancement Issues). An issue template is used to ensure that the requirement has the desired information. Issues are closed to indicate the requirement has been met. Closing a requirement issue is done with a pull request, which is linked to the relevant requirement, for tracability. Closing the requirement issue requires a code review (see above for details), and requires implementation of passing tests that test the requirement (i.e., verification tests). The tests are reviewed with the code implementing the requirement. Issues are assigned to a milestone (i.e., release) indicating the requirements for that release. Github automatically tracks any changes to the issues (i.e., requirements)

Tracability Notes

Hazards and non-conformances are tracked as issues with the label bug (See prog_models, prog_algs, prog_server). In the template for a bug report, there is a section asking for relevant enhancement issues (i.e., requirements). This linking establishes tracability from hazards/non-conformances to the underlying requirement. These linkings are automatically marked by the github system in the requirement issue. Additionally, to close an enhancement issue (i.e., requirement), passing verification tests must be created and checked in. The PR where these tests are created and the implementation is completed is linked to the issue establishing tracability from requirement -> verification test. These tests run automatically at every change/PR.

Additionally, requirements are assigned to milestones/releases, establishing bi-directional tracability to these

Summary: The following tracabilities are maintained: * Hazard <-> Requirement * Non-conformance <-> Requirement * Requirement <-> Verification Test & Results * Requirement <-> Implementation * Release/Milestone <-> Requirement