Developers Guide & Project Plan¶
This document includes some details relevant for developers working on any of the Python Prognostics Packages Tools (prog_models, prog_algs, prog_server)
Contents
Installing from a Branch¶
- To install the package package from a specific branch. First clone the repository and checkout the branch. Then navigate into the repository directory and use the following command:
pip install -e .
This command installs the package using the checked-out version.
Contributing¶
New external (non-NASA or NASA contractor) developers must complete either the organizational or individual Contributor License Agreement (CLA).
Curious about what needs to be done? Have an idea for a new feature? Find a bug? Check out open issues prog_models, prog_algs, prog_server.
Notes for Developers¶
Configuration-controlled items: source code, tests, test workflow, issues (requirements, non-conformances, bugs, etc.), milestones, examples, tutorial, templates, project plan (this document), and documentation
The package itself is stored in the src directory.
This is a research tool, so when making a design decision between operational efficiency and usability, generally choose the more usable option
When supplied by or to the user, values with names (e.g., inputs, states, outputs, event_states, event occurance, etc.) should be supplied as dictionaries (or dict-like objects) where they can be referred to by name.
Utils, Visualize, and metrics subpackages shall be independent (i.e., not have any dependencies with the wider package or other subpackages)
Whenever possible Models, UncertainData types, State Estimators, and Predictors should be interchangable with any other class of the same type (e.g., any model should be interchangable with any other model)
Demonstrate common use cases as an example.
Every feature should be demonstrated in an example * The most commonly used features should be demonstrated in the tutorial
Except in the most extreme cases, maintain backwards compatibility for the convenience of existing users * If a feature is to be removed, mark it as depreciated for at least 2 releases before removing
- Examples are included in the examples/ directory.
Examples should cover the major use cases and features. If a major new feature is added, make sure there’s an example demonstrating the feature.
For new examples- add to examples __all__ and example tests (tests/test_examples).
Also for new examples- add to getting started page in sphinx_config.
Examples should include comments explicitly describing each step.
- Tests are included in the tests/ directory.
Each new feature should have a test. Check this in each PR review.
Check test coverage to improve completeness, automatically reported by bot in each PR.
For tests- make sure test are quality. They should cover expected input ranges, error handling.
- Documentation
Documentation is autogenerated using sphinx.
Configuration is in sphinx_config.
Documentation is rebuilt on each release.
On each release, documentation can be seen at nasa.github.io/prog_models, nasa.github.io/prog_algs, and nasa.github.io/prog_server.
- GitHub Actions Test Workflow
Automated tests are defined in the .github/ directory.
The repository administrator can add tests to the set required to pass for each PR must be done by .
- Template
An empty template of a prognostics model is maintained at prog_models/prog_model_template.py.
An empty template of a state estimator and predictor is maintained at prog_algs/state_estimator_template.py and prog_algs/predictor_template.py.
Any changes to the basic model setup should be documented there.
A tutorial is included in tutorial.ipynb. This required Juypter Notebooks. All major features should be illustrated here.
Project Roles¶
Project Lead/Software Assurance Officer: Christopher Teubert
Developers: See prog_models developers, prog_algs developers, prog_server developers
Branching Strategy¶
Our project is following the git strategy described here. Release branches are not required. Details specific to each branch are described below.
master: Every merge into the master branch is done using a pull request (never commiting directly), is assigned a release number, and must comply with the release checklist. The release checklist is a software assurance tool.
dev: Every commit on the dev branch should be functional. All unit tests must function before commiting to dev or merging another branch. Merging into dev should only be done through a PR, unless only documentation has been updated.
Feature Branches: These branches include changes specific to a new feature. Before merging into dev unit tests should all run, tests should be added for the feature, and documentation should be updated as appropriate.
PR Checklist¶
- Each PR into dev or master must be checked by someone on the team other than the author.
Reviewer should look for bugs, efficiency, readability, testing, and coverage in examples (if relevant).
Each PR adding a new feature should include a test verifying that feature.
All tests must be passing.
All errors from static analysis must be resolved.
Any added dependencies are included in requirements.txt
- All new files have the following notice on them:
- “Copyright © 2021 United States Government as represented by the Administrator of the National Aeronautics and Space Administration. All Rights Reserved.”
All warnings from static analysis must be reviewed and resolved - if deemed appropriate.
Final merging can only be performed by the software assurance officer.
For merging into master branch - see the Release Checklist below.
Release Checklist¶
[Complete - done in PRs to dev] Code review - all software must be checked by someone other than the author
Check that each new feature has corresponding tests
[Complete - checked automatically in PRs to dev] Confirm that every page has the copyright notice
Confirm dependencies in requirements.txt, setup.py
Confirm that all issues associated with the release have been closed (i.e., requirements have been met) or assigned to another release
Run unit tests python -m tests
For prog_models: run manual tests python -m tests.test_manual
Review the template(s)
Review static-analysis/linter results
Test the tutorial
Run the examples
Check that all examples are tested
Check new files in PR for any accidentally added
- Check documents
Check that all desired examples are in docs
General review: see if any updates are required
Rebuild sphinx documents: sphinx-build sphinx_config/ docs/
Write release notes
Update version number in src/*/__init__.py and setup.py
For releases adding new features- ensure that NASA release process has been followed.
Post-Release Checklist¶
For prog_server: Update openapi specs on SwaggerHub
Send notes to Software Release Office (SRO) of updated version number
Publish to PyPi
Send Highlight
Updating Documentation¶
Use the following command to update documentation (requires sphinx): .. code-block: bash
sphinx-build sphinx_config docs
Uploading new version to PyPI¶
New versions are uploaded upon release (i.e., merging into master branch). The Release Checklist must be complete prior to release
See here
NPR 7150¶
NASA software must comply with the requirements enumarated in NPR 7150. See the NASA Software Engineering and Assurance Handbook for more information.
Software Classification: Class-E (Research Software)
Safety Criticality: Not Safety Critical
Assessment done by Christopher Teubert in March 2021
Intention is to expand the scope of use to also include its inclusion in Ground software tools that support mission planning or formulation; Ground software that operates a research, development, test, or evaluation laboratory (i.e., not a major engineering/research facility); or Ground software that provides decision support for non-mission-critical situations and airborn software whose anomalous behavior would cause or contribute to a failure of system function resulting in a minor failure condition for the airborne vehicle, or whose anomalous behavior would cause or contribute to a failure of system function with no effect on airborne vehicle operational capability or pilot workload. The classification for this increased scope is described below
Effective version: v1.4 (Est. Fall 2022)
[Future] Software Classification: Class-D (Basic Research and Technology Software)
[Future] Safety Criticality: Not Safety Critical
Assessment done by Christopher Teubert in April 26 2022
The safety criticality assessment was made based on the responses to the following questions:
Does the software reside in a safety-critical system? NO
The software causes or contributes to a hazard? NO
The software provides control or mitigation for hazards? NO
The software controls safety-critical functions? NO
The software processes safety-critical commands or data? NO
The software detects and reports, or takes corrective action, if the system reaches a specific hazardous state? NO
The software mitigates damage if a hazard occurs? NO
The software resides on the same system (processor) as safety-critical software? NO
Does the software process data or analyze trends that lead directly to safety decisions (e.g., determining when to turn power off to a wind tunnel to prevent system destruction)? NO
Compliance Notation Legend¶
FC: Fully Compliant
T: Tailored (Specific tailoring described in mitigation) SWE-121
PC: Partially Compliant
NC: Not Compliant
NA: Not Applicable
Compliance Matrix¶
This matrix tracks the compliance of this software and the project’s software engineering practices with the requirements from 7150.2.
Note: for requirements with different evidence for the different softwares, they have a capital M (for ProgModels), A (for ProgAlgs) or S (for ProgServer) following the requirement.
Life Cycle Management¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
033 |
Assess Aquisiton Options |
FC |
See section below |
013 |
Maintain Software Plans |
FC |
This document |
024a |
Conformance with Project Plan |
NC |
Not required for Class-E |
024b |
Project Plan Configuration Mgmt |
FC |
Project plan is a tracked object in github |
034 |
Software Acceptance Criteria |
NC |
Not required for Class-E |
036 |
Software Processes |
FC |
See notes below |
037M |
Document Milestones |
FC |
|
037A |
Document Milestones |
FC |
|
037S |
Document Milestones |
FC |
|
039a |
Monitor integration |
FC |
Integration status visible on github pull requests |
039b |
Review Verification Activities |
FC |
Verification status and results visible on github actions |
039c |
Review Trade Studies |
FC |
Trade studies will be documented in github issues |
039d |
Audit Development Processes |
FC |
Development processes are visible in pull requests and commits |
039e |
Software Reviews |
FC |
Software Assurance Officer gives final approval after reviews |
040aM |
Products |
FC |
Kept in Repo |
040aA |
Products |
FC |
Kept in Repo |
040aS |
Products |
FC |
Kept in Repo |
040b |
Tracability |
FC |
Maintained in PR and issue documentation |
040c |
Non-conformances |
FC |
See github issues |
040dM |
Change tracking |
FC |
See Commits |
040dA |
Change tracking |
FC |
See Commits |
040dS |
Change tracking |
FC |
See Commits |
042 |
Electronic Accesss to Source |
FC |
See respective github repository |
139 |
Comply with 7150 |
FC |
This document |
121 |
Tailored Reqs |
NA |
No tailoring |
125 |
Compliance Matrix |
FC |
This document |
029 |
Software Classification |
FC |
This document |
027 |
COTS/GOTS/MOTS/OSS |
NC |
Not Required for Class-E |
Note on software processes [SWE 034]: The project manager shall establish and maintain the software processes, software documentation plans, list of developed electronic products, deliverables, and list of tasks for the software development that are required for the project’s software developers, as well as the action required (e.g., approval, review) of the Government upon receipt of each of the deliverables. * Processes are tracked in this document. * Documentation, electronic products, and deliverables are tracked in GitHub * Tasks are tracked in GitHub issues * Actions required are listed in the checklists above
Cost Estimation¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
015 |
Maintain 1 cost estimate |
NA |
See note below |
151 |
Cost Estimate Requirements |
NA |
See note Below |
174 |
Submit Planning Parameters |
NA |
Specified Center measurement repo does not exist |
Software Cost Estimation Note:
The Python Prognostics Packages are a collaborative product of multiple projects. As projects use the software they will implement features and fix bugs to accomplish the goals of their projects, with some input from the Project Manager as the chief software architect. These improvements are incorporated into the shared product for the all participating projects to benefit from.
Responsibility for cost accounting for these contributions is delegated to the project(s) conducting them. Cost of contributing to the Prognostics Python Packages should be represented in their project plans and other documents.
Schedules¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
016M |
Schedule Requirements |
FC |
|
016A |
Schedule Requirements |
FC |
|
016S |
Schedule Requirements |
FC |
|
046 |
Maintain Schedule |
FC |
See Milestones (from SWE016, above) |
Classification¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
020 |
Software Classification |
FC |
This document |
176 |
Software Classification |
FC |
This document |
Software Assurance¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
022 |
Software Assurance |
FC |
This document |
Safety Critical Software¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
205 |
Safety Cricial Software |
FC |
See above |
023 |
Safety Critical Reqs |
NA |
Not safety critical |
134 |
Safety Critical Reqs |
NA |
Not safety critical |
219 |
Safety Critical Reqs |
NA |
Not safety critical |
220 |
Safety Critical Reqs |
NA |
Not safety critical |
Automatic Generation of Source Code¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
206 |
Autogen Software |
NA |
No autogen |
Reuse¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
147 |
Reusability Requirements |
NC |
Not required for Class-E |
148M |
Software Catalog |
FC |
|
148A |
Software Catalog |
FC |
|
148S |
Software Catalog |
FC |
Will be posted soon |
Cybersecurity¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
156 |
Perform CyberSecurity Assessment |
FC |
See below |
154 |
Perform CyberSecurity Risks |
NC |
Not Required for Class-E |
157 |
Protect Against Unauthorized Access |
NC |
Not Required for Class-E |
159 |
Test CyberSecurity Mitigation |
NC |
Not Required for Class-E |
207 |
Secure Coding Practices |
NC |
Not Required for Class-E |
185 |
Static Analysis |
FC |
See Static Analysis Notes under Implementation |
- Cybersecurity risks were assessed, the identified cybersecurity threats and our mitigations are described below:
Code injection * Risk: insertion of hazardous code into an open-source project by malicious actor * Mitigation: Strict code review requirements in the repository. Static analysis/security alerts. Vetting for contributors.
Dependencies * Risk: Dependencies could introduce cybersecurity vulnerabilities * Mitigation: GitHub “dependabot” alerts will identify any known issues with package decencies. Also, the project is actively trying to limit the number of dependencies, and only use well-known packages from trusted developers.
Language * Risk: Python itself may introduce cybersecurity vulnerabilities * Mitigation: Python is a well-known language, this risk is low. To mitigate this we only support actively maintained versions.
Bi-Directional Traceability¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
052 |
Tracability |
FC |
See Tracability Notes, below |
Requirements¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
050M |
Software Requirements |
FC |
|
050A |
Software Requirements |
FC |
|
050S |
Software Requirements |
FC |
|
053 |
Requirement Change Tracking |
FC |
Tracked in enhancement issues, see comment from SWE050, above |
054 |
Track Inconsistencies |
FC |
Tracked in enhancement issues, see comment from SWE050, above |
055 |
Requirement Validation |
NC |
Not Required for Class-E |
Implementation¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
061 |
Coding Standards |
NC |
Not Required for Class-E |
135 |
Static Analysis |
FC |
See list of static analysis tools, below. |
062 |
Unit Testing |
FC |
Unit tests are created with each enhancement, run automatically with each PR. |
186 |
Unit Test Repeatability |
FC |
Unit tests are created with each enhancement, run automatically with each PR. |
063M |
Software Version Description |
FC |
|
063A |
Software Version Description |
FC |
|
063S |
Software Version Description |
FC |
Static Analysis Methods Used: * CodeFactor.io (prog_models, prog_algs, prog_server): Runs automatically in each PR. If issues are detected, they are noted in the PR chat. * LGTM (prog_models, prog_algs, prog_server): Runs automatically in each PR. If issues are detected, they are noted in the PR chat. * Codecov (prog_models, prog_algs, prog_server): Runs automatically in each PR. If issues are detected, they are noted in the PR chat. * CodeQL Scanning: Runs automatically in each PR. If issues are detected, they are noted in the PR chat. * Github Dependabot Alerts: Tracks dependencies, alerts of any issues.
Testing¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
065a |
Test Plan |
FC |
See this document. |
065bM |
Test Procedures |
FC |
|
065bA |
Test Procedures |
FC |
|
065bS |
Test Procedures |
FC |
|
065cM |
Tests |
FC |
See tests directory. |
065cA |
Tests |
FC |
See tests directory. |
065cS |
Tests |
FC |
See tests directory. |
065dM |
Test Reports |
FC |
|
065dA |
Test Reports |
FC |
|
065dS |
Test Reports |
FC |
|
066 |
Verification |
FC |
Each requirement has verification tests created before closing. Tests run using GitHub actions |
068 |
Evaluate Test Results |
NC |
Not required for Class-E |
071 |
Update Test Plans |
FC |
Workflow, tests, and this document are updated as requirements change |
186M |
Code Coverage |
FC |
See Codecov |
186A |
Code Coverage |
FC |
See Codecov |
186S |
Code Coverage |
FC |
See Codecov |
192 |
Reqs that Trace to Hazard |
Not required for Class-E |
Operations, Maintenance, and Retirement¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
075 |
Ops, Maintenance, and Retirement |
NC |
Not required for Class-E |
077 |
Record-Keeping |
FC |
Records maintained in repository and Sharepoint folder |
194 |
Delivery Verification |
FC |
See Release Checklist and automated unit and verification tests |
195 |
Maintainance Standards |
FC |
See this document, GitHub records |
196 |
Retirement Plan |
NC |
Not required for Class-E |
Configuration Management¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
079 |
Configuration Management Plan |
FC |
See this document |
080M |
Evaluate Sotware Product Changes |
FC |
See PRs |
080A |
Evaluate Sotware Product Changes |
FC |
See PRs |
080S |
Evaluate Sotware Product Changes |
FC |
See PRs |
081 |
Identify Configuration Items |
FC |
See this document |
082a |
Levels of Control |
FC |
See this document |
082b |
Authorization Authority |
FC |
See this document |
082c |
Authorization Authority |
FC |
See this document |
083M |
Configuration Status |
FC |
See Branches |
083A |
Configuration Status |
FC |
See Branches |
083S |
Configuration Status |
FC |
See Branches |
084 |
Configuration Audits |
NC |
Not Required for Class-E |
085 |
Release Procedures |
FC |
See this document |
Non-Conformances¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
201M |
Track non-conformances |
FC |
See Github Issues |
201A |
Track non-conformances |
FC |
See Github Issues |
201S |
Track non-conformances |
FC |
See Github Issues |
Transition to a Higher Class¶
SWE # |
Description |
Compliance |
Evidence |
---|---|---|---|
021 |
Transition to a higher class |
FC |
Plans have been updated to reflect the updated classification |
Aquisition Options¶
Assessed, there are some existing prognostics tools but no general Python package that can support model-based prognostics like we need and no general python package that can support model-based prognostics is an Service Oriented Architecture (SOA) like we need (prog_server).
Requirement Tracking¶
Requirements are tracked as issues with the “Enhancement” label (See prog_models, prog_algs, prog_server Enhancement Issues). An issue template is used to ensure that the requirement has the desired information. Issues are closed to indicate the requirement has been met. Closing a requirement issue is done with a pull request, which is linked to the relevant requirement, for tracability. Closing the requirement issue requires a code review (see above for details), and requires implementation of passing tests that test the requirement (i.e., verification tests). The tests are reviewed with the code implementing the requirement. Issues are assigned to a milestone (i.e., release) indicating the requirements for that release. Github automatically tracks any changes to the issues (i.e., requirements)
Tracability Notes¶
Hazards and non-conformances are tracked as issues with the label bug (See prog_models, prog_algs, prog_server). In the template for a bug report, there is a section asking for relevant enhancement issues (i.e., requirements). This linking establishes tracability from hazards/non-conformances to the underlying requirement. These linkings are automatically marked by the github system in the requirement issue. Additionally, to close an enhancement issue (i.e., requirement), passing verification tests must be created and checked in. The PR where these tests are created and the implementation is completed is linked to the issue establishing tracability from requirement -> verification test. These tests run automatically at every change/PR.
Additionally, requirements are assigned to milestones/releases, establishing bi-directional tracability to these
Summary: The following tracabilities are maintained: * Hazard <-> Requirement * Non-conformance <-> Requirement * Requirement <-> Verification Test & Results * Requirement <-> Implementation * Release/Milestone <-> Requirement