GitLocker: The Coding Marketplace

Description:

jwst 1.15.1

JWST Calibration Pipeline

[!IMPORTANT]
JWST requires a C compiler for dependencies and is currently limited to Python 3.10, 3.11, or 3.12.

[!NOTE]
Linux and MacOS platforms are tested and supported. Windows is not currently supported.

[!WARNING]
Installation on MacOS Mojave 10.14 will fail due to lack of a stable build for dependency opencv-python.

Installation
Please contact the JWST Help Desk for installation issues.
The easiest way to install the latest jwst release into a fresh virtualenv or conda environment is
pip install jwst

Detailed Installation
The jwst package can be installed into a virtualenv or conda environment via pip.
We recommend that for each installation you start by creating a fresh
environment that only has Python installed and then install the jwst package and
its dependencies into that bare environment.
If using conda environments, first make sure you have a recent version of Anaconda
or Miniconda installed.
If desired, you can create multiple environments to allow for switching between different
versions of the jwst package (e.g. a released version versus the current development version).
In all cases, the installation is generally a 3-step process:

Create a conda environment
Activate that environment
Install the desired version of the jwst package into that environment

Details are given below on how to do this for different types of installations,
including tagged releases, DMS builds used in operations, and development versions.
Remember that all conda operations must be done from within a bash/zsh shell.
Installing latest releases
You can install the latest released version via pip. From a bash/zsh shell:
conda create -n <env_name> python=3.11
conda activate <env_name>
pip install jwst

You can also install a specific version:
conda create -n <env_name> python=3.11
conda activate <env_name>
pip install jwst==1.9.4

Installing the development version from Github
You can install the latest development version (not as well tested) from the
Github master branch:
conda create -n <env_name> python=3.11
conda activate <env_name>
pip install git+https://github.com/spacetelescope/jwst

Installing a DMS Operational Build
There may be occasions where an exact copy of an operational DMS build is
desired (e.g. for validation testing or debugging operational issues).
We package releases for DMS builds via environment snapshots that specify the
exact versions of all packages to be installed.
To install a particular DMS build, consult the
Software vs DMS build version map
table shown below to determine the correct jwst tag. For example, to install the
version of jwst used in DMS build 9.0, use jwst tag 1.8.2. The overall
procedure is similar to the 3-step process outlined in the previous section, but the
details of each command vary, due to the use of environment snapshot files that specify
all of the particular packages to install. Also note that different snapshot files are
used for Linux and Mac OS systems.
Linux:
conda create -n jwstdp-1.12.5 --file https://ssb.stsci.edu/releases/jwstdp/1.12.5/conda_python_stable-deps.txt
conda activate jwstdp-1.12.5
pip install -r https://ssb.stsci.edu/releases/jwstdp/1.12.5/reqs_stable-deps.txt

MacOS:
conda create -n jwstdp-1.12.5 --file https://ssb.stsci.edu/releases/jwstdp/1.12.5/conda_python_macos-stable-deps.txt
conda activate jwstdp-1.12.5
pip install -r https://ssb.stsci.edu/releases/jwstdp/1.12.5/reqs_macos-stable-deps.txt

Each DMS delivery has its own installation instructions, which may be found in
the corresponding release documentation linked from this page:
https://github.com/astroconda/astroconda-releases/tree/master/jwstdp
The installation procedures may change from time to time, so consulting the
documentation page for the specific version in question is the best way to get
that version installed.
Installing for Developers
If you want to be able to work on and test the source code with the jwst package,
the high-level procedure to do this is to first create a conda environment using
the same procedures outlined above, but then install your personal copy of the
code overtop of the original code in that environment. Again, this should be done
in a separate conda environment from any existing environments that you may have
already installed with released versions of the jwst package.
As usual, the first two steps are to create and activate an environment:
conda create -n <env_name> python=3.11
conda activate <env_name>

To install your own copy of the code into that environment, you first need to
fork and clone the jwst repo:
cd <where you want to put the repo>
git clone https://github.com/<your_github_username>/jwst.git
cd jwst

Note: python setup.py install and python setup.py develop commands do not work.
Install from your local checked-out copy as an "editable" install:
pip install -e .

If you want to run the unit or regression tests and/or build the docs, you can make
sure those dependencies are installed too:
pip install -e ".[test]"
pip install -e ".[docs]"
pip install -e ".[test,docs]"

Need other useful packages in your development environment?
pip install ipython jupyter matplotlib pylint

Calibration References Data System (CRDS) Setup
Note: As of November 10, 2022, the process of deprecating the CRDS PUB Server will start.
For details, refer to the CRDS PUB Server Freeze
and Deprecation page
CRDS is the system that manages the reference files needed to run the pipeline.
For details about CRDS, see the User's
Guide
The JWST CRDS server is available at https://jwst-crds.stsci.edu
It supports the automatic processing pipeline at STScI.
Inside the STScI network, the same server is used by the pipeline by default with no modifications.
To run the pipeline outside the STScI network, CRDS must be configured by setting
two environment variables:
export CRDS_PATH=<locally-accessable-path>/crds_cache/jwst_ops
export CRDS_SERVER_URL=https://jwst-crds.stsci.edu

<locally-accessable-path> can be any the user has permissions to use, such as $HOME.
Expect to use upwards of 200GB of disk space to cache the latest couple of contexts.
To use a specific CRDS context, other than the current default, set the CRDS_CONTEXT
environment variable:
export CRDS_CONTEXT=jwst_1179.pmap

Documentation
Documentation (built daily from the Github master branch) is available at:
https://jwst-pipeline.readthedocs.io/en/latest/
To build the docs yourself, clone this repository and build the documentation with:
pip install -e ".[docs]"
cd docs
make html
make latexpdf

Contributions and Feedback
We welcome contributions and feedback on the project. Please follow the
contributing guidelines to submit an issue or a pull request.
We strive to provide a welcoming community to all of our users by abiding with
the Code of Conduct.
If you have questions or concerns regarding the software, please open an issue
at https://github.com/spacetelescope/jwst/issues or
contact the JWST Help Desk.
Software vs DMS build version map
The table below provides information on each release of the jwst package
and its relationship to software builds used in the STScI JWST DMS operations
environment. The Released column gives the date on which the jwst tag
was released on PyPi and the Ops Install column gives the date on which
the build incorporating that release was installed in DMS operations.
Note that the CRDS_CONTEXT listed is a minimum context that can be used with
that release. A release should work with any contexts between
the specified context and less than the context for the next release.

jwst tag
DMS build
SDP_VER
CRDS_CONTEXT
Released
Ops Install
Notes

1.15.1
B11.0rc2
TBD
1242
2024-07-08
TBD
Second release candidate for B11.0

1.15.0
B11.0rc1
TBD
1241
2024-06-26
TBD
First release candidate for B11.0

1.14.1

1238
2024-06-27

PyPI-only release for external users

1.14.0
B10.2.1
2024.1.1
1238
2024-03-29
2024-06-12
Final release candidate for B10.2.1

1.13.4

1185
2024-01-25

PyPI-only release for external users

1.13.3
B10.1
2023.4.0
1181
2024-01-05

Final release candidate for B10.1

1.13.2
B10.1rc3
2023.4.0
1181
2023-12-21

Third release candidate for B10.1

1.13.1
B10.1rc2
2023.4.0
1181
2023-12-19

Second release candidate for B10.1

1.13.0
B10.1rc1
2023.4.0
1179
2023-12-15

First release candidate for B10.1

1.12.5
B10.0.1
2023.3.1
1166
2023-10-19
2023-12-05
Patch release B10.0.1

1.12.4

2023-10-12

Pinning dependencies for external users

1.12.3
B10.0
2023.3.0
1135
2023-10-03
2023-12-05
Final release candidate for B10.0

1.12.2
B10.0rc3

1135
2023-10-02

Third release candidate for B10.0

1.12.1
B10.0rc2

1132
2023-09-26

Second release candidate for B10.0

1.12.0
B10.0rc1

1130
2023-09-18

First release candidate for B10.0

1.11.4
B9.3.1
2023.2.1
1107
2023-08-14
2023-08-24
Final release for B9.3.1 patch

1.11.3
B9.3
2023.2.0
1097
2023-07-17

Final release candidate for B9.3

1.11.2
B9.3rc3

1097
2023-07-12

Third release candidate for B9.3

1.11.1
B9.3rc2

1094
2023-06-29

Second release candidate for B9.3

1.11.0
B9.3rc1

1094
2023-06-21

First release candidate for B9.3

1.10.2

1077
2023-04-14

Pinning dependencies for external users

1.10.1
B9.2.x
2023.1.1
1077
2023-04-13
2023-05-23
Final release candidate for B9.2

1.10.0
B9.2rc1

1075
2023-03-31

First release candidate for B9.2

1.9.6
B9.1.2
2022.5.2
1068
2023-03-09
2023-03-15
Final release candidate for B9.1.2

1.9.5

1061
2023-03-02

First release candidate for B9.1.2

1.9.4
B9.1.1
2022.5.1
1041
2023-01-27
2023-02-28
Final release candidate for B9.1.1

1.9.3
B9.1
2022.5.0
1030
2023-01-12
2023-02-28
Final release candidate for B9.1

1.9.2
B9.1rc2

2023-01-04

Second release candidate for B9.1 (hotfix)

1.9.1
B9.1rc2

2023-01-03

Second release candidate for B9.1

1.9.0
B9.1rc1

2022-12-27

First release candidate for B9.1

1.8.5
B9.0

1019
2022-12-12

Documentation patch release for B9.0

1.8.4
B9.0

2022-11-16

Documentation patch release for B9.0

1.8.3
B9.0

2022-11-11

Documentation patch release for B9.0

1.8.2
B9.0
2022.4.0
1017
2022-10-19
2022-11-17
Final release candidate for B9.0

1.8.1
B9.0rc2

2022-10-17

Second release candidate for B9.0

1.8.0
B9.0rc1

2022-10-10

First release candidate for B9.0

1.7.2
B8.1.2
2022.3.1
0984
2022-09-12
2022-09-21
Final release candidate for B8.1.2

1.7.1
B8.1.2rc2

2022-09-07

Second release candidate for B8.1.2

1.7.0
B8.1.2rc1

2022-09-01

First release candidate for B8.1.2

1.6.2
B8.1
2022.3.0
0953
2022-07-19
2022-08-19
Final release candidate for B8.1

1.6.1
B8.1rc2

2022-07-15

Second release candidate for B8.1

1.6.0
B8.1rc1

2022-07-11

First release candidate for B8.1

1.5.3
B8.0.1
2022.2.1
0913
2022-06-20
2022-06-30
Patch release B8.0.1

1.5.2
B8.0
2022.2.0
0874
2022-05-20
2022-06-16
Final release candidate for B8.0

1.5.1
B8.0rc2

2022-05-17

Second release candidate for B8.0

1.5.0
B8.0rc1

2022-05-05

First release candidate for B8.0

1.4.6
B7.9.3
2022.1.2
0800
2022-03-25

Final release candidate for B7.9.3

1.4.5
B7.9.3rc2

2022-03-23

Second release candidate for B7.9.3

1.4.4
B7.9.3rc1

2022-03-16

First release candidate for B7.9.3

1.4.3
B7.9.1
2022.1.1
0800
2022-02-03

Final B7.9.1

1.4.2
B7.9
2022.1.0
0797
2022-01-20

Final release candidate for B7.9

1.4.1
B7.9rc2

2022-01-15

Second release candidate for B7.9

1.4.0
B7.9rc1

2022-01-10

First release candidate for B7.9

Pre-launch releases

1.3.3
B7.8.2
2021.4.0
0764
2021-10-05

Same as 1.3.2, but with installation bug fix

1.3.2
B7.8.2
2021.4.0
0764
2021-09-03

Final release candidate for B7.8.2

1.3.1
B7.8.1
2021.3.0
0742
2021-08-09

Final release candidate for B7.8.1

1.3.0
B7.8.1rc1

0741
2021-08-02

First release candidate for B7.8.1

1.2.3
B7.8
2021.2.0
0732
2021-06-08

Final release candidate for B7.8

1.2.2
B7.8rc3

2021-06-08

Third release candidate for B7.8

1.2.1
B7.8rc2

2021-06-07

Second release candidate for B7.8

1.2.0
B7.8rc1

0723
2021-05-24

First release candidate for B7.8

1.1.0
B7.7.1
2021.1.0
0682
2021-02-26

Final release candidate for B7.7.1

1.0.0
B7.7.1rc1

0678
2021-02-22

First release candidate for B7.7.1

0.18.3
B7.7
2020.4.0
0670
2021-01-25

Final release candidate for B7.7

0.18.2
B7.7rc3

0668
2021-01-19

Third release candidate for B7.7

0.18.1
B7.7rc2

0664
2021-01-08

Second release candidate for B7.7

0.18.0
B7.7rc1

0645
2020-12-21

First release candidate for B7.7

0.17.1
B7.6
2020.3.0
0641
2020-09-15

Final release candidate for B7.6

0.17.0
B7.6rc1

0637
2020-08-28

First release candidate for B7.6

0.16.2
B7.5
2020.2.0
0619
2020-06-10

Same as 0.16.1, but with installation bug fix

0.16.1
B7.5
2020.2.0
0619
2020-05-19

Final release candidate for B7.5

0.16.0
B7.5rc1

0614
2020-05-04

First release candidate for B7.5

0.15.1
B7.4.2
2020.1.0
0586
2020-03-10

Final release candidate for B7.4.2

0.15.0
B7.4.2rc1

0585
2020-02-28

First release candidate for B7.4.2

0.14.2
B7.4
2019.3.0
0570
2019-11-18

Final release candidate for B7.4

0.14.1
B7.4rc2

0568
2019-11-11

Second release candidate for B7.4

0.14.0
B7.4rc1

0563
2019-10-25

First release candidate for B7.4

0.13.8
B7.3.1
2019.2.0
0541
2019-09-05

Patch for Build 7.3 released as Build 7.3.1

0.13.7
B7.3
2019.1.0
0535
2019-06-21

Final release candidate for Build 7.3

0.13.6
B7.3rc4

0534
2019-06-20

Fourth release candidate for Build 7.3

0.13.5
B7.3rc3

0534
2019-06-19

Third release candidate for Build 7.3

0.13.4
B7.3rc2

0534
2019-06-18

Second release candidate for Build 7.3

0.13.3
B7.3rc1

0532
2019-06-04

First release candidate for Build 7.3

0.13.2

0500
2019-05-14

DMS test, no delivery to I&T

0.13.1

0500
2019-03-08

DMS test, no delivery to I&T

0.13.0

0500
2019-02-15

DMS test, no delivery to I&T

0.12.3
B7.2.1

0500
2019-01-15

DMS Build 7.2.1 patch release

0.12.2
B7.2
2018_2
0495
2018-11-07

Final release candidate for Build 7.2

0.12.1
B7.2rc2

0495
2018-11-01

Second release candidate for Build 7.2

0.12.0
B7.2rc1

0493
2018-10-09

First release candidate for Build 7.2

0.11.0

0482
2018-09-10

DMS test, no delivery to I&T

0.10.0

0477
2018-07-31

DMS test, no delivery to I&T

0.9.6
B7.1.3
2018_1
0468
2018-06-08

Final release candidate for Build 7.1.3

0.9.5
B7.1.3rc3

0468
2018-06-06

Third release candidate for Build 7.1.3

0.9.4
B7.1.3rc2

0463
2018-05-29

Second release candidate for Build 7.1.3

0.9.3
B7.1.3rc1

0457
2018-05-11

First release candidate for Build 7.1.3

0.9.2

0441
2018-03-28

DMS test, no delivery to I&T

0.9.1

0432
2018-02-16

DMS test, no delivery to I&T

0.9.0
B7.1.2

0422
2017-12-22

DMS patch release to I&T 2018-02-15

0.8.0
B7.1.1

0422
2017-11-06

DMS patch release to I&T 2018-01-17

0.8.0
B7.1
2017_1
0422
2017-11-06

Final release for Build 7.1

0.7.7
B7.0
2016_2
0303
2016-12-13

Final release for Build 7.0

Unit Tests
Unit tests can be run via pytest. Within the top level of your local jwst repo checkout:
pip install -e ".[test]"
pytest

Need to parallelize your test runs over all available cores?
pip install pytest-xdist
pytest -n auto

Regression Tests
Latest regression test results can be found here (STScI staff only):
https://plwishmaster.stsci.edu:8081/job/RT/job/JWST/
The test builds start at 6pm local Baltimore time Monday through Saturday on jwcalibdev.
To run the regression tests on your local machine, get the test dependencies
and set the environment variable TEST_BIGDATA to our Artifactory server
(STSci staff members only):
pip install -e ".[test]"
export TEST_BIGDATA=https://bytesalad.stsci.edu/artifactory

To run all the regression tests (except the very slow ones):
pytest --bigdata jwst/regtest

You can control where the test results are written with the
--basetemp=<PATH> arg to pytest. NOTE that pytest will wipe this directory clean
for each test session, so make sure it is a scratch area.
If you would like to run a specific test, find its name or ID and use the -k option:
pytest --bigdata jwst/regtest -k nirspec

If developers need to update the truth files in our nightly regression tests,
there are instructions in the repository wiki.
https://github.com/spacetelescope/jwst/wiki/Maintaining-Regression-Tests