Usage

This document shows the usage of the different NIPS methods within a custom processing script. It describes the applications, and how the functions can be used when working with the pipeline. We stress that this is work in progress and therefore some functionalities that were available in NIPS1.x might not be implemented yet.

For the application of standard extraction schemes, see the single examples for FS, MOS or IFS. They utilize the automatic methods m_process_exposure(). The design of these methods follows the guideline described here, but includes some workflow optimization and error detection. If no user-defined processing is necessary, it is recommended to use those methods instead of a custom assembled script.

Initialization

At the very beginning, it is necessary initialize the PipelineParameters class with the mandatory parameters:

• output_path

• archive_path

• ref_path

• nid

• env

• mode

while some others are optional, like

• slitid

  • FS: Runs pipeline for all slits

  • MOS: msaconfig parameter has to be set

  • IFS: Sets the target slice id to 15, but runs pipeline for all 30 slices

• category - Default: ‘INTERNAL’

• steps - Default: Run all steps needed for this mode

• pardict (custom parameters for each step)

• writeout (products writeout scheme) - Default: AWCS, RADM, RECT, EXTR, CUBE

• display (save the products as pdf display) - Default: False

• target_id (custom target ID name) - Default: Generated automatically

• …

The instrument model is created from the information in the corresponding input data and the model file in the reference repository. It is used to assign awcs to pixels.

The beginning of an extraction script usually looks like this:

from nips.pipelines.pipeline import PipelineParameters, Pipeline

# Define basic parameters
#########################
output_path = './'
archive_path = '/Volumes/CT1/Archive'
ref_path = '/Volumes/CT1/JWST-NIPS-REF_REPOS/nipsref'
nid = 3162
env = 'OTIS'

qflags

The qflags input parameter allows the user to specify quality flags that should not be used by pipeline steps such as RECT, EXTR, EXTI and CUBE as well as several display methods.. The table below lists all available flags and their bit setting. The user can provide either a list of flags, e.g. qflags = [4, 8, 32] or a cumulative number, e.g. qflags = 44.

Bit	Flag	Data Quality Condition
1	0	Open pixel
2	2	Adjacent open pixel
3	4	Hot pixel
4	8	Warm pixel
5	16	Shorted pixel
6	32	Telegraph pixel
7	64	Low QE pixel
8	128	Bad reference pixel
9	256	Unreliable linearity correction
10	512	Unreliable flat-field correction
11	1024	Pixel has high (>12 e-/s) total noise
12	2048	Unreliable bias subtraction
13	4096	Pixel affected by ghost/crosstalk
14	8192	Pixel affected by cosmic ray hit
15	16384	Saturation occured before 2nd sample (group)
16	32768	No linearity correction/sat. check applied
17	65536	Not assigned.
18	131072	Not assigned.
19	262144	Not assigned.
20	524288	PXBG - no correction
21	1048576	PXBG - low-quality correction
22	2097152	AUTC - no correction
23	4194304	AUTC - low-quality correction
24	8388608	LEAK - no correction
25	16777216	LEAK - low-quality correction
26	33554432	MSFL - contamination by a failed-open shutter
27	67108864	MFSL - contamination by a zero order
28	134217728	MSFL - contamination by a bright spoiler
29	268435456	AWCS - no assigned wavelength
30	536870912	AWCS - wavelength outside extraction range
31	1073741824	AWCS - relative y coordinate outside extraction range
32	2147483648	PAMP - no correction
33	4294967296	PAMP - low-quality correction
34	8589934592	WLZP - no correction
35	17179869184	WLZP - low-quality correction
36	34359738368	DFLT - no correction
37	68719476736	DFLT - low-quality correction
38	137438953472	SFLT - no correction
39	274877906944	SFLT - low-quality correction
40	549755813888	FFLT - no correction
41	1099511627776	FFLT - low-quality correction
42	2199023255552	PTHL - no correction
43	4398046511104	PTHL - low-quality correction
44	8796093022208	BARS - no correction
45	17592186044416	BARS - low-quality correction
46	35184372088832	RADM - no correction
47	70368744177664	RADM - low-quality correction
48	140737488355328	MBGS - no correction
49	281474976710656	MBGS - low-quality correction
50	562949953421312	RECT - no coverage (pixel filling factor of zero)
51	1125899906842624	RECT - low coverage (pixel filling factor <0.5)
52	2251799813685248	RECT - partial coverage (pixel filling factor <0.99)
53	4503599627370496	Not assigned.
54	9007199254740992	Not assigned.
55	18014398509481984	Not assigned.
56	36028797018963968	Not assigned.
57	72057594037927936	Not assigned.
58	144115188075855872	Not assigned.
59	288230376151711744	Not assigned.
60	576460752303423488	Not assigned.
61	1152921504606846976	Not assigned.
62	2305843009213693952	Not assigned.
63	4611686018427387904	Not assigned.
64	9223372036854775808	Do not use

Setting the parameters

FS operations

Furthermore, mode has to be set:

# Mode
mode = 'FS'

Then the pipeline can be initialized and run without further parameters (only if it is an internal exposure):

# Construct pipeline parameters object
######################################
pl_parameters = PipelineParameters()
pl_parameters.m_set(output_path, archive_path, ref_path, mode)

In this setup the pipeline will assume we run an external lamp exposure for all fixed slits. It will create all five fixed slit targets and process them individually. If the pipelien should be run for just a specific fixed slit the slitid keyword has to be parsed to the pipeline parameters

# Slitid
slitid = 'S200A1'
# Construct pipeline parameters object
######################################
pl_parameters = PipelineParameters()
pl_parameters.m_set(output_path, archive_path, ref_path, mode, slitid=slitid)

By default for an internal target, the pipeline will create a target ID from the NID_ENV keywords, the mode and the slitid. The target ID and therefore the output folder name and all products inside this folder would look like those examples:

3162_OTIS_MOS_3_349_021_1X1_INTERNAL
3162_OTIS_MOS_3_349_021_1X3_INTERNAL
3162_OTIS_FS_S200A1_INTERNAL

To customize this target ID the target_id keyword can be specified. This keyword can be a simple string in case there is only one aperture extracted:

# Slitid
slitid = 'S200A1'
# TargetID
target_id = 'MyTarget'
# Construct pipeline parameters object
######################################
pl_parameters = PipelineParameters()
pl_parameters.m_set(output_path, archive_path, ref_path, mode, slitid=slitid, target_id=target_id)

The resulting target would then look like this:

3162_OTIS_MyTarget

Note

The customized target ID will always be appended to the NIS and ENV parameters.

If there is more than one slitid given as an input or if one uses the MSAfile keyword instead, the target ID has to be provided in form of a dictionary with the slitids as keys.:

# Slitid
slitid = ['S200A1', 'S200A2']
# TargetID
target_id = {'S200A1': 'MyTarget1', 'S200A2': 'MyTarget1'}
# Construct pipeline parameters object
######################################
pl_parameters = PipelineParameters()
pl_parameters.m_set(output_path, archive_path, ref_path, mode, slitid=slitid, target_id=target_id)

To customize the steps that should be applied to the data, two parameters can be set: The list of the steps that should be applied:

# Steps
steps = ['awcs','sflt','rect']
# Construct pipeline parameters object
######################################
pl_parameters = PipelineParameters()
pl_parameters.m_set(output_path, archive_path, ref_path, mode, slitid=slitid, steps=steps)

or a list of steps that should be excluded from the default step list:

# Excluded steps
exclude = ['dflt']
# Construct pipeline parameters object
######################################
pl_parameters = PipelineParameters()
pl_parameters.m_set(output_path, archive_path, ref_path, mode, slitid=slitid, exclude=exclude)

Another usefull input parameter is the pardict. This is a dictionary with parameters for each individual step. To find out more about which parameters can be set for each step, look at the parameter classes in each step module (for example RectParameters. Here is an example of how to set up the pardict:

# Pardict
pardict = {'rect' : {'ny' : 33}}
# Construct pipeline parameters object
######################################
pl_parameters = PipelineParameters()
pl_parameters.m_set(output_path, archive_path, ref_path, mode,
                    slitid=slitid, steps=steps, pardict=pardict)

Note

Currently those parameters will be applied to all targets run in the pipeline (i.e. if all slits are extracted, they will all be extracted with those parameters). This might change in the future.

Last, if it is desired to run the exact pipeline run again, just using the config files saved in the output folders one can achieve this by setting the set_from_file option to true

# Construct pipeline parameters object
######################################
pl_parameters = PipelineParameters()
pl_parameters.m_set(output_path, archive_path, ref_path, mode,
                    set_from_file=True)

Warning

This is still under development and might not work perfectly yet.

Back to top

MOS operations

The setup for MOS is very similar to the FS mode and all options introduced in the previous sections are aplicable to the mos except the slitid. When slitid is not set in a mos pipeline run, the pipeline need the input of a msa configuration file. This file can be either a fits file (.fits) or uncompressed binary file (.usa).

Note

At the moment a compressed binary file with the extentsion .msa is not supported yet.

To set up a MOS pipeline run using the msa config file one has to set the slitid to None or not set it at all:

# Mode
mode = 'MOS'
msaconfig = './msaconfig/SPCB-GD-C.msa.fits'
# Construct pipeline parameters object
######################################
pl_parameters = PipelineParameters()
pl_parameters.m_set(output_path, archive_path, ref_path, mode,
                    msaconfig=msaconfig)

Back to top

IFU operations

In case of an IFS obervstion, the slitid does not represent the slice that will be process but rather the slice in which the target falls. If this is not give it will be set to 14 (middle of the aperture). The pipeline always extracts and processes all 30 slices and if the cube step is given in the step list (by default it is), it created the final data cube. Besides this, the input parameters are the same as for FS and MOS mode.

Exposure processing

Once the parameters are set, running the pipeline consists only of two lines and is the same for each mode.

# Initialize and run the pipeline
#################################
pl = Pipeline()
pl.m_set(pl_parameters)
pl.m_process_exposure()

At the moment, there are no methods present to run a single pipeline step individually outside of the pipeline but it can be achieved by setting the steps keyword to just a single step.

Back to top