PipelineParameters

class nips.pipelines.PipelineParameters(logger_id=None)

Bases: nrspylib.utils.logging_template.LoggingTemplate

Class for the computation parameters the Pipeline

This class is used for the handling of the parameters of the NIPS master- pipeline. This class is specifically intended for computation of default parameters and the I/O of those into json config files. In addition it acts as a collector class for the parameters of individual pipeline steps.

logger

Logger instance. (Inherited)

Type

logging.Logger or None

_sid

List of aperture names for internal use. This is identical with slitid for MOS and FS mode. In IFS mode slitid represents the slice number of the target while _sid is a list of all 30 slices (as strings).

Type

list

archive_path

Path to data archive.

Type

str

category

Target category.

Type

str

datatype

Data type identification string. It will be set to ‘PIPELINEPARAM’ by the constructor.

Type

str

display

Flag whether to isplay output in a pdf file.

Type

bool

env

Test environment.

Type

str

expid

Exposure id. Unique identifier for each exposure. Combination of env and nid. expid = nid_env

Type

str

expid_autc

Exposure ids for autocal files. Unique identifier for each exposure. Combination of env and nid. expid = nid_env

Type

list

expid_pxbg

Exposure ids for background files. Unique identifier for each exposure. Combination of env and nid. expid = nid_env

Type

list

extmode

Extraction Mode.

Type

str

output_path

Path to master folder of pipeline.

Type

str

mode

Mode of observations.

Type

str

msafile

Name of MSA confidguration file.

Type

str

model_path

Path to model.

Type

str

model_name

Name of model.

Type

str

nid

Number id of exposure (only unique for one environment)

Type

str

output_path

Path to master folder of pipeline.

Type

str

qflags

Flags for all pipeline operations

Type

int

refdict

Dictionary to define costume reference files.

Type

dict

reffiles

Dictionary of reference file names with slitid (or sliceid for IFS) as keys.

Type

dict

ref_path

Path to reference files repository.

Type

str

slitid

List of slit ids.

Type

list

slitlet

Name of MOS slitlet.

Type

str

step_param

Dictionary of step parameter instances with slitid (or sliceid for IFS) as keys.

Type

dict

steps

List of pipeline step names that will be applied.

Type

list

target_file

Name of the target configuration file.

Type

str

writeout

Writeout scheme for the pipeline run.

Type

str

msetup

Instance of the master setup class. To set folders and file structure on master pipeline level.

Type

MasterSetup

esetup

Instance of the exposure setup class. To set folders and file structure on exposure level.

Type

ExposureSetup

tsetup

Instance of the target setup class. To set folders and file structure on target level.

Type

TargetSetup

target

Instance of the Target class. Defines Target of this pipeline run.

Type

Target

tid

Target ID.

Type

int

Methods Summary

m_check_parameters()
m_is_set([excluded, category, verbose])	Method to check if the class instance is initialized.
m_read_from_dic(dict)	Method to read the contents of the instance from a simple dictionary.
m_read_from_json()	Method to read the contents of the instance from a json file.
m_read_from_json_dic(json_dic, json_dic_steps)	Method to read the contents of the instance from a json dictionary.
m_set(output_path, archive_path, ref_path, …)	Method to set the pipeline parameters from input.
m_write_to_json()	Method to write the contents of the instance to a json file.
m_write_to_json_dic(si)	Method to write the contents of the instance to a dictionary.

Methods Documentation

m_check_parameters()

m_is_set(excluded=(), category=None, verbose=False)

Method to check if the class instance is initialized.

This method can be used to check if the instance of the class is initialized. Arguments are available to conduct checks limited to a given categoy of attributes. It is also possible to exclude some of the attributes from the check.

Parameters

• excluded (list of strings) – list that can be used to exclude some of the attributes from the check. Defaulted to ().

• category (str) – allows to limit the check to only a certain category of attributes None = all (default), “ext” = extraction step attribute, “awcs” = WCS assignement attributes.

• verbose (boolean) – if set to True, a message will be printed when an attribute is still set to None. If set to False (default) the method will proceed silently and just return False.

Returns

• True if all attributes are initialized (i.e. are not None), False

• otherwise.

Notes

Constant attributes defined by the constructor are excluded from the check. Any change of the list of attributes should be reflected accordingly in this method.

m_read_from_dic(dict)

Method to read the contents of the instance from a simple dictionary.

This method reads the contents of the instance from a dictionary. The dictionary does not have to contain all the keywords, it can be a sub- sample.

Parameters

dict (dictionary) – input dictionary

Returns

Return type

None.

m_read_from_json()

Method to read the contents of the instance from a json file. This method reads the contents of the instance from a json file on disk. It relies on the m_read_from_json_dic() method.

Parameters

inputfile (str) – name of the input json file

Returns

Return type

None.

m_read_from_json_dic(json_dic, json_dic_steps)

Method to read the contents of the instance from a json dictionary. This method reads the contents of the instance from a json dictionary. It relies on the method m_read_from_json() from a file.

Parameters

json_dic (dictionary) – input json dictionary

Returns

Return type

None.

m_set(output_path, archive_path, ref_path, nid, env, mode, slitid=None, category='INTERNAL', msafile=None, steps=None, exclude=None, nid_autc=None, nid_pxbg=None, slitlet='1X1', target_file=None, target_id=None, target=None, pardict=None, refdict=None, qflags=None, model_path=None, model_name=None, extmode='standard', writeout='Default', display=True, set_from_file=False, logging_level=20, stdout_logging_level=20, master_name='master', zero_gwa_xtil=False, zero_gwa_ytil=False, write_target_log=True)

Method to set the pipeline parameters from input.

If configpath is set, then all of the other inputparamters will be ignored and instance is set with paramters fromt he config files.

Parameters

• output_path (str) – Path to master folder of pipeline.

• archive_path (str) – Path to data archive.

• ref_path (str) – Path to reference files repository.

• nid (str) – Number id of exposure (only unique for one environment)

• env (str) – Test environment. Options: [‘IABG’, ‘ISIM’, ‘OTIS’]

• mode (str) – Mode of observations. Options: [‘FS’, ‘MOS’, ‘IFS’]

• slitid (list) – List of slit ids. Is ignored when msafile is set.

• category (str) – Target category. Options: ‘POINT’: ‘point source category’, ‘EXTENDED’: ‘extended source catagory’, ‘UNKNOWN’: ‘unknown source category’, ‘BACKGROUND’: ‘background souce category’, ‘INTERNAL’: ‘internal calibration source category’ (Default: ‘INTERNAL’)

• msafile (str) – Name of MSA configuration file. Once this keyword is set, the slitid keyword is ignored.

• steps (list) –

  List of pipeline step names that will be applied. Options: [‘pamp’, ‘wlzp’, ‘dflt’, ‘sflt’, ‘fflt’, ‘pthl’, ‘bars’, ‘radm’, ‘rect’, ‘extr’, ‘exti’, ‘cube’] If not set, the default list of steps for each mode is used:

  ’FS’ :

  [‘awcs’, ‘pamp’, ‘wlzp’, ‘dflt’, ‘sflt’, ‘fflt’, ‘pthl’, ‘radm’, ‘rect’, ‘extr’]

  ’MOS’:

  [‘awcs’, ‘pamp’, ‘wlzp’, ‘dflt’, ‘sflt’, ‘fflt’, ‘pthl’, ‘bars’, ‘radm’,’rect’, ‘extr’]

  ’IFS’:

  [‘awcs’, ‘pamp’, ‘dflt’, ‘sflt’, ‘fflt’, ‘pthl’, ‘radm’, ‘rect’, ‘cube’]

• exclude (list) – List of pipeline steps to be excluded. Options above.

• nid_autc (str/list) – Number id of exposure for the autocal flat exposure(s) (only unique for one environment)

• nid_pxbg (str/list) – Number id of exposure for the background exposure(s) (only unique for one environment)

• slitlet (str) – Name of MOS slitlet. Options: [‘1X1’, ‘1X3’, ‘1X5’, ‘LONG’]

• target_file (str) – Name of the target configuration file. If not given the pipeline will create a standard internal target.

• target_id (str/dict) – Alternative name for target. If set to None, it will be automatically generated from the configuration parameters. If there is more than one slitid, this keyword has to be a dictionary with the slitids as keys! (currently igonred for external targets)

• target (Target instance/dict) – Target or dictionary of targets. If set to None, it will be automatically generated from the configuration parameters. If there is more than one slitid, this keyword has to be a dictionary with the slitids as keys!

• pardict (dict) –

  Dictionary to set step parameters from input. It has the form of:

  {'awcs': {'parameter1' : value1, 'parameter2' : value2},
   'dflt': {'parameter1' : value1}}
  

  for all targets, or:

  {'awcs': {'S200A1': {'parameter1' : value1, 'parameter2' : value2},
            'S200B1': {'parameter1' : value1, 'parameter2' : value2}}...
   'dflt': {'parameter1' : value1}}
  

  for setting a different parameters for each target. It can also be a mixed form of both.

• refdict (dict) –

  Dictionary to define costume reference files. It has the form of:

  {'sflt': '/path/to/reference/file/sflt',
   'dflt': '/path/to/reference/file/dflt'}
  

  if setting one reference file for all targets, or:

  {'sflt': {'S200A1':'/path/to/reference/file/sflt',
            'S200A2':'/path/to/reference/file/sflt'},
   'fflt': {'S200A1':'/path/to/reference/file/fflt',
            'S200A2':'/path/to/reference/file/fflt'}
  

  for setting a different reference file for each target. It can also be a mixed form of both.

• qflags (list or int) –

  The qflags input parameter allows the user to specify quality flags that should not be used by pipeline steps such as RECT, EXTR, EXTI and CUBE as well as several display methods.. The table below lists all available flags and their bit setting. The user can provide either a list of flags, e.g. qflags = [4, 8, 32] or a cumulative number, e.g. qflags = 44.

  FLAG	Bit Setting	Data Quality Condition
  0	0000 0000 0000 0000	OK
  1	0000 0000 0000 0001	Open pixel
  2	0000 0000 0000 0010	Adjacent open pixel
  4	0000 0000 0000 0100	Hot pixel
  8	0000 0000 0000 1000	Warm pixel
  16	0000 0000 0001 0000	Shorted pixel
  32	0000 0000 0010 0000	Telegraph pixel
  64	0000 0000 0100 0000	Low QE pixel
  128	0000 0000 1000 0000	Bad reference pixel
  256	0000 0001 0000 0000	Unreliable linearity correction
  512	0000 0010 0000 0000	Unreliable flat field correction
  1024	0000 0100 0000 0000	Pixel has high (>12 e-/s) total noise
  2048	0000 1000 0000 0000	Unreliable bias subtraction
  4096	0001 0000 0000 0000	Pixel affected by ghost/crosstalk
  8192	0010 0000 0000 0000	Pixel affected by cosmic ray hit
  16384	0100 0000 0000 0000	Saturation occured before 2nd sample (group)
  32768	1000 0000 0000 0000	No linearity correction/sat. check applied

• model_path (str) – Path to user input model. If None, model will be picked through the reference file repository (Default: None)

• model_name (str) – Name of user input model. If None, model will be picked through the reference file repository (Default: None)

• extmode (str) – extraction mode (in _extmodes). ‘large’: an oversizing of 50% (100%) is applied on each side when extracting the spectra in IFS or FS (MOS) mode. ‘standard’: an oversizing of 20% (50%) is applied on each side when extracting the spectra in IFS or FS (MOS) mode. ‘tight’: an oversizing of 0% (0%) is applied on each side when extracting the spectra in IFS or FS (MOS) mode. ‘vintage’: an oversizing of 5% is applied on each side when extracting the spectra. This is similar to the default oversizing used in NIPS 1.X versions

• writeout (str) –

  Writeout scheme for the pipeline run. Options:

  ’Default’:

  Writes out AWCS product, the last product before rectificiation (usually RADM), RECT product, EXTR product, EXTI product, and CUBE product.

  ’All’ :

  Writes out every step product.

  ’Costum’ :

  Costume output scheme (not implemented yet)

  (Default: ‘Default’)

• display (bool) – Flag whether to isplay output in a pdf file. (Default: False)

• set_from_file (bool) – If this is set, then all of the other inputparamters will be ignored and instance is set with paramters from the config files. (Default: False)

• logging_level (int) – logging level [10 - ‘debug’, 20 - ‘info’, 30 - ‘warning’, 40 - ‘error’, 50 - ‘critical’] (Default: 20)

• stdout_logging_level (int) – logging level for the output in the terminal. Note this can never be lower than the logging_level.

• zero_gwa_xtil (bool) – If set to true, the model will be initiated without using the sensor reading for the gwa_xtil and the gwa calibration is not used. (Default: False)

• zero_gwa_ytil (bool) – If set to true, the model will be initiated without using the sensor reading for the gwa_ytil and the gwa calibration is not used. (Default: False)

• write_target_log (bool) – If set to False, the pipeline will not write out a log for each Target (it will still output log details in the terminal, tho). This flag should be set to False when extracting a very large number of targets (> 1000) which is usually the case for imaging mode. (Default: True)

m_write_to_json()

Method to write the contents of the instance to a json file. This method writes the contents of the instance to a json file on disk. It relies on the m_write_to_json_dic() to create the input dictionary.

Parameters

inputfile (str) – name of the output json file

Returns

Return type

None.

m_write_to_json_dic(si)

Method to write the contents of the instance to a dictionary. This method writes the contents of the instance to a dictionary suitable for creating a json file using the m_write_to_json() method.

Parameters

None –

Returns

Return type

Returns the json dictionary.