CALFUSE Pipeline Status

December 5, 2008
Van Dixon
v3.2/src/libcf/cf_calculate_y_centroid.c
v3.2/src/libcf/cf_find_spectra.c
v3.2/src/libcf/cf_screen_jitter.c

In HIST mode, there are no spectra in other apertures to cause confusion, so we need not require that the target centroid lie within X pixels of either the airglow centroid or the tabulated centroid. Both cf_find_spectra.c and cf_calculate_y_centroid.c were modified to effect this change.

Modified cf_screen_jitter.c so that the jitter correction is not applied to ERO observations of Jupiter (program X006).

v3.2/include/calfuse.h
v3.2/src/fuv/cf_bad_pixels.c
v3.2/src/fuv/cf_screen_photons.c
v3.2/src/libcf/cf_apply_filters.c
v3.2/src/libcf/cf_fuv_init.c
v3.2/src/libcf/cf_screen_jitter.c
v3.2/src/libcf/cf_set_photon_flags.c
v3.2/src/analysis/cf_combine.c
v3.2/src/analysis/cf_xcorr.c
v3.2/src/analysis/idf_combine.c
v3.2/src/analysis/cf_make_all_obs.csh
v3.2/src/analysis/cf_make_900_obs.csh

I've modified CalFUSE to treat bright-earth and airglow observations differently than before. Here are the details:

When creating the IDF files for bright-earth observations (programs M106 and S100) or 900-level airglow exposures, CalFUSE sets the header keyword EXP_STAT = 2. For 900-level exposures, the SRC_TYPE is set to EE, and the following warning is written to the file header: "Airglow exposure. Not an astrophysical target."

The pipeline then processes the file as usual. In particular, the limb-angle flag is set in the timeline table (extension 3 of the IDF), and the "Time with low limb angle" is written to the file header. However, if EXP_STAT = 2, the following things change:

The extracted spectra include photons obtained at low limb angles. The detector-image plots do, too. But the count-rate plots indicate times when the line of sight passes below the limb-angle limit.

For programs M106 and S100, all of the data are included in the observation-level ALL, ANO, and NVO files. For all other programs, 900-level airglow exposures are excluded from the sum. A new observation-level file is produced: a single quick-look (gif) plot showing the summed spectrum from all of the 900-level exposures in the observation. It has a name like M112580100900specttagf.gif.

April 11, 2008
Van Dixon
v3.2/src/analysis/cf_xcorr.c
v3.2/src/analysis/cf_combine.c
v3.2/src/analysis/cf_make_all_obs.csh

We've changed the logic used to combine individual exposures into observation-level files for data obtained in HIST mode. If the target is too faint to perform a cross-correlation, but is still in the slit (i.e., its flux is within 50% of the reference spectrum), add the spectra using a shift of zero. If an exposure suffers a limb-angle or SAA violation, include it anyway, assuming that HIST targets are bright enough that the additional background flux is not a problem.

In HIST mode, there are no spectra in other apertures to cause confusion, so we need not require that the target centroid lie within 30 pixels of the expected value.

Feb 22, 2008
Benjamin Ooghe
INSTALLING_CalFUSEv3.2.2
INSTALLING_A_FORTRAN_COMPILER
FIXING_SELINUX_ISSUES

Added scripts to set environment variables in Bourne-based shells (sh and bash) for systems without CSH and automated the setting of the shell path.

v3.2/src/configure.linux
v3.2/src/configure.linux64
v3.2/src/configure.macosx
v3.2/src/configure.solaris
v3.2/src/cfitsio/configure
v3.2/src/Makefile.Linux
v3.2/src/Makefile.Linux64
v3.2/src/Makefile.MacOSX
v3.2/src/analysis/Makefile.Linux
v3.2/src/analysis/Makefile.Linux64
v3.2/src/analysis/Makefile.MacOSX
v3.2/src/analysis/idf_combine.c
v3.2/src/analysis/cf_coadd.c
v3.2/src/analysis/cf_arith.c
v3.2/src/analysis/cf_combine.c
v3.2/src/analysis/cf_pack.c
v3.2/src/cal/jitter/Makefile.Linux
v3.2/src/cal/jitter/Makefile.Linux64
v3.2/src/cal/jitter/Makefile.MacOSX
v3.2/src/cal/jitter/cf_jitter.c
v3.2/src/cal/jitter/cf_jitter_diag.c
v3.2/src/fes/Makefile.Linux
v3.2/src/fes/Makefile.Linux64
v3.2/src/fuv/Makefile.Linux
v3.2/src/fuv/Makefile.Linux64
v3.2/src/fuv/Makefile.MacOSX
v3.2/src/fuv/cf_gainmap.c
v3.2/src/fuv/cf_countmap.c
v3.2/src/libcf/Makefile.Linux
v3.2/src/libcf/Makefile.Linux64
v3.2/src/libcf/Makefile.MacOSX
v3.2/src/libcf/cf_idf_io.c
v3.2/src/slalib/mk
v3.2/src/slalib/mk.sv
v3.2/src/slalib/sla_source.tar

Updated configure files and Makefiles for all systems in order to compile with more different fortran compilers (f77, f90, g77, gfortran) and on recent systems (Solaris 10, recent Linux, Mac OS X 10.4) including 64-bit architecture machines. This required the modification of source files to correct bad int/long declarations that cause problems on 64-bit architectures, on Solaris 10 systems, and on some fortran compilers.

v3.2/include/calfuse.h
v3.2/src/libcf/cf_fes_proc_update.c
v3.2/src/libcf/cf_fes_proc_check.c

Fixed memory problem causing occasional bad values in the NVO flux arrays. In copy_spec, the shift+start value was sometimes negative, causing the routine to read negative indexes in the flux and error arrays. Also, the final values of the flux_nvo and error_nvo arrays weren't always written.

In cf_generate_pseudo_photons, verify for each fill-data event that the LOCATION_SHLD flag is not set to ensure that it does not fall outside of the active area of the detector.

In cf_estimate_drift_coefficients, set max value of index to jmax-1 rather than jmax.

February 22, 2008
Van Dixon
v3.2/src/fuv/cf_convert_to_farf.c
v3.2/src/fuv/cf_flux_calibrate.c
v3.2/src/fuv/cf_ttag_init.c
v3.2/src/libcf/cf_modify_hist_pha.c
v3.2/src/libcf/cf_screen_jitter.c
v3.2/src/libcf/cf_set_good_time_intervals.c

Modified cf_convert_to_farf to write out the new PHA values assigned by cf_modify_hist_pha.

Added -e flag to cf_ttag_init. It sets the header keyword HKEXISTS to YES in the output IDF file.

In cf_screen_jitter, don't apply jitter correction to observations taken after the spacecraft lost pointing control.

If HV array in HSKP file consists of a single 0 followed by all -1's, use header info to populate timeline table. Change name of subroutine fill_hv_array to hv_from_hskp and add hv_from_header.

Within cf_bad_pixels, we scale nmax, the program's guess at the number of bad pixels it will have to deal with, by 1.5 to help prevent array overflows. To make warning messages more meaningful, we call cf_error_init after each call to cf_extraction_limits and make CF_PRGM_ID and CF_VER_NUM static variables.

March 29, 2007
Van Dixon
v3.2/src/libcf/cf_init_support.c
v3.2/src/cal/jitter/cf_jitter.c

On error, cf_check_digitizer now sets EXP_STAT flag to -2, rather than -999, to be more consistent with cf_hist_init.

We modified cf_jitter to handle the time period (in March 2007) when the ACS telemetry rate was cut in half and generally made the code more robust.

March 23, 2007
Van Dixon
v3.2/src/fuv/cf_hist_init.c
v3.2/src/libcf/cf_init_support.c
v3.2/src/libcf/cf_screen_jitter.c
v3.2/src/libcf/cf_satellite_jitter.c

In cf_hist_init, set EXP_STAT to -1 if a raw hist file contains no extensions. Treat the limits of the active area as good pixels.

In cf_init_support, give a more detailed error message if the housekeeping file is not usable.

In cf_screen_jitter and cf_satellite_jitter, calculate time_diff (the offset between the jitter and IDF versions of EXPSTART) once, rather than each time it's used.

In the subroutine cf_combine_pothole_data, if an entire dead spot falls outside of the aperture, skip it and move on to the next one.

March 7, 2007
Van Dixon
v3.2/src/libcf/cf_init_support.c
v3.2/src/libcf/cf_velang.c
v3.2/src/libcf/space_vel.c
v3.2/include/calfuse.h v1.81

The subroutine space_vel used a convoluted scheme to compute what is essentially a dot product. I simplified the code, deleted an argument, and modified the subroutines that call it.

February 13, 2007
Van Dixon
v3.2/src/fuv/cf_remove_motions.c
v3.2/src/libcf/cf_fifo_dead_time.c
v3.2/src/libcf/cf_init_support.c

When writing the LiF and SiC count rate arrays to the IDF file, cf_remove_motions crashed if the array values were too large to be stored as unsigned integers. As such enormous values are always wrong, they are now set to zero.

Sometimes, the header keyword TTPERIOD is erroneously set to 0. If so, cf_init_support and cf_fifo_dead_time now set it to 1.0, the most likely value.

January 2, 2007
Van Dixon
v3.2/src/fuv/cf_screen_photons.c
v3.2/src/fuv/cf_convert_to_farf.c
v3.2/src/fuv/cf_remove_motions.c
v3.2/src/libcf/cf_apply_dead_time.c
v3.2/src/libcf/cf_fifo_dead_time.c
v3.2/src/libcf/cf_target_count_rate.c
v3.2/src/libcf/pole_ang.c
v3.2/src/libcf/cf_velang.c
v3.2/src/libcf/cf_screen_high_voltage.c
v3.2/src/libcf/cf_satellite_jitter.c
v3.2/src/libcf/cf_grating_motion.c
v3.2/src/libcf/cf_time_xy_distort.c
v3.2/src/libcf/cf_init_support.c
v3.2/src/libcf/cf_set_photon_flags.c
v3.2/src/cal/jitter/cf_jitter.c
v3.2/src/analysis/cf_combine.c
v3.2/src/analysis/cf_nvo.c
v3.2/src/analysis/cf_pack.c
v3.2/src/analysis/idf_combine.c

To fix a bug in our correction for photon-event losses to FIFO overflows, we modified the subroutine cf_screen_fifo_overflow, renamed it cf_fifo_dead_time, and moved it from cf_screen_photons to cf_convert_to_farf. The subroutine cf_apply_dead_time issues a warning if DET_DEAD > 1.5, rather than TOT_DEAD. Because we no longer use the HV status flag to denote FIFO overflows, mention of the FIFO flag was removed from cf_set_photon_flags. Within cf_convert_to_farf, we now set the header keywords DET_DEAD, IDS_DEAD, and TOT_DEAD to unity by default.

Within cf_remove_motions, we now treat the LiF and SiC count-rate arrays as floats. We pass the WEIGHT array to cf_target_count_rate, so that it returns dead-time corrected count rates. If these count rates are too large to store as shorts, we use the TSCALE and TZERO keywords to store them as unsigned shorts. For HIST data, we scale the default LiF and SiC rates (read from the housekeeping file) by the value of DET_DEAD to account for detector dead-time losses.

The new subroutine pole_ang returns the angle between the orbit pole and a given RA and DEC. It is called by cf_velang, which populates the POLEANGL keyword.

The subroutine cf_find_spectra has been modified: if the new argument airglow_centroid is TRUE, then the program uses the airglow lines to compute the centroid of the target spectrum. To invoke this option, call cf_remove_motions with the -a flag.

The program cf_jitter has been extensively modified. Among other changes, it can (if necessary) compute the commanded quaternion from the target coordinates in the housekeeping file header. Values of the TRKFLG array now range from -1 to 5, with -1 meaning that tracking is bad, 0 that no information is available, and 1 to 5 representing increasingly reliable tracking estimates. The subroutines cf_satellite_jitter and cf_screen_jitter have been modified accordingly. In particular, the minimum value of TRKFLG that is to be considered reliable and the X and Y aperture limits are read from the pipeline parameter file.

Both the subroutine cf_grating_motion and the grating-motion calibration file have been completely revised. The new model depends on the target's beta and pole angles. An additional set of corrections tracks long-term drifts in the spectra's X and Y positions.

A new subroutine, cf_time_xy_distort, corrects for long-term drifts in the detector X and Y coordinate scales. It is called by cf_convert_to_farf.

We fixed a bug in the way that cf_init_support constructs the AIC count-rate array.

Lots of little changes in the analysis routines: When selecting which detector segment to include in the composite spectrum, cf_nvo now rejects segments with OBSTIME < 10 sec if the entire exposure is > 100 s. The program idf_combine adds comment fields to the header keywords WOFFS and WOFFL. The program cf_combine now counts SPEC### and WAVE### using the variable ngood, rather than nfiles. Finally, we made small changes to the way that cf_pack copies the SPEC keywords to the output file header.

June 22, 2006
Van Dixon
v3.2/src/fuv/cf_hist_init.c
v3.2/src/fuv/cf_extract_spectra.c
v3.2/src/fuv/cf_ttag_init.c
v3.2/src/libcf/cf_screen_jitter.c
v3.2/src/libcf/cf_screen_burst.c
v3.2/src/libcf/cf_optimal_extraction.c
v3.2/src/libcf/cf_fuv_init.c
v3.2/src/libcf/cf_error_msg.c

The pipeline no longer issues WARNING messages when a raw or intermediate data file is empty, either because it contains no photons or because the entire exposure has been flagged as bad; when the jitter file is missing (though out-of-date jitter files inspire a WARNING); when the detector voltage is low; and when the pipeline cannot perform optimal extraction. It issues level 1 verbose messages instead.

Now that OPUS has been updated, the variable morekeys has been set to zero in both cf_hist_init and cf_ttag_init.

May 24, 2006
Van Dixon
v3.2/src/fuv/cf_extract_spectra.c
v3.2/src/analysis/cf_combine.c
v3.2/src/analysis/cf_nvo.c
v3.2/src/analysis/cf_pack.c
v3.2/src/analysis/cf_xcorr.c
v3.2/src/analysis/idf_combine.c
v3.2/idl/cf_obsplot.pro

We modified cf_extract_spectra so that it will no longer discard (that is, set to zero the flux and errors arrays of) spectra with non-zero values of EXP_STAT. Those spectra are now rejected by cf_combine, instead.

The program cf_combine has been extensively revised: it now runs through the input files with a single loop; it supports the same command-line options as idf_combine; and it ignores spectra with EXP_STAT != 0 (unless the -a flag is set).

Both cf_nvo and cf_xcorr now use the Lyman beta line to align the spectra of background observations. They use the interstellar O VI and C II lines to align white dwarf spectra. For BKGD targets, cf_nvo always selects the channel with the highest effective area, regardless of its observed flux. The mean flux, used when deciding which segment to use, is now calculated using the same wavelength regions for each channel. For PC targets, the O VI lines are not included in the mean flux calculation.

The program cf_pack now writes an index to the various file extensions to the primary HDU of the output *000all* file.

Both cf_combine and idf_combine now write a number of header keywords as longs, rather than floats.

Three changes to cf_obsplot.pro: For BKGD targets, omit flux comparison when selecting which detector segments to use. Compute mean flux using same wavelength intervals for each band. For PC targets, don't include O VI in the calculation of the mean flux.

Divided cf_astigmatism_and_dispersion into two routines, incorporating astig_check_input_image into the new program cf_astigmatism and cf_x2lambda into cf_dispersion. Modified cf_assign_wavelength and cf_bad_pixels to call the new routines. Because cf_astig_farf is not used, it is no longer defined in calfuse.h or compiled by the Makefile.

For targets observed in HIST mode, cf_extraction_limits.c now pads apertures by SPECBINY pixels.

In cf_screen_burst, require that the number of windows in subroutine median_filter be >= 1. In cf_screen_fifo_overflow, changed length of data_rate array from nevents to nseconds. In cf_screen_jitter, screening is now applied to targets observed with APERTURE = RFPT. In cf_set_photon_flags, modify header keyword EXP_STAT only if was initially 0.

In the analysis directory, modified cf_make_all_obs.csh to be smarter when deciding whether to cross-correlate spectra or combine IDF files. Also modified logic for discarding 000 files. In cf_nvo, ignore segments with OBS_TIME = 0 and allow use of SiC data for 1000-1100 A region. (Sometimes the LiF channels are not used.) In cf_pack, if a segment is missing, write an empty image extension to the combined data file. In cf_xcorr, scale each spectrum to the mean flux of the reference spectrum.

February 3, 2006
Van Dixon
v3.2/src/fuv/cf_compute_y_centroid.c
v3.2/src/fuv/cf_find_spectra.c
v3.2/src/libcf/cf_scale_bkgd.c
v3.2/src/fuv/cf_extract_spectra.c

We tightened the limits over which cf_find_spectra computes the spectral centroid to prevent a bright star in a nearby aperture or the enhanced background near the detector edge from pulling the extraction window off center. In both cf_compute_y_centroid and cf_find_spectra, we returned the EMAX_LIF and EMAX_SIC keywords to their original behavior: if the centroid differs from the default value by more than this limit, the default value is used.

We fixed a bug introduced in the last modification of cf_scale_bkgd that essentially turned off the background subtraction.

We added a command-line flag to cf_extract_spectra that forces the use of optimal extraction, even for extended sources.

January 24, 2006
Van Dixon
v3.2/src/fuv/cf_screen_photons.c
v3.2/src/libcf/cf_scale_bkgd.c

Because cf_screen_burst wants to exclude airglow photons from the calculated background, we must identify them before calling the routine. To that end, I've made cf_screen_airglow the first of the screening routines.

The model background consists of a spatially-uniform component and a scattered-light image. The scattered-light component was over-estimated, because the data count totals were computed using a larger region of the detector than the scattered-light model totals. Now, we apply the same limits to both data and models.

The program returns a slice of the background that is supposed to match that covered by the probability array. Limit checks in the subroutine get_limits prevented the background slice from falling off of the detector. Sometimes, you want to background slice to fall off of the detector. In such cases, the limit checks could introduce an unwanted offset in the model background. We've deleted the limit checks and use other means to avoid bad index values.

cf_mirror_motion now exits without modifying the data if target is an extended source.

December 2, 2005
Van Dixon
v3.2/src/libcf/cf_astigmatism_and_dispersion.c
v3.2/src/libcf/cf_apply_dead_time.c
v3.2/src/libcf/cf_convert_to_ergs.c
v3.2/src/libcf/cf_count_rate_y_distort.c
v3.2/src/libcf/cf_doppler_and_heliocentric.c
v3.2/src/libcf/cf_extraction_limits.c
v3.2/src/libcf/cf_fpa_position.c
v3.2/src/libcf/cf_init_support.c
v3.2/src/libcf/cf_make_mask.c
v3.2/src/libcf/cf_optimal_extraction.c
v3.2/src/libcf/cf_satellite_jitter.c
v3.2/src/libcf/cf_screen_bad_pixels.c (NEW)
v3.2/src/libcf/cf_screen_fifo_overflow.c
v3.2/src/libcf/cf_screen_jitter.c (NEW)
v3.2/src/fuv/cf_bad_pixels.c
v3.2/src/fuv/cf_extract_spectra.c
v3.2/src/fuv/cf_remove_motions.c
v3.2/src/fuv/cf_screen_photons.c

To improve our bad-pixel correction, we've added a new routine, cf_screen_bad_pixels, which flags photon events falling in known pothole regions. cf_make_mask now fills gaps in the bad-pixel mask that open when it is converted from pixels to wavelengths. In the optimal-extraction algorithm, cf_optimal_extraction now scales the variance estimate by the bad-pixel mask to better approximate the observed counts. The main pothole routine, cf_bad_pixels, now uses cf_x2lambda() and cf_get_potholes(), rather than its own subroutines that do the same job. Also, it no longer adds random numbers to the output pixel coordinates -- that's no longer necessary. Finally, cf_bad_pixels no longer trims potholes at the edges of the extraction windows. We modified cf_astigmatism_and_dispersion so that it would not complain when events fall outside of the astig images.

We added a test to cf_apply_dead_time to catch bad dead-time values: If there is no housekeeping file, AND the engineering snapshot times are bad, AND the mean value of either dead-time array is > 1.5, set the array to unity and issue a warning.

In cf_convert_to_ergs, we changed the effective-area arrays to type double to match the AEFF_CAL files.

For histogram data, cf_count_rate_y_distort must use a single count rate value for all of the data. To better estimate it, we replace the time-dependent FEC-rate array with its weighted mean value.

To improve our calculation of the Doppler correction, we changed the variables velocity and doppler to doubles in cf_doppler_and_heliocentric.

Rather than using different HIST_PAD values for each aperture, cf_extraction_limits now always pads the apertures by 8 pixels (top and bottom) when in HIST mode.

More efforts to catch corrupted keywords in cf_init_support: When the housekeeping file is missing and the engineering snapshot times are corrupted, we use EXPSTART and EXPEND to estimate eng_time. If EXPEND is corrupted, we use EXPTIME.

To improve the jitter-correction algorithm, we've removed the data-screening functions from cf_satellite_jitter and moved them to a new routine, cf_screen_jitter. The modules cf_remove_motions and cf_screen_photons have been modified accordingly.

We caught a bug in cf_screen_fifo_overflow: It was using the RATE_LIF and RATE_SIC arrays to estimate the raw count rate, but those arrays are populated from the housekeeping file and do not reflect data losses due to FIFO overflows. Now we calculate the count rate across the entire detector directly from the photon-event data.

We added a flag to cf_extract_spectra that allows the user to disable optimal extraction from the command line.

September 7, 2005
Van Dixon
v3.1/src/analysis/cf_arith.c (RENAMED)
v3.1/src/analysis/cf_coadd.c
v3.1/src/analysis/cf_combine.c
v3.1/src/analysis/cf_make_all_exp.c (NEW)
v3.1/src/analysis/cf_nvo.c
v3.1/src/analysis/cf_pack.c
v3.1/src/analysis/cf_xcorr.c
v3.1/src/analysis/getshift.c
v3.1/src/analysis/modhead.c
v3.1/src/libcf/cf_init_support.c
v3.1/src/libcf/cf_fpa_position.c
v3.1/src/libcf/cf_read_fpa_pos.c (DELETED)
v3.1/src/libcf/cf_screen_burst.c
v3.1/src/libcf/cf_write_extracted_spectrum.c

We are required to provide MAST an all file for each exposure and one set of all, ano, and nvo files per observation. These files are created by cf_coadd, which has been completely re-written. It calls cf_make_all_exp for each exposure and cf_make_all_obs.csh once per observation. The new program cf_make_all_exp copies the primary HDU of the channel with the longest EXPTIME into an otherwise empty file. cf_nvo now sets the keyword NEXTEND back to 1. In cf_pack, the keyword COMB_COR is set to COMPLETE.

The name of cf_arith3 has been changed back to cf_arith. If called with no arguments, cf_arith, cf_combine, cf_xcorr, and getshift don't return an error, just a "usage" message. If the input list given to cf_combine contains only file names (no shifts), the program assumes that all shifts are zero. The programs cf_xcorr and cf_combine have been modified to use the same sign convention for spectral shifts as FUSE_REGISTER. The program modhead now accepts multi-word strings as keyword values.

In cf_init_support, use LWRS background limits for RFPT observations. In cf_fpa_position, correct for both X and Z displacement of FPA's. Delete call to cf_read_fpa_pos. In cf_screen_burst, ignore photons with arrival times greater than MAX_EXPTIME. In cf_write_extracted_spectrum, delete FITS_write_date from cf_copy_hist_header.

July 12, 2005
Van Dixon
v3.1/src/analysis/cf_combine.c
v3.1/src/analysis/cf_nvo.c
v3.1/src/analysis/cf_pack.c
v3.1/src/cal/jitter/cf_jitter.c
v3.1/src/libcf/cf_satellite_jitter.c

In cf_combine, we use fgets to read the input file and assume that pix_shift is an integer. In cf_nvo, we modify the cross-correlation routine to use LiF 2B as the wavelength standard if header keyword FESCENT is set to FES B. In cf_pack, we set the header keyword COMB_COR to COMPLETE.

cf_jitter writes a new keyword, JIT_VERS, to the header of each jitter file. If this keyword is missing or is less than 2, cf_satellite_jitter now warns that the jitter file format is out of date, which may result in the rejection of good data.

June 16, 2005
Van Dixon
v3.1/src/libcf/cf_rebin_probability_array.c
v3.1/src/libcf/cf_scale_bkgd.c
v3.1/src/fuv/cf_extract_spectra.c
v3.1/include/calfuse.h v1.71

A bug in the spectral-extraction subroutines caused CalFUSE always to use the smaller, point-source extraction window, even for extended sources. This could lead to an underestimate of the flux for extended sources. The pipeline now uses either the point-source or extended-source aperture, as appropriate.

June 8, 2005
Van Dixon
v3.1/src/analysis/cf_combine.c
v3.1/src/analysis/cf_make_all.csh
v3.1/src/analysis/cf_nvo.c
v3.1/src/analysis/cf_pack.c
v3.1/src/analysis/cf_xcorr.c
v3.1/src/analysis/extract_jitter.c
v3.1/src/analysis/idf_combine.c
v3.1/src/libcf/cf_calculate_y_centroid.c
v3.1/src/libcf/cf_find_spectra.c
v3.1/src/libcf/cf_idf_io.c
v3.1/src/libcf/cf_modify_hist_pha.c
v3.1/src/libcf/cf_rebin_probability_array.c
v3.1/include/calfuse.h v1.70

We modified cf_combine to support the generation of our new observation-level spectral files. It now writes the keywords NSPEC, SPEC###, and WOFF### to the primary HDU of the output file.

Three C routines and a shell script generate a new set of observation-level spectral files. For targets that are bright enough, we cross-correlate and combine the extracted spectra from each exposure. For fainter targets, we combine the IDF files from each exposure and extract a single spectrum for the whole observation. Then we repeat this exercise for the night-only data. Finally, we combine observation-level spectra from the various channels into a single spectral file for submission to the National Virtual Observatory.

cf_make_all is the shell script that calls these routines.
cf_xcorr determines the relative wavelength offset of each exposure.
cf_pack combines the observation-level spectra from each channel into a single ALL or ANO (all data, night-only) file.
cf_nvo selects data from the best channel for each waveband, shifts it to a common wavelengh scale, and generates the NVO file.

In idf_combine, comment out warning when FPA motion is detected. The pipeline corrects for this.

In cf_calculate_y_centroid, don't override user-specified centroid, even if it is far from the expected value. Fixed bug in tabulation of the expected Y centroids.

In cf_find_spectra, use airglow centroid of LWRS aperture to constrain airglow centroids of smaller apertures. Use airglow centroid of target aperture to constrain search for centroid of target spectrum.

In cf_rebin_probability_array, rounding errors left non-zero probabilities in regions beyond the ends of the extraction windows. Re-normalization raised these probabilities to unity. Now, if total probability in a column is less than 0.1, it is set to zero.

Between August and December of 2004, pipeline modifications were limited to bug fixes and a few new data-analysis routines. Since January of 2005, we made extensive modifications to the code, both to correct errors and to add new capabilities. All changes are documented in the headers of the individual modules and subroutines. Rather than repeat that information, we list here only the principal changes to the pipeline's capabilities.

In v3.0, data obtained during periods of uncertain pointing information were often flagged as bad. Now, the jitter files are generated using a simpler, more robust algorithm to determine when pointing information is reliable. If it's not, the pipeline does nothing, rather than flagging the data as bad.

In v3.0, data obtained when the detector voltage was less than 95% of the expected value were flagged as bad. Now, for detectors 1A, 1B, and 2B, the level is 85%. For detector 2A, data with HV > 90% of full is processed without comment, data with HV > 80% of full generates a warning message, and data with HV < 80% of full is flagged as bad. (These limits are stored in the VOLT_CAL file.) For detector 2A data in the 80-90% range, the user should examine the pulse-hight distribution of photon events in the target aperture to ensure that the peak of the distribution is well separated from the background.

In v3.0, the geometric-distortion correction moved small segments of the HIST spectra out of their extraction windows. The solution was to increase the size of the HIST extraction windows in Y. A new keyword, HIST_PAD, which depends on both the detector and the aperture, is read from the header of the corresponding extension of the CHID_CAL file. While making this change, we found and corrected a bug in the calculation of the spectral Y centroid that affected both TTAG and HIST data.

In v3.0, the HIST and TTAG wavelength scales did not agree. This was a result of the walk correction, which was applied to TTAG but not HIST data. Since the walk correction is never zero, HIST data were never fully converted to the FARF. We now assign a pulse height (the mean for a particular date and aperture) to each HIST photon and apply the walk correction to all data.

When scaling the model background image to match the data, the subroutine cf_scale_bkgd (formerly cf_ttag_bkgd) did not properly account for regions of the detector excluded because of airglow contamination. As a result, it underestimated the total background. The program has been extensively revised.

The final extracted spectral (*fcal.fit) files now have the standard FITS binary table format that was employed in old (v2.4.3 and earlier) versions of the pipeline.

Errors in the calculation of TIME_SUNRISE and TIME_SUNSET have been corrected. As these values are never greater than 6000 s, they are now stored (and treated) as SHORTs.

If the pipeline cannot compute the centroids of the detector stim pulses, it now reads a new calibration file (STIM_CAL), in which the time-dependent average values of the drift coefficients are tabulated.

The burst-detection algorithm computes the mean count rate on the unilluminated regions of the detector and writes it to the IDF. It now does so for the entire exposure, not just for times previously identified as good.

Several keywords have been added to the IDF file headers, including the limb-angle limits BRITLIMB and DARKLIMB from the SCRN_CAL file and the calculated orbital period (ORBPERID).

The AIRG_CAL files, which identify regions of each detector likely to be contaminated by airglow features, are now expressed in FARF, rather than RAW, coordinates. This change allows us to use smaller airglow windows and thus to throw away less data.

The default pulse-height limits (both upper and lower) have been modified for all four detectors.

August 24, 2004
Van Dixon
v3.0/src/libcf/cf_fes_proc_update.c
v3.0/src/libcf/cf_fes_proc_check.c
v3.0/src/libcf/cf_count_rate_y_distort.c
v3.0/src/libcf/cf_calculate_y_centroid.c
v3.0/src/libcf/cf_optimal_extraction.c
v3.0/include/calfuse.h v1.65

Installed two subroutines needed to process FES files. Modified calfuse.h to support them.

In cf_count_rate_y_distort, interpolate stretch correction among tabulated Y values to prevent counts from piling up at 10-pixel boundaries.

In cf_calculate_y_centroid, confirm that all spectral centroids, not just that of the target aperture, are within bounds.

In cf_optimal_extraction, use QUALITY array to set X limits of extraction region. Insure that sigma_out array is non-zero.

August 12, 2004
Van Dixon
v3.0/src/fuv/cf_assign_wavelength.c
v3.0/src/fuv/cf_bad_pixels.c
v3.0/src/fuv/cf_extract_spectra.c
v3.0/src/fuv/cf_remove_motions.c
v3.0/src/fuv/cf_screen_photons.c
v3.0/src/libcf/cf_astigmatism_and_dispersion.c
v3.0/src/libcf/cf_apply_filters.c
v3.0/src/libcf/cf_calculate_y_centroid.c
v3.0/src/libcf/cf_fuv_init.c
v3.0/src/libcf/cf_find_spectra.c
v3.0/src/libcf/cf_grating_motion.c
v3.0/src/libcf/cf_identify_channel.c
v3.0/src/libcf/cf_init_support.c
v3.0/src/libcf/cf_make_mask.c
v3.0/src/libcf/cf_mirror_motion.c
v3.0/src/libcf/cf_modify_hist_times.c
v3.0/src/libcf/cf_optimal_extraction.c
v3.0/src/libcf/cf_proc_check.c
v3.0/src/libcf/cf_rebin_background.c
v3.0/src/libcf/cf_satellite_jitter.c
v3.0/src/libcf/cf_screen_burst.c
v3.0/src/libcf/cf_screen_high_voltage.c
v3.0/src/libcf/cf_screen_limb_angle.c
v3.0/src/libcf/cf_set_photon_flags.c
v3.0/src/libcf/cf_thermal_distort.c
v3.0/src/libcf/cf_ttag_bkgd.c
v3.0/src/libcf/cf_write_extracted_spectrum.c
v3.0/src/libcf/eclipse.c
v3.0/src/libcf/solar_ang.c
v3.0/src/libcf/space_vel.c
v3.0/src/libcf/state_geod.c
v3.0/src/libcf/state_limb.c
v3.0/include/calfuse.h v1.64

Re-organized the pipeline, moving cf_screen_photons before cf_remove_motions. This required changes to both programs, as well as a number of subroutines.

Numerous modifications to cf_fuv_init: We now allow for the possibility that the time associated with a calibration file exactly equals EXPSTART. If the BRIT_OBJ keyword is set to DEFOCUS, we modify SRC_TYPE to E. Write CF_VERS to trailer file. Set ASTG_COR to PERFORM, as the pipeline doesn't do it automatically.

Changes to routines in cf_init_support: Add BRIT_OBJ keyword to header. Rewrite fill_hv_array() to make it more robust. If HV values in header are good, use them to determine whether HV arrays for 2A and 2B are swapped.

In both eclipse and state_geod, we corrected the FORTRAN call to sla_preces. To address warnings from the Mac OS X compiler, we made minor modifications to month_day.

If more than 90% of an exposure is lost to a burst, jitter, etc., cf_set_photon_flags will write the cause to the trailer file.

Changes to cf_identify_channel: Don't bother shifting extraction windows to correct for FPA position. If shift is known, it is not needed. Because the extraction windows don't quite match the WGTS arrays, pad windows by XPAD pixels in X. Change i (which steps through targ_ap array) from long to int.

Corrected bug in cf_mirror_motion that caused it to swap motion corrections for sides 1 and 2. (This routine also got a new calibration file.)

In cf_grating_motion, we estimate the time between sunrises using the orbit period in the file header, rather than assuming a 6000-second orbit.

In cf_satellite_jitter, we skip the jitter correction if the jitter file lacks the JIT_STAT keyword. The program returns EXP_JITR to the calling routine.

Both cf_find_spectra and cf_calculate_y_centroid now use the photon flags to exclude events arriving during bursts, etc., from their centroid calculations.

In cf_astigmatism_and_dispersion, must change ASTG_COR to COMPLETE or SKIPPED by hand, as the pipeline won't do it automatically.

Changes to extraction subroutines: Fixed bug in cf_ttag_bkgd: nxb was undefined if RUN_BKGD = NO. Don't bother shifting the weights array to account for FPA motions, as this should already have been corrected in the conversion to the FARF. In cf_optimal_extraction: If weights_out[j] = 0, set variance = bkgd. Iterate optimal extraction to accuracy of 0.01 counts. In the boxcar extraction algorithm, we reduced the parray limit to 1E-4 to better match fluxes from optimal extraction for bright stars. The routine cf_write_extracted_spectrum now has an option (-o) to override the default root name for the extracted spectra.

Removed unused variables from the following routines: state_limb, cf_rebin_background, cf_screen_high_voltage, cf_screen_limb_angle, cf_thermal_distort, solar_ang, space_vel, and cf_bad_pixels.

Made minor i/o changes to cf_screen_burst. The routine cf_proc_check now calls cf_verbose rather than cf_if_warning when a pipeline step is inappropriate for TTAG data.

We made a lot of changes to the way that we deal with time for HIST data: In cf_apply_filters, we mask the BRST, SAA, and LIMB flags when calculating exposure time. In cf_set_photon_flags, we set only DAY flag; we all write other flags to EXP_STAT keyword. A new routine, cf_modify_hist_times, sets the photon-arrival times of photons in HIST files to the midpoint of the longest good-time interval.

Modified cf_fuv_init to allow for the possibility that the MJD for a calibration file equals EXPSTART for an exposure.

May 13, 2004
Van Dixon
v2.4/src/fuv/cf_update_qual.c
v2.4/src/fuv/cf_dtcor.c
v2.4/include/calfuse.h v1.174

Modified cf_update_qual and cf_dtcor to deal gracefully with bad counters in the data file headers.

May 4, 2004
Van Dixon
v3.0/src/fuv/cf_assign_wavelength.c
v3.0/src/fuv/cf_countmap.c
v3.0/src/fuv/cf_extract_spectra.c
v3.0/src/fuv/cf_gainmap.c
v3.0/src/fuv/cf_hist_init.c
v3.0/src/fuv/cf_remove_motions.c
v3.0/src/fuv/cf_ttag_init.c
v3.0/src/libcf/cf_apply_dead_time.c
v3.0/src/libcf/cf_calculate_y_centroid.c
v3.0/src/libcf/cf_convert_to_ergs.c
v3.0/src/libcf/cf_doppler_and_heliocentric.c.c
v3.0/src/libcf/cf_error_msg.c
v3.0/src/libcf/cf_find_spectra.c
v3.0/src/libcf/cf_fuv_init.c
v3.0/src/libcf/cf_init_support.c
v3.0/src/libcf/cf_rebin_background.c
v3.0/src/libcf/cf_set_good_time_intervals.c
v3.0/src/libcf/cf_set_photon_flags.c
v3.0/src/libcf/cf_set_user_gtis.c
v3.0/src/libcf/cf_source_aper.c
v3.0/src/libcf/cf_ttag_bkgd.c
v3.0/src/libcf/cf_write_extracted_spectrum.c
v3.0/include/calfuse.h v1.61, v1.62, v1.63

Bernard performed housecleaning (removing unused variables, changing variable types to match subroutine definitions, etc.) on the following routines: cf_assign_wavelength, cf_hist_init, cf_ttag_init, cf_countmap, and cf_bad_pixels.c. Also these subroutines: cf_apply_dead_time, cf_calculate_y_centroid, cf_doppler_and_heliocentric.c, cf_error_msg, cf_init_support, cf_set_good_time_intervals, cf_set_photon_flags, cf_set_user_gtis, and cf_source_aper. In addition, he modified cf_ttag_init to create a photon list containing a single dummy event when the raw file is empty and cf_init_support to better catch errors in the header counter keywords.

We made significant modifications to cf_extract_spectra, replacing cf_rebin_and_flux_calibrate_background with cf_rebin_background. cf_optimal_extraction now performs all calculations with the weights array, rather than ergcm2. It returns counts and sigma arrays, rather than flux and variance as before; cf_extract_spectra must flux-calibrate each array, and cf_write_extracted_spectrum no longer converts from variance to sigma. The boxcar algorithm now uses the probability (weights) array to set the limits of the extraction window, better replicating the results of optimal extraction. Optimal extraction now iterates to 10% accuracy. The output weights and bkgd arrays are calculated by the optimal-extraction routine, so better match the final flux-calibrated spectra.

In verbose mode, cf_convert_to_ergs prints out the relative weighting of the two effective-area files.

The first call to cf_identify_channel was moved from cf_find_spectra to cf_remove_motions. The documentation of cf_remove_motions was improved.

The keywords MKBK_COR and BKGD_COR are gone, so cf_fuv_init no longer populates them.

The routine cf_rebin_and_flux_calibrate_background was replaced by cf_rebin_background. Flux calibration is now performed by cf_extract_spectra. cf_ttag_bkgd returns a null-valued background image if RUN_BKGD = NO.

April 5, 2004
Van Dixon
v3.0/src/libcf/cf_geometric_distort.c
v3.0/include/calfuse.h v1.60

April 1,2004
Van Dixon
v3.0/src/fuv/cf_extract_spectra.c
v3.0/src/libcf/cf_geometric_distort.c
v3.0/src/libcf/cf_hist_init.c
v3.0/src/libcf/cf_init_support.c
v3.0/src/libcf/cf_optimal_extraction.c
v3.0/src/libcf/cf_proc_check.c
v3.0/src/libcf/cf_rebin_probability_array.c
v3.0/src/libcf/cf_ttag_bkgd.c
v3.0/src/libcf/cf_write_extracted_spectrum.c
v3.0/include/calfuse.h v1.59

Abandon use of WPC array throughout the pipeline. Instead, modify cf_optimal_extraction to evenly distribute photon events across one X pixel when in HIST mode. If optimal-extraction loop does not converge within 50 iterations, exit and use results from boxcar extraction.

In cf_proc_check, call cf_verbose rather than cf_if_warning if program not appropriate for HIST data.

Change the way that routines deal with empty data files. In cf_hist_init, issue a warning -- but continue processing -- if NEVENTS = 0. (A similar change for cf_ttag_init will have to wait for the next version of the pipeline.) In cf_extract_spectra, write null-valued output spectra if either EXPTIME or NEVENTS = 0. In cf_write_extracted_spectrum, if valid_spectrum = FALSE, write warning message to file header.

Include math.h in cf_hist_init to define fmod(). This solves the long-standing problem of the program crashing on Linux machines.

In cf_geometric_distort, modify header keyword SPECBINY to reflect Y spacing in the FARF.

In cf_ttag_bkgd, set Y limits of 2-D background arrays equal to those of the probability (weights) array. Since the sizes are now equal, cf_optimal_extraction does not need to test them. Make centroid of probability array a float in cf_extract_spectra and associated subroutines.

In cf_rebin_probability_array, shift probability array to account for position of FPA.

March 30, 2004
Van Dixon
v2.4/src/fuv/cf_hist_init.c
v2.4/include/calfuse.h v1.173

March 16,2004
Van Dixon
v3.0/src/fuv/cf_extract_spectra.c
v3.0/src/fuv/cf_hist_init.c
v3.0/src/fuv/cf_ttag_init.c
v3.0/src/fuv/cf_bad_pixels.c
v3.0/src/fuv/cf_countmap.c
v3.0/src/fuv/cf_gainmap.c
v3.0/src/fuv/cf_convert_to_farf.c
v3.0/src/libcf/cf_astigmatism_and_dispersion.c
v3.0/src/libcf/cf_init_support.c
v3.0/src/libcf/cf_satellite_jitter.c
v3.0/src/libcf/cf_thermal_distort.c
v3.0/src/libcf/cf_write_extracted_spectrum.c
v3.0/include/calfuse.h v1.57, v1.58

Implement WPC array in cf_extract_spectra and associated subroutines.
In cf_hist_init.c, fix error in correcting hot pixels.
In cf_ttag_init.c, exit if no photons in raw data file.
In cf_bad_pixels.c, change maximum size of the output arrays in cf_combine_pothole_data and insure that lif_cnt and sic_cnt arrays are non zero.
In cf_countmap.c and cf_gainmap.c, allow user to select coordinate system: RAW, FARF (default), final.
In cf_astigmatism_and_dispersion.c, make subroutine cf_x2lambda globally accessible.
In cf_convert_to_farf.c, pass weight array to cf_thermal_distort.
In cf_init_support.c, read begin and end counters as float. Set countrate to zero when cntb is more than 16777216.
In cf_satellite_jitter.c, polulate the PLATESC keyword in the input file header.
In cf_thermal_distort.c, added weights to the calculation (needed for histogram mode) and corrected bug in procedure for finding stim pulses.
In cf_write_extracted_spectrum.c, populate archive search keywords and change name of POTHOLE column to QUALITY.

May 22, 2003
Van Dixon
v2.4/src/idl/cf_plot_rate.pro
v2.4/src/fuv/fcal.c
v2.4/src/fuv/cf_make_ttag_bkgd.c
v2.4/include/calfuse.h v1.172

Modified cf_plot_rate.pro, fcal.c, and cf_make_ttag_bkgd.c to deal gracefully with zero-length exposures. Resulting spectral file has FLUX and ERROR arrays set to zero.

May 19, 2003
Van Dixon
v2.4/src/analysis/cf_combine.c
v2.4/include/calfuse.h v1.171

Imported two of Jeff's modifications to cf_combine:
Close each input file in initial loop. Without this, cfitsio crashes upon opening the 60th file.
Weight each pixel by exposure time after excluding ``bad'' pixels.

May 2, 2003
Van Dixon
v2.4/src/analysis/cf_uninterp.c
v2.4/include/calfuse.h v1.170

Added cf_uninterp.c to re-calibrate extracted spectra using only the most recent flux-calibration file.

April 29, 2003
Van Dixon
v2.3/src/fuv/cf_ttag_jitter.c
v2.3/src/fuv/cf_hist_jitter.c
v2.3/include/calfuse.h v1.169

Modified jitter routines to allow possiblity that jitter filename is in lower-case letters.

April 25, 2003
Van Dixon
v2.3/src/fuv/cf_fuv_init.c
v2.3/include/calfuse.h v1.167, v1.168

In cf_fuv_init, never linearly extrapolate calibration files forward in time. If an observation is not bracketed by calibration files, simply use the latest one.

April 16, 2003
Van Dixon
v2.3/src/fuv/cf_fuv_init.c
v2.3/include/calfuse.h v1.166

In cf_fuv_init, test for corrupted detector bias values. Updated comment field of HV_FLAG. Modified associated warning messages.

April 15, 2003
Van Dixon
v2.3/src/fuv/cf_ttag_screen.c
v2.3/include/calfuse.h v1.165, v1.166

April 2, 2003
Van Dixon
v2.3/src/fuv/cf_ttag_geodopp.c
v2.3/src/fuv/cf_ttag_jitter.c
v2.3/include/calfuse.h v1.164

Simplified DX/DY test in cf_ttag_geodopp. Test that TRKFLG is <= 5 (the highest allowed value) in cf_ttag_jitter.

April 2, 2003
Van Dixon
v2.3/src/fuv/cf_ttag_geodopp.c
v2.3/include/calfuse.h v1.163

Test DX and DY values to make sure they are valid before combining into mean Doppler shift. NaN values lead to bad-pixels masks full of zeros, which lead to empty extracted spectra.

March 10, 2003
Van Dixon
v2.3/src/fuv/cf_ttag_geodopp.c
v2.3/include/calfuse.h v1.162

Alain Lecavelier points out that, for high-count-rate TTAG data, the pipeline produces error bars that are about half of what one would expect from SQRT(N) statistics. Turns out that the change that I made to cf_ttag_geodopp in March 2002 was in error. I returned the code to its previous state, and the error bars are now much better behaved. Error bars for histogram data are unaffected by this change.

February 24, 2003
Van Dixon
v2.2/src/fuv/cf_make_ttag_bkgd.c
v2.2/src/fuv/cf_make_hist_bkgd.c
v2.2/include/calfuse.h v1.161

Peter Young's Dec Alpha compiler identified a bug in these routines. EXPNIGHT is stored in the header as type LONG, but these routines read it as type FLOAT. CFITSIO should make the conversion automatically, but apparently fails to do so on his machine. I've modified the calls to FITS_read_key() in both routines.

February 20, 2003
Van Dixon
v2.2/src/libcf/read_tle.c
v2.2/src/libcf/sgp4.c
v2.2/src/fuv/cf_ttag_jitter.c
v2.2/src/fuv/cf_make_ttag_bkgd.c
v2.2/src/fuv/cf_extract.c
v2.2/src/fuv/cf_driftcorr2.c
v2.2/src/fuv/cf_coadd.c
v2.2/src/analysis/ttag_combine.c
v2.2/include/calfuse.h v1.160

Peter Young's Dec Alpha compiler identified two sets of bugs in these routines. Several calls to FITS_read_key() or FITS_update_key() used an ampersand incorrectly:
FITS_read_key(fptr, TSTRING, "INSTRUME", &instrument, comment, &status);
and a few routines used abs() when fabs() is required.

P.Y.'s compiler refused to recognize one of the #define keywords in sgp4.c, so Bryce Roberts replaced it with the actual value throughout the routine.

December 13, 2002
Van Dixon
v2.2/src/fuv/cf_ttag_screen.c
v2.2/src/fuv/cf_hist_screen.c
v2.2/src/fuv/cf_ttag_burst.c
v2.2/src/fuv/cf_ttag_jitter.c
v2.2/include/calfuse.h v1.159

Modify cf_ttag_screen and cf_hist_screen to write all EXP_* keywords as type long. Modify cf_ttag_burst and cf_ttag_jitter to pass through all photon events in the stim-pin regions, even during bad times. Have them add EXP_BRST and EXP_JITR to EXP_BAD, and write all as type long.

November 8, 2002
Van Dixon
v2.2/src/fuv/cf_extract.c
v2.2/include/calfuse.h v1.158

November 6, 2002
Van Dixon
v2.2/src/fuv/cf_hist_jitter.c
v2.2/src/fuv/cf_ttag_jitter.c
v2.2/include/calfuse.h v1.157

Make cf_hist_jitter and cf_ttag_jitter issue warning, but continue running, if keyword JIT_STAT is not found.

November 1, 2002
Van Dixon
v2.2/src/fuv/cf_hist_jitter.c
v2.2/idl/cf_plot_hist_rate.pro
v2.2/include/calfuse.h v1.156

Make cf_hist_jitter and cf_plot_hist_rate exit gracefully if jitter file not found.

October 29, 2002
Van Dixon
v2.2/src/analysis/cf_arith.c
v2.2/src/analysis/cf_combine.c
v2.2/include/calfuse.h v1.155

October 24, 2002
Van Dixon
v2.2/src/fuv/Makefile
v2.2/src/fuv/cf_make_hist_bkgd.c
v2.2/include/calfuse.h v1.154

October 11, 2002
Van Dixon
v2.2/src/fuv/cf_ttag_jitter.c
v2.2/include/calfuse.h v1.153

Three changes to cf_ttag_jitter: Initialized nrej to 0, made program exit quietly if jitter file not found, and set all times to floats in test step, to correct bug whereby some photons were being erroneously rejected. Note that the program still rejects photons with times later than the end of the input good-time interval. This happens because the burst-correction routine converts the GTI boundaries to integers.

August 29, 2002
Van Dixon
v2.2/src/fuv/cf_hist_init.c
v2.2/src/fuv/cf_ttag_init.c
v2.2/include/calfuse.h v1.152

August 28, 2002
Van Dixon
v2.2/src/fuv/cf_hist_jitter.c
v2.2/src/libcf/cf_fcal.c
v2.2/idl/cf_plot_hist_rate.pro
v2.2/include/calfuse.h v1.151

cf_hist_jitter reads the jitter file, determines how much time (if any) the target spent out of the aperture, and writes this number to the keyword EXP_JITR. cf_fcal subtracts this value from EXPTIME when flux-calibrating the spectrum. The IDL procedure cf_plot_hist_rate.pro plots the count rate during a histogram exposure using counter values from the housekeeping file.

July 31, 2002
Van Dixon
v2.2/src/fuv/cf_hist_count_map.c
v2.2/src/fuv/cf_ttag_count_map.c
v2.2/src/fuv/cf_ttag_gain_map.c
v2.2/src/fuv/cf_make_ttag_bkgd.c
v2.2/src/fuv/cf_ttag_jitter.c
v2.2/src/fuv/cf_update_qual.c
v2.2/src/libcf/cf_fuv_init.c
v2.2/include/calfuse.h v1.150

The SDP group decided that CalFUSE 3.0 should use a radically different scheme for processing time-tagged data. In the mean time, we have made incremental progress on CalFUSE v2, and we have decided that this version of the program should go behind the firewall as soon as possible. Accordingly, I copied v2.1.7 to v2.2.0 and have continuued to make incremental changes to the code. Highlights are

Modify cf_hist_count_map, cf_ttag_count_map, cf_ttag_gain_map, and cf_update_qual to read version 002 YSTR files.
Correct a Linux bug in cf_make_ttag_bkgd.
Add cf_ttag_jitter and modify cf_fuv_init to support it.

June 14, 2002
Van Dixon
v2.1/src/fuv/cf_make_ttag_bkgd.c
v2.1/include/calfuse.h v1.149

June 14, 2002
Van Dixon
cf_make_ttag_bkgd.c
cf_undistort.c
cf_driftcorr.c
cf_extract.c
cf_dpix.c
cf_ttag_screen.c
cf_hist_dopp.c
cf_wcal.c
cf_doppler_util.c
calfuse.h v1.151

In cf_make_ttag_bkgd, correct error when using MKBK_COR=OMIT (1.15) and fill in holes made by the removal of geocoronal lines. The latter is needed in determination of X variations of scattered light. Re-wrote cf_undistort to handle shifts as floats. Modified cf_driftcorr to write output as an array of shorts. Modified cf_extract, cf_dpix to read linear wavelength scale. Modified read_wavelengths in cf_doppler_util and calls to it in cf_ttag_screen and cf_hist_dopp. Modified cf_wcal to implement linear wavelength solution.

May 30, 2002
Van Dixon
cf_doppler_util.c
cf_ttag_screen.c
cf_hist_dopp.c
cf_dpix.c
cf_extract.c
calfuse.h v1.150

In cf_doppler_util, modified read_wavelength no longer to generate and return an xpix array, then modified the calling routines cf_ttag_screen and cf_hist_dopp accordingly. Modified cf_dpix and cf_extract to handle the linear wavelength scale.

May 29, 2002
Van Dixon
cf_doppler_util.c
cf_driftcorr.c
cf_update_qual.c
calfuse.h v1.149

In cf_doppler_util, modified the subroutine read_wavelength to handle a linear wavelength scale. Installed Rich's new version of cf_driftcorr, which puts a correction for the count-rate dependent Y stretch into the DY image. Because we no longer need to correct the bad-pixel map for the Y stretch, I moved back from v1.8 to v1.5 of cf_update_qual.

Fixed a couple of bugs in cf_dpix: made LWRS the default aperture and allowed the boundary between the SiC and LiF regions to vary with detector.

Updated #define keywords in cf_dtcorr. Turned on X-distortion correction in cf_undistort and moved it to follow cf_driftcorr2. Implemented linear wavelength scale in cf_wcal.

April 16, 2002
Van Dixon
cal_fuse_exp_ttag.csh
cal_fuse_exp_hist.csh
calfuse.h v1.145

April 8, 2002
Van Dixon
cf_make_hist_bkgd
cf_make_ttag_bkgd
cf_ttag_burst
bchr*001.fit
scrn*013.fit
calfuse.h v1.144

New software from Rich: New BCHR (background characterization) files contain the binary extensions that used to be in the SCRN files. Pipeline modules were modified to read the new files.

March 29, 2002
Van Dixon
cf_make_ttag_bkgd
cf_ttag_burst
bkgd*005.fit
bkgd*006.fit
bkgd*007.fit
bkgd*008.fit
scrn*012.fit
calfuse.h v1.143

New software from Rich: The burst module has been modified to update the EXPNIGHT value. The background module has a new procedure for determining the intrinsic background from the data. The background files should do a better job at short wavelengths. The screening files redefine the background sample regions and add a few more geocoronal line positions.

March 17, 2002
Van Dixon
cf_make_ttag_bkgd
cf_make_hist_bkgd
cf_ttag_geodopp
cf_hist_geodopp
calfuse.h v1.142

Turns out that all of these programs were treating the error arrays as sigmas rather than variances (sigma squared). Modified to propagate errors correctly.

Introduced new keyword, RAWTIME, which is equal to EXPTIME before CalFUSE starts tossing out data. Modified cf_fuv_init and ttag_combine to propagate it correctly.

Modified cf_dtcor to determine exposure time from the keywords CTIME_B and CTIME_E, which are the times of the initial and final engineering packets from which the counter data were taken.

Modified logic of voltage test in cf_fuv_init. Add COMMENT lines to data file header if low voltage found. Respond gracefully if bias keywords do not exist.

Modified cf_fuv_init to check detector voltage and complain if low. New volt***001.fit calibration files, VOLT_CAL keyword.

Richard's new version of cf_update_qual modifies the bad-pixel maps to account for the count-rate dependence of the detector Y scale.

November 30, 2001
Van Dixon
cf_dpix, cf_extract, cf_wormcorr, cf_optextract, cf_spexextract, cf_plot_extract.pro, cal_fuse_exp_ttag.csh, cal_fuse_exp_hist.csh, parm**007.fit
calfuse.h v1.134

In earlier versions of the pipeline, the shift to correct for FPA offsets was performed by cf_wcal. We have moved it earlier in the pipeline, right after the astigmatism correction, and combine the FPA corrections with the other shifts in order to have a single float-to-integer conversion. The new module is called cf_dpix.

cf_extract now reads the keyword RUN_OPTI, rather than OPT_EXTR, from the parameter file. It also writes the Y centroid of each spectrum to the corresponding output file header.

Sylvestre added software to identify the worm in the LiF 1B spectrum: cf_wormcorr reads the keyword RUN_WORM from the parameter file. If RUN_WORM = LIF, SIC, or BOTH, the program attempts to identify the region of the worm and write the range of affected pixels (WORM_BEG and WORM_END) to the output file header. The IDL routine cf_plot_extract.pro draws a ring around the worm, if found.

Modified cal_fuse_exp_ttag.csh and cal_fuse_exp_hist.csh to call both cf_dpix and cf_wormcorr. Modified calfuse.h to include new processing keywords DPIX_COR and WORM_COR.

Changed the way that cf_ttag_screen treats user-defined good-time intervals. Now takes intersection of them with pre-defined intervals (from OPUS or cf_ttag_combine) before screening. Also changed the module to apply all screens (SAA, day/night, limb angle, pulse height) regardless of where the good-time intervals came from. Screening can be turned off by modifying the scrn*.fit files, if desired.

Defined NO_PIXEL in calfuse.h. Used by geometric-distortion and extraction routines.

Modified cf_dtcor to set counters to zero if values are too large to read as longs.

Modified cf_fuv_init to check header keyword OPUSVERS and complain if the data file was processed using an old version of OPUS.

Modified cf_ttag_init to reset the keywords SPECBINX, SPECBINY, SIA_TBL, and NUM_HIST without checking their current value. This code was moved from cf_ttag_image, where it had been commented out in April 2001.

Rich Robinson has made numerous upgrades to ttag_combine. Among other things, it can now handle the DX, DY and DNFLG columns added by cf_ttag_init.

Jeff Kruk has produced new flux-calibration files based on the latest wavelength calibration.

Don Lindler has produced new wavelength files appropriate for data that do NOT have the walk correction applied to their stim pins. They were created by taking the previous line locations and correcting them before fitting by:

Version 008 had new PHA limits. Version 009 had new information for background models. Version 010 has both.

New version of background routine reads the keyword BKGDTYPE in the screening file. Default value is 1; if set to -1, routine does not perform an iterative fit to the observed scattered light.

Installed new wavelength calibration files, v013, which are corrected for the motion of the stim pins due to walk correction in previous versions of the pipeline.

Don't apply walk correction to photons/pixels within STIM_PAD (512) pixels of the detector edge. New walk-correction files (v003) have zeros in the stim-pin region.

If MKBK_COR is set to OMIT, cf_make_ttag_bkgd changes it to SKIPPED. cf_extract copies the keyword OPT_EXTR into the FITS file headers.

New cf_make_ttag_bkgd iteratively fits dark-count and scattered-light components of the background to deal with the fact that the dark count varies with time. Spits out a file filled with zeros if MKBK_COR is set to OMIT.

August 23, 2001
Van Dixon
calfuse.h 1.122
cf_fuv_init
cf_ttag_burst
cf_ttag_walk
cf_hist_walk
cf_astig
cf_make_hist_bkgd
cf_make_ttag_bkgd
cf_bkgd

Enable user to turn off the burst rejection, walk correction, background subtraction, or astigmatism correction by changing keywords in parm**006.fit. cf_fuv_init reads the PARM file and changes the appropriate keywords (e.g., BRST_COR) from PERFORM to OMIT. Subsequent pipeline steps check their keyword; if it is OMIT, they change the value to SKIPPED, issue a warning, and exit, copying input to output if appropriate. Updated CALFUSE version to 2.0.2.

Moved cf_fpa_pos from cf_wcal to libcf. ttag_combine uses this routine to determine the FPA positions and complains if an FPA moves by more than 10 microns between exposures.

Header keywords SPECBINX and SPECBINY should be 1 for ttag data. An old version of OPUS set these to 0. Changed cf_ttag_init to update these parameters if it finds them equal to 0. Also updated cf_dtcor to use Dave Sahnow's new parameters for dead-time correction. Added a check: if DET_DEAD > 2.0, assume it's wrong and set to 1.0.

New version of cf_astig employs Sylvester's cross-correlation routine to determine spectral centroids. New PHAM, WAVE, and FLUX files. Updated CALFUSE version to 2.0.1.
This is the beta-test version of CalFUSE v2.0.

July 2, 2001
Van Dixon
calfuse.h 1.118
cf_ttag_init.c
cf_ttag_screen.c
cf_ttag_burst.c
cf_make_ttag_bkgd.c
cf_make_hist_bkgd.c

Implemented Richard's new background modeling routine, which independently computes the day- and night-time contributions to the scattered-light background. To do so, he adds an extra column, DNFLG, to all of the photon-event files. Package includes improvements to the burst-screening algorithm. Updated CALFUSE version to 2.0.0.

Changed sign returned by the LSR correction subroutines to conform with astronomical standard. Updated CALFUSE version to 1.9.9

Alex Fullerton determined that we have been applying the heliocentric velocity correction with the wrong sign. The orbital dopper correction has been applied with the correct sign. The function helio_vel.c inverts the sign returned by the SLALIB routines, so the simplest correction is to remove this sign inversion.

April 10, 2001
Van Dixon
calfuse.h 1.111
cf_hist_walk.c 1.1
cal_fuse_exp_hist.csh

Installed new module to apply walk correction to histogram data. Modified shell script cal_fuse_exp_hist.csh to call this routine. Added PHAM entry to CALIBRATION_FILE_KEYS in calfuse.h. Updated CALFUSE version to 1.9.8

April 2, 2001
Van Dixon
calfuse.h 1.110
cf_ttag_walk.c 1.1
cal_fuse_exp_ttag.csh
cal_fuse_exp_hist.csh

Installed new module to apply walk correction. Modified shell scripts cal_fuse_exp_ttag.csh and cal_fuse_exp_hist.csh to call this routine. Added WALK entry to CALIBRATION_FILE_KEYS in calfuse.h. Updated CALFUSE version to 1.9.7

March 28, 2001
Van Dixon
calfuse.h 1.109
cf_extract.c 1.52
cal_fuse_exp_ttag.csh
cal_fuse_exp_hist.csh

Installed new module to perform optimal extraction. Modified shell scripts cal_fuse_exp_ttag.csh and cal_fuse_exp_hist.csh to call this routine. In calfuse.h, added definition of cf_optextract and new arguments to cf_spextract. Updated CALFUSE version to 1.9.6

March 27, 2001
Van Dixon
calfuse.h 1.108
cf_make_bkgd.c 1.22
cal_fuse_exp_ttag.csh
cal_fuse_exp_hist.csh

Installed new module to scale background calibration file. Modified shell scripts cal_fuse_exp_ttag.csh and cal_fuse_exp_hist.csh to call this routine (instead of cf_make_bkgd_temp). Add AIRG entry to CALIBRATION_FILE_KEYS in calfuse.h. Updated CALFUSE version to 1.9.5

March 19, 2001
Van Dixon
calfuse.h 1.107
cf_ttag_burst.c 1.9
cal_fuse_exp_ttag.csh
cal_fuse_exp_hist.csh

Installed new module to reject data during detector bursts. Modified shell scripts cal_fuse_exp_ttag.csh and cal_fuse_exp_hist.csh to call this routine. Modified Makefiles to install it. Updated CALFUSE version to 1.9.4

March 14, 2001
Van Dixon
calfuse.h 1.106
cf_astig.c 1.8
cal_fuse_exp_ttag.csh
cal_fuse_exp_hist.csh

Installed new program to implement astigmatism correction, now called cf_astig. Modified shell scripts cal_fuse_exp_ttag.csh and cal_fuse_exp_hist.csh to call this routine. Modified Makefiles to install it. Updated CALFUSE version to 1.9.3

Add checks to cf_ttag_screen.c to ensure that user-defined good time intervals actually fit within the time spanned by the dataset, and that in each case the end of a good time interval is later than the beginning. Invalid user GTI's are ignored. Also added printout to trailer file of the number of events that were rejected by pulse height limits. Updated CALFUSE version to 1.9.2

February 8, 2001
J.C. Hsu
calfuse.h 1.104
cf_linear.c 1.0
cf_transform1d.c 1.0
cf_1dff.c 1.0
cf_make_1dff.c 1.0

Install programs needed to perform 1-D flat fielding. All Makefiles updated. The "official" pipeline shell scripts were not updated; users wishing to try the 1-D flatfields should edit their local copies of the shell scripts for the time being. Thus, the pipeline version was not updated.

December 21, 2000
Paul Barrett, J.C. Hsu
Makefile.Alpha
cf_ttag_screen.c 1.54
cf_ttag_image.c 1.22
cf_ttag_init.c 1.16
cf_wcal.c 1.25

In response to problems encountered by Vincent LeBrun installing calfuse on a DEC Alpha running True64, we made a number of changes. The main one was to correct type mismatches in cfitsio calls: in a number of cases an "int" variable was referenced as TLONG and vice versa. Added -ieee flag to compiler options in Makefile.Alpha. cf_ttag_image: fix type mismatches in cfitsio calls cf_ttag_init: fix type mismatches in cfitsio calls cf_ttag_screen: fix type mismatches in cfitsio calls cf_wcal: fix type mismatches in cfitsio calls This is the first instance in which v1.9.1 differs from v1.8.7, and no change in functionality was involved, so the pipeline version was not updated.

November 15, 2000
Jeff Kruk
cf_make_ff_temp.c 1.6
cf_make_bkgd_temp.c 1.6
cf_ttag_screen.c 1.53

Fixed typos in print statements in several routines. No change in functionality resulted, so the pipeline version number was not incremented.

Created CALFUSE version 1.9.1 as the new development version. calfuse.h was updated to reflect the new version number.

The image centroid (in Y) for extended sources is once again calculated including airglow lines. In many cases extended sources are very faint and the airglow will help locate the spectra. Extended sources will ordinarily fill the slits in the same manner as airglow, so the accuracy of the centroid calculation will not be adversely affected.

October 13, 2000
Jeff Kruk
cf_ttag_screen.c 1.52
cf_extract.c 1.46
cf_fuv_init.c 1.9

Three calls to FITS_read_key (infits, TSTRING, keyword, &arrayname,...) in the program cf_ttag_screen were modified to remove the extraneous ampersand.

One call to FITS_read_key (infits, TSTRING, keyword, &arrayname,...) in cf_extract.c was modified to remove the extraneous ampersand. Also fixed a bug in the test for the variable ystop_bis being out of limits.

October 2, 2000
Jeff Kruk, JC Hsu
cf_hist_dopp 1.25
calfuse.h 1.101
cal_fuse_exp_ttag.csh
cal_fuse_exp_hist.csh

The program cf_hist_dopp was modified to incorporate corrections for grating thermal motion. The mean motion for the exposure is added to the doppler correction.

The shell script cal_fuse_exp_hist.csh was modified to call cf_make_shift and to pass the resulting shift file to cf_hist_dopp.

The include file calfuse.h was modified to include the function prototype for lif_sic_border() (see below), and to change the MKSH_COR histogram pipeline entry in CALIBRATION_STEP_KEYS to PERFORM.

The shell scripts cal_fuse_exp_ttag.csh and cal_fuse_exp_hist.csh were modified to test cfstat before calling the idlplot routines.

October 2, 2000
Jeff Kruk
cf_check_point 1.17
cf_hist_screen 1.21
cf_ttag_screen 1.51

The programs cf_hist_screen and cf_ttag_screen were modified to update the keyword EXPNIGHT, newly created to provide the number of seconds of exposure time (after screening) that were obtained during the night side of the oribt. This keyword will be used by an upcoming version of cf_bkgd.

The function cf_check_point was modified to return a flag indicating whether the satellite was in daylight or night at the time being checked.

The pipeline version number was updated for an additional change made later in the day.

The function lif_sic_border() was extracted from cf_hist_dopp.c and make a function in libcf so that it would be available for the upcoming 1-D version of cf_ff. The libcf Makefiles were modified accordingly.

No change in functionality resulted, so no update to the pipeline version number was made.

The function state_limb() was modified (v1.08) to include fortran function prototype wrappers for the new slalib calls introduced by the previous change (slaDcc2s, slaEvp, slaDranrm).

This change was solely for running Make on the distribution version of calfuse, so no update to the pipeline version number was made.

September 26, 2000
Martial Andre
state_limb 1.05, 1.06, 1.07
cf_check_point 1.16
cf_orbit 1.03
calfuse.h 1.98, 1.99, 1.100

The function state_limb() was modified (v1.05) to calculate the limb angle constraint in the same way as is done by the mission planners: it checks if the limb itself is in daylight or night, rather than the old simplistic method of seeing if the satellite was in day or night. This required a modification to the call to state_limb. Versions 1.06, 1.07 of state_limb were to fix minor bugs.

The function cf_check_point (v1.16) and the program cf_orbit (v1.03) were modified to make use of this new limb angle constraint calculation.

The include file calfuse.h (v1.98, v1.99) was modified to provide the proper function prototype for the new versions of state_limb and cf_check_point.

These changes led to the increase in pipeline version number to 1.8.5 (implemented by version 1.100 of calfuse.h).

Ed Murphy & Paul Barrett produce distribution version of pipeline. Major change was to incorporate "makemake" files.

Since this routine is still in testing, I have not updated the pipeline version number. The new version of cf_astig2.c is 1.07.

Jeff Kruk modified cf_dtcor.c to populate three new keywords in the FUV header: DEAD_DET Detector dead time factor DEAD_IDS IDS dead time factor DEAD_TOT Combined dead time factor New version of cf_dtcor.c is 1.18. He also changed sunmath.h to math.h.

Modify cf_astig2 to get the calibration file from the ASTG_CAL keyword rather than being hardcoded.

Since this routine is still in testing, I have not update the pipeline version number. The new version of cf_astig2.c is 1.06.

Jeff Kruk found that cf_astig2 crashed on all segment 1B data. The routine lacked any boundary checking and was running off the bottom of the image when attempting to centroid the SiC spectrum. The centroiding routines have been updated to include boundary checking.

Since this routine is still in testing, I have not update the pipeline version number. The new version of cf_astig2.c is 1.05.

Martial fixed a bug in the initialization of column_sum that caused a fatal bus error. The change is too small to update the pipeline version number.

I added calls to cf_proc_check and cf_proc_update to cf_update_qual to rid the trailer files of the warnings concerning the QUAL_COR keyword. New version of cf_update_qual is 1.03. The change is too small to update the pipeline version number.

To avoid the centroid to be shifted close to the bright edge, the procedure should compute the size of the scanning window for each different channel (the changes done in version 1.41 were wrong!!!)

Propagation of quality should have been done with float since the very beginning (now fixed).

July 26, 2000
J. C. Hsu, R. Robinson, Ed Murphy
cf_update_qual v1.02
cf_driftcorr2 v1.23

J. C. Hsu has written a program cf_update_qual to update the quality flags in the *_img.fit files after the drift correction has been determined. The FARF positions of the dead spots and edges of the detectors were determined by Rich Robinson and are in the qual*001.fit calibration files. The drift correction is used to correct for the strecth/shift in going from the FARF to the observed data frame.

When implementing cf_update_qual, Ed Murphy noticed that cf_driftcorr2 was centroiding the stims incorrectly. The error arose because the algorithm was computing the stim centroid in a box that had an extent of +-256 in X. Since the stims are at pixel ~230 (on segment 1a) the centroiding box was including counts along the X=0 column, which skewed the centroid. In the particular observation that he used to test the routine (M999 flat field) the X=0 column was particularly bright due to the high count rates in a stim lamp observation. This problem probably does not effect most observations, since the X=0 column is normally much weaker. He also modified cf_driftcorr2 to use the FARF stim positions given in calfuse.h as opposed to the positions that were hard coded into the routine.

Fixed bug of implementation of a pointer in im_centroid wich caused a bus error.

Jeff Kruk added an IDS dead time correction to the dead time correction routine cf_dtcor.c.

Now the default resolution of the coadded spectra is 0.1 Angstrom. This way, we make sure that astronomers outside JHU don't use them for science purpose (since some of the channels are currently misaligned => artefacts). Note that the user is still free to have maximal resolution by tuning the arguments.

Peter Young noticed that the Doppler shift in pixels for the SiC and LiF spectra on segment 1a were both in the same direction. Since the dispersions run in opposite directions, the two shifts should be opposite in sign. The DISPAPIX keyword in the wavelength calibration file records the average dispersion in Angstroms/pixel. It should be positive when the wavelength increases with increasing pixel number and it should be negative when the wavelength decreases with increasing pixel number (SiC1 and LiF2). However, I found that the DISPAPIX keyword was always positive in the recent versions of the wavelength calibration files. Therefore, for SiC1 and LiF2, the sign of the Doppler and heliocentric corrections are incorrect.

I have modified the code in cf_ttag_screen and cf_hist_dopp to read in the absolute value of DISPAPIX and compute the sign of the shift based on the aperture to which the photon belongs. The new version of cf_ttag_screen is 1.49 and the new version of cf_hist_dopp is 1.23.

I have also removed the flat-fielding step from the pipeline. Even though we do not have a flat-field file, we have been applying a flat field (filled with 1's) to the data so that we could incorporate a bad pixel mask. However, we are not stretching/shifting the flat field file (since it is all 1's) to match the data which means that the bad pixel mask does not line up with the data. J.C. Hsu is in the process of writing a new module to properly apply the bad pixel mask. However, the routine cf_ttag_geodopp.c still requires the _ff.fit file, so we must still run cf_make_ff_temp. This intermediate _ff.fit data file contains all 1's over the active area of the detectors, and 0's everywhere else. Since we are not running the cf_ff, the locations where the flat field is 0 are not being flagged as bad. This was causing cf_ttag_geodopp to report a lot of "Divide by zero in ffbuf" errors. For now, I have turned off the warning in cf_ttag_geodopp.

The algorithm of finding the centroid has been totally changed. Now the position of the spectrum is reached via a maximisation of the flux in the extraction window. Moreover, the airglows are systematically screened to center only on the source.

The way the windows are calculated changed drastically. Using the parameter of the distortion correction, the shape of each window has been simplified (butterfly or trapeze) and the weigth files are filled with parabolic profiles (first approximation). For more details see /data1/fuse/calfuse/v1.8/src/cal/extraction_masks/README

Jeff Kruk upgraded cf_wcal to read either the new FPA keywords (FPASXPOS or FPALXPOS) or the old keywords (FP1SXPOS, FP1LXPOS, FP2SXPOS, or FP2LXPOS). New version is 1.24.

Ed Murphy found and corrected a bug in cf_extract that prevented the SiC extraction windows from being reset to the default position if the centroid was more than EMAX_SIC pixels from the exptected position.

Paul Barret changed the interpolation algorithm back to a constant (to prevent ringing around the outside of the spectrum) and fixed a bug in that implementation.

Jeff Kruk has modified cf_wcal.c (now version 1.23) to correct the wavelength scale for the position of the FPA.

Paul Barrett has produced new versions of the detector distortion files: geom*005.fit in /data1/fuse/calfuse/calfiles.

The pipeline can now generate a few diagnostic plots (using IDL) while it is running if the CF_IDLDIR environment variable is set to point to the location of the IDL programs. To set up the pipeline to make the plots, add the following to your .cshrc or .login file:
setenv CF_IDLDIR '/data1/fuse/calfuse/current/idl/'

The plots are in GIF format so that they can, at some future time, be linked to the SDOG pages.

Development on V1.7.4 of the pipeline has been frozen. Only bug fixes will be made from now on. The current stable version of the pipeline now points to V1.7.4. You will need new calibration files, new parameter files, and a new master_calib_file.dat

It has been suggested that a higher order interpolation scheme be used when expanding the binned histogram data, instead of the current constant interpolation scheme. A quadratic average interpolation algorithm has been implemented in cf_hist_init. This algorithm uses data from neighboring pixels to interpolate the data in binned pixles while conserving the total number of counts per binned pixel. This routine has been tested.

This modification to cf_hist_init will now allow histogram data to be used for determining distortion corrections by providing necessary spatial resolution.

In the SDOG report for P1320101 Jeff Kruk found that the pipeline seemed to be excessively screening out data that was near, but not violating, the night time limb angle limit. I found flawed logic in the limb angle check in cf_check_point.c that meant that the bright limb angle was always used regardless of the day/night flag. The problem has been corrected (cf_check_point.c now V1.15) and the routine has been tested.

Bill Oegerle noticed that the pipeline was not subtracting the background from some recently processed datasets. The pipeline has been in this state since sometime around the end of December 1999. I fixed the problem in both versions of the pipeline: new versions are V1.6.8 and V1.7.2

The problem was found in cf_make_bkgd_temp.c, which is a temporary module designed to read in the background calibration file and write it into the *ttagf_bkgd.fit file needed by the pipeline. The module does not stretch or shift the background image but does scale the background by the integration time. In the calibration file, the background image is composed of short integers with the value 1 everywhere. The keyword BSCALE gives the number of square cm per pixel, so that the file contains the number of counts per pixel per second. The original version of cf_make_bkgd_temp.c read in the calibration image and wrote it out directly, without converting the input image to floats. All it did was multiply BSCALE by the integration time, so that when cf_bkgd.c read in the file and scaled by BSCALE, the resulting image would be background counts per pixel for the full integration.

In late December 1999, it was decided to rewrite cf_make_bkgd_temp.c to rebin the data for histogram mode observations. In this case, the new routine read in the calibration file array (all 1), converted the background image to floats by applying BSCALE, rebinned the data (if necessary), and wrote out an array of floats. However, the header was copied from the input file, so the BSCALE keyword still remained even though the data had already been scaled as part of the routine. In addition, the step where BSCALE was multiplied by the integration time also remained in the new version. Therefore, when cf_bkgd.c read in the *ttagf_bkgd.fit intermediate file, it used the BSCALE keyword a second time. The result was that the subtracted background was about two orders of magnitude too small.

Because the pipeline now unbins the histogram data, we have reverted back to the original versions of the routines which do not rebin the data and do not suffer from this problem. The new version of cf_make_bkgd_temp.c is V1.4.

A few histogram data sets were crashing the pipeline becuause of a bad pixel value that was larger than 32767. This value was large enough that it would not fit into a short integer in the pipeline and cfitsio would return a data conversion error. Jim Rose has agreed to modify the OPUS pipeline to find pixel values larger than 32767 and change them to the data fill value 21865. As seen below, the pipeline will convert the data fill value to 0.0 and change its quality flag to bad.

A number of histogram data sets have fill data in them (decimal 21865) which the pipeline was not filtering out. I have modifed v1.6.7 and v1.7.1 of the pipeline to check pixels for the fill data value. If the fill data value is found, it will replace the data with 0.0 and change the quality flag to bad.

Until now, the DATE keyword in the header was not modified by any of the modules in the pipeline, even though its comment stated that it was the "date the file was written." I have modified cf_proc_update.c to update the DATE keyword. Since cf_proc_update is called at the end of each module of the pipeline, the output files and all intermediate data products will now have accurate dates and times in the DATE keyword. This change was made to both V.1.6.7 and V1.7.1 of the pipeline.

Until now, the cf_extract procedure was unable to perform a good centroiding in Y when very few counts were involved. Since we used the formula : Ycenter = (å Yi * nbcounts_i)/ånbcounts_i, where nbcounts_i is the total number of counts along the row i, the result was depending on the Y limits of the sum. For instance, if the upper limit equals infinity, the result is infinity (do not forget that the noise is different of 0). So, to use this formula, we need a first guess for the position of the center in order

to choose a domain of calculation surrounding it. At present, this first guess is the maximum peak.

The new extraction windows are now available (spex*005.fit). This is the first set of In Orbit Based Extraction Windows : all the apertures have the same window (based on 100 percent with LWRS).

The associated weight files are also available (wgts*003.fit). Note that since the raw windows extracted from In Orbit Data are very noisy, I had to smooth and correct the geometrical distorsions , sometimes " by hand ".


Detector	Data Used	Notes
1A (LiF)	/data2/fuse/CAL/M101/M1010201	Good
1A (SiC)	/data2/fuse/CAL/M103/M1030303	Good
1B (LiF)	/data2/fuse/CAL/M101/M1010201	Corrected by hand : mirror transformation applied
1B (SiC)	/data2/fuse/CAL/M103/M1030304	Corrected by hand : linear regression for the narrow domain
2A (LiF)	/data2/fuse/CAL/M101/M1010201	Good
2A (SiC)	/data2/fuse/CAL/M103/M1030304	Good
2B (LiF)	NOT AVAILABLE	Here I didn't find a signal strong enough ... I used the old Ground Based Files + geometrical correction
2B (SiC)	/data2/fuse/CAL/M103/M1030304	Ground Based Files + geometrical correction

Note also that you can change the windows with "cf_modifwindows" wich apply an homothetic factor to the LiF channel and the SiC channel :

By popular demand, I have written a program to convert an ASCII file to the 1-D FITS extracted format that comes out of the pipeline. The program is called ascii2fits and is in the /usr/local/fusesw/calfuse/current/bin directory. The program takes 3 arguments: Calling sequence:ascii2fits ascii_file template_fits_file output_fits_file The ASCII file must have six columns: wavelength (float) flux (float) errors (float) quality flags (int) counts (float) errors on counts (float) The template FITS file is used to populate the header keywords. It can be any raw or pipeline processed file.

Overlapping pixels are now averaged by cf_hist_geodopp. This is a temporary fix to minimize the effect on the spectrum due to a mis-match between the BARF and FARF, i.e. it removes the spikes from the spectrum.

Version 1.7.1 of the pipeline is now in the FARF frame, however, the calibration files are not.

The dead time correction step has been removed from the pipeline. The new version of pipeline is V1.6.5. The pipeline module itself was fine. However, Dave Sahnow has found that the on-board counters behave strangely at low count rates (in addition to behaving strangely in the SAA). Until this issue is resolved, we will not dead time correct the data.

Histogram data currently show strong spikes approximately once per Angstrom. This behavior is not unexpected. It comes from the fact that we have to compress the image in the X direction by about 110 pixels in order to bring the data into the reference frame of the calibration files. Until we can redefine the calibration file reference frame and remake all our calibration files, this behavior will be present in the data.

Version 1.6.4 of the CALFUSE pipeline is now frozen. In the future, only bug fixes will occur. The /usr/local/fusesw/calfuse/current directory will now point to v1.6. All development will take place in v1.7.

REMINDER: Many people directly reference "v1.5" rather than "current" in their PATH and LD_LIBRARY_PATH. If you do this, be sure to change both PATH and LD_LIBARARY_PATH to v1.6. Otherwise, you will be running the pipeline with outdated copies of the library routines since we are now using dynamically linked libraries (linked in at run time).

CalFUSE Pipeline Status