I'm ready to advance to a highly technical and in-depth discussion on processing Spitzer data.

From CoolWiki
Jump to navigationJump to search


The basic idea here is the same as the basic idea for making a simple mosaic. The difference here is the following: whereas before you were combining products from our pipeline (which takes into account a variety of things, nearly all of which are instrumental artifacts), now you are starting from individual frames, so it becomes much more of your responsibility to take into account instrumental artifacts. You need to tell MOPEX not just the individual frames and errors, but also the individual mask files, including masks for permanently damaged pixels (which change with time as the telescope ages) and masks for instrumental artifacts (like cosmic rays and scattered light). This process has MANY MORE buttons and dials than the prior, simpler process - which is good once you're an expert, because you can remove the artifacts and leave a great image. But all these options also means you can really easily shoot yourself in the foot if you are not careful.

For IRAC, to do this properly, for most star-forming regions, not only do you have 4 channels, but you also have a short and a long exposure (from the HDR setting in the AOR) for each pointing. So you have 8 mosaics to make (and artifacts to remove separately from all the components of those 8 mosaics).

For MIPS, you may or may not have to remove the artifacts manually for MIPS-24. The pipeline automatically produces two sets of BCDs for 70 and 160, and you need to decide which you should use, based on your goals for the final image. Artifact removal from these two longest bandpasses can be quite tricky indeed.

  • Formal, complete data filenaming conventions for these Spitzer data (not at all sugar-coated)
  • Data Handbooks for these Spitzer data (tells you which files are important, but also not at all sugar-coated)

high-level overview, plus general cryptic notes put here until we have time to turn them into something more user-friendly.

media:processingoverview.doc -- more wordy doc file with more description, waiting to be translated into here.

run sean's artifact mitigation software on irac data. uses idl.

requires a flat directory structure that is just a nightmare!

after it finished: moved all log files to sean_intermed moved final files to sean_final


moved all other intermediate files to sean_intermed

mv over_cor_sfx_SPITZER_I1_*_bcd.fits sean_final
mv over_cor_sfx_SPITZER_I2_*_bcd.fits sean_final
mv over_cor_sfx_SPITZER_I3_*_bcd.fits sean_final
mv over_cor_sfx_SPITZER_I4_*_bcd.fits sean_final
mv cor_sfx_SPITZER_I1*bunc.fits sean_final
mv cor_sfx_SPITZER_I2*bunc.fits sean_final
mv cor_sfx_SPITZER_I3*bunc.fits sean_final
mv cor_sfx_SPITZER_I4*bunc.fits sean_final
mv SPITZER_I1*bimsk.fits sean_final
mv SPITZER_I2*bimsk.fits sean_final
mv SPITZER_I3*bimsk.fits sean_final
mv SPITZER_I4*bimsk.fits sean_final 
cp *list sean_final
cp *list sean_intermed
mv *log sean_intermed
mv *I1*fits sean_intermed
mv *I2*fits sean_intermed
mv *I3*fits sean_intermed
mv *I4*fits sean_intermed

irac_preprocess is still subdir for all code, and make_corrected_bcds.tcsh is still in this main dir.

all *list files got copied to sean_intermed, sean_final.

  • list files in this dir need to be updated to point to sean_final.
awk '{printf("sean_final/%40s\n",$1)}' ch1234.list > ch1234.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch1_long.list > ch1_long.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch2_long.list > ch2_long.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch3_long.list > ch3_long.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch4_long.list > ch4_long.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch1_med.list > ch1_med.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch2_med.list > ch2_med.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch3_med.list > ch3_med.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch4_med.list > ch4_med.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch1_long_dmasks.list > ch1_long_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch2_long_dmasks.list > ch2_long_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch3_long_dmasks.list > ch3_long_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch4_long_dmasks.list > ch4_long_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch1_med_dmasks.list > ch1_med_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch2_med_dmasks.list > ch2_med_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch3_med_dmasks.list > ch3_med_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch4_med_dmasks.list > ch4_med_dmasks.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch1_long_sigmas.list > ch1_long_sigmas.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch2_long_sigmas.list > ch2_long_sigmas.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch3_long_sigmas.list > ch3_long_sigmas.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch4_long_sigmas.list > ch4_long_sigmas.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch1_med_sigmas.list > ch1_med_sigmas.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch2_med_sigmas.list > ch2_med_sigmas.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch3_med_sigmas.list > ch3_med_sigmas.list.txt
awk '{printf("sean_final/%40s\n",$1)}' ch4_med_sigmas.list > ch4_med_sigmas.list.txt

sscsci101% which mosaic.pl mosaic.pl: Command not found. sscsci101% source ../rebull/mopex/mopex-script-env.csh sscsci101% which mosaic.pl /ssc/sci/pleiades/rebull/mopex//bin/mosaic.pl

Generate the FIF as a separate step (not required to be separate, but i am getting strange errors now UNLESS i make it a separate step) if you do this, though, turn off FIF generation for individual mosaics, since it will save you time (and, evidently, errors). edit the file cdf/justfif_gui.nl

mosaic.pl -n justfif_gui.nl

make the mosaic: mosaic.pl -n mosaic_long_sr.nl > mosaic8.log &

maybe useful to make one just of the first year to compare with the mosaics you made mon/tues:

grep 13375232 ch1_long.list.txt > ch1_long.list235.txt
grep 13375232 ch2_long.list.txt > ch2_long.list235.txt
grep 13375232 ch3_long.list.txt > ch3_long.list235.txt
grep 13375232 ch4_long.list.txt > ch4_long.list235.txt
grep 13375232 ch1_med.list.txt > ch1_med.list235.txt
grep 13375232 ch2_med.list.txt > ch2_med.list235.txt
grep 13375232 ch3_med.list.txt > ch3_med.list235.txt
grep 13375232 ch4_med.list.txt > ch4_med.list235.txt
grep 13375232 ch1_long_dmasks.list.txt > ch1_long_dmasks.list235.txt
grep 13375232 ch2_long_dmasks.list.txt > ch2_long_dmasks.list235.txt
grep 13375232 ch3_long_dmasks.list.txt > ch3_long_dmasks.list235.txt
grep 13375232 ch4_long_dmasks.list.txt > ch4_long_dmasks.list235.txt
grep 13375232 ch1_med_dmasks.list.txt > ch1_med_dmasks.list235.txt
grep 13375232 ch2_med_dmasks.list.txt > ch2_med_dmasks.list235.txt
grep 13375232 ch3_med_dmasks.list.txt > ch3_med_dmasks.list235.txt
grep 13375232 ch4_med_dmasks.list.txt > ch4_med_dmasks.list235.txt
grep 13375232 ch1_long_sigmas.list.txt > ch1_long_sigmas.list235.txt
grep 13375232 ch2_long_sigmas.list.txt > ch2_long_sigmas.list235.txt
grep 13375232 ch3_long_sigmas.list.txt > ch3_long_sigmas.list235.txt 
grep 13375232 ch4_long_sigmas.list.txt > ch4_long_sigmas.list235.txt
grep 13375232 ch1_med_sigmas.list.txt > ch1_med_sigmas.list235.txt
grep 13375232 ch2_med_sigmas.list.txt > ch2_med_sigmas.list235.txt
grep 13375232 ch3_med_sigmas.list.txt > ch3_med_sigmas.list235.txt
grep 13375232 ch4_med_sigmas.list.txt > ch4_med_sigmas.list235.txt

grep 13375232 ch1234.list.txt > ch1234.list235.txt

make a mosaic just of the 235 files. update a (new copy of a) namelist and invoke it. mosaic.pl -n mosaic235.nl > m.log

mosaic.pl -n justfif_gui.nl > fif.log
mkdir p235pbcd3.6long
mkdir p235pbcd3.6short
mkdir p235pbcd4.5long
mkdir p235pbcd4.5short
mkdir p235pbcd5.8long
mkdir p235pbcd5.8short
mkdir p235pbcd8long
mkdir p235pbcd8short
cp FIF.tbl p235pbcd3.6long
cp FIF.tbl p235pbcd3.5short
cp FIF.tbl p235pbcd4.5long
cp FIF.tbl p235pbcd4.5short
cp FIF.tbl p235pbcd5.8long
cp FIF.tbl p235pbcd5.8short
cp FIF.tbl p235pbcd8long
cp FIF.tbl p235pbcd8short
mosaic.pl -n mosaic235i1l.nl > m1.log
mosaic.pl -n mosaic235i2l.nl > m2.log
mosaic.pl -n mosaic235i3l.nl > m3.log
mosaic.pl -n mosaic235i4l.nl > m4.log

mosaic.pl -n justfif_gui.nl > fif2.log
mkdir pbcd3.6long
mkdir pbcd3.6short
mkdir pbcd4.5long
mkdir pbcd4.5short
mkdir pbcd5.8long
mkdir pbcd5.8short
mkdir pbcd8long
mkdir pbcd8short
cp FIF.tbl pbcd3.6long
cp FIF.tbl pbcd3.6short
cp FIF.tbl pbcd4.5long
cp FIF.tbl pbcd4.5short
cp FIF.tbl pbcd5.8long
cp FIF.tbl pbcd5.8short
cp FIF.tbl pbcd8long
cp FIF.tbl pbcd8short
mosaic.pl -n mosaicfulli1l.nl > m1lfull.log
mosaic.pl -n mosaicfulli1s.nl > m1sfull.log
mosaic.pl -n mosaicfulli2l.nl > m2lfull.log
mosaic.pl -n mosaicfulli2s.nl > m2sfull.log
mosaic.pl -n mosaicfulli3l.nl > m3lfull.log
mosaic.pl -n mosaicfulli3s.nl > m3sfull.log
mosaic.pl -n mosaicfulli4l.nl > m4lfull.log
mosaic.pl -n mosaicfulli4s.nl > m4sfull.log
grep 13375232 ch1_long.list > ch1_long.list235
grep 13375232 ch2_long.list > ch2_long.list235
grep 13375232 ch3_long.list > ch3_long.list235
grep 13375232 ch4_long.list > ch4_long.list235
grep 13375232 ch1_med.list > ch1_med.list235
grep 13375232 ch2_med.list > ch2_med.list235
grep 13375232 ch3_med.list > ch3_med.list235
grep 13375232 ch4_med.list > ch4_med.list235
grep 13375232 ch1_long_dmasks.list > ch1_long_dmasks.list235
grep 13375232 ch2_long_dmasks.list > ch2_long_dmasks.list235
grep 13375232 ch3_long_dmasks.list > ch3_long_dmasks.list235
grep 13375232 ch4_long_dmasks.list > ch4_long_dmasks.list235
grep 13375232 ch1_med_dmasks.list > ch1_med_dmasks.list235
grep 13375232 ch2_med_dmasks.list > ch2_med_dmasks.list235
grep 13375232 ch3_med_dmasks.list > ch3_med_dmasks.list235
grep 13375232 ch4_med_dmasks.list > ch4_med_dmasks.list235
grep 13375232 ch1_long_sigmas.list > ch1_long_sigmas.list235
grep 13375232 ch2_long_sigmas.list > ch2_long_sigmas.list235
grep 13375232 ch3_long_sigmas.list > ch3_long_sigmas.list235
grep 13375232 ch4_long_sigmas.list > ch4_long_sigmas.list235
grep 13375232 ch1_med_sigmas.list > ch1_med_sigmas.list235
grep 13375232 ch2_med_sigmas.list > ch2_med_sigmas.list235
grep 13375232 ch3_med_sigmas.list > ch3_med_sigmas.list235
grep 13375232 ch4_med_sigmas.list > ch4_med_sigmas.list235
cp *list235 COMEGETME
cp p235pbcd3.6long/Combine/mosaic.fits COMEGETME/p235i1long.fits
cp p235pbcd4.5long/Combine/mosaic.fits COMEGETME/p235i2long.fits
cp p235pbcd5.8long/Combine/mosaic.fits COMEGETME/p235i3long.fits
cp p235pbcd8long/Combine/mosaic.fits COMEGETME/p235i4long.fits
cp p235pbcd3.6long/Combine/mosaic_cov.fits COMEGETME/p235i1long_cov.fits
cp p235pbcd4.5long/Combine/mosaic_cov.fits COMEGETME/p235i2long_cov.fits
cp p235pbcd5.8long/Combine/mosaic_cov.fits COMEGETME/p235i3long_cov.fits
cp p235pbcd8long/Combine/mosaic_cov.fits COMEGETME/p235i4long_cov.fits
cp p235pbcd3.6long/Combine/mosaic_unc.fits COMEGETME/p235i1long_unc.fits
cp p235pbcd4.5long/Combine/mosaic_unc.fits COMEGETME/p235i2long_unc.fits
cp p235pbcd5.8long/Combine/mosaic_unc.fits COMEGETME/p235i3long_unc.fits
cp p235pbcd8long/Combine/mosaic_unc.fits COMEGETME/p235i4long_unc.fits

p pbcd3.6long/Combine/mosaic.fits COMEGETME/i1long.fits
cp pbcd4.5long/Combine/mosaic.fits COMEGETME/i2long.fits
cp pbcd5.8long/Combine/mosaic.fits COMEGETME/i3long.fits
cp pbcd8long/Combine/mosaic.fits COMEGETME/i4long.fits
cp pbcd3.6long/Combine/mosaic_cov.fits COMEGETME/i1long_cov.fits
cp pbcd4.5long/Combine/mosaic_cov.fits COMEGETME/i2long_cov.fits
cp pbcd5.8long/Combine/mosaic_cov.fits COMEGETME/i3long_cov.fits
cp pbcd8long/Combine/mosaic_cov.fits COMEGETME/i4long_cov.fits
cp pbcd3.6long/Combine/mosaic_unc.fits COMEGETME/i1long_unc.fits
cp pbcd4.5long/Combine/mosaic_unc.fits COMEGETME/i2long_unc.fits
cp pbcd5.8long/Combine/mosaic_unc.fits COMEGETME/i3long_unc.fits
cp pbcd8long/Combine/mosaic_unc.fits COMEGETME/i4long_unc.fits
cp pbcd3.6short/Combine/mosaic.fits COMEGETME/i1short.fits
cp pbcd4.5short/Combine/mosaic.fits COMEGETME/i2short.fits
cp pbcd5.8short/Combine/mosaic.fits COMEGETME/i3short.fits
cp pbcd8short/Combine/mosaic.fits COMEGETME/i4short.fits
cp pbcd3.6short/Combine/mosaic_cov.fits COMEGETME/i1short_cov.fits
cp pbcd4.5short/Combine/mosaic_cov.fits COMEGETME/i2short_cov.fits
cp pbcd5.8short/Combine/mosaic_cov.fits COMEGETME/i3short_cov.fits
cp pbcd8short/Combine/mosaic_cov.fits COMEGETME/i4short_cov.fits
cp pbcd3.6short/Combine/mosaic_unc.fits COMEGETME/i1short_unc.fits
cp pbcd4.5short/Combine/mosaic_unc.fits COMEGETME/i2short_unc.fits
cp pbcd5.8short/Combine/mosaic_unc.fits COMEGETME/i3short_unc.fits
cp pbcd8short/Combine/mosaic_unc.fits COMEGETME/i4short_unc.fits

tim found the overlap correction had gone bad in ch3. redoing, using the SSC's overlap corrector. overlap.pl -n overlap_long_sr.nl > overlap.log