I'm ready to advance to a highly technical and in-depth discussion on processing Spitzer data.

From CoolWiki
Revision as of 23:12, 11 March 2008 by Rebull (talk | contribs)
Jump to navigationJump to search

STILL UNDER DEVELOPMENT... working on it

  • Formal, complete data filenaming conventions for these Spitzer data (not at all sugar-coated)
  • Data Handbooks for these Spitzer data (tells you which files are important, but also not at all sugar-coated)


The basic idea here is the same as the basic idea for making a simple mosaic. The difference here is the following: whereas before you were combining products from our pipeline (which takes into account a variety of things, nearly all of which are instrumental artifacts), now you are starting from individual frames, so it becomes much more of your responsibility to take into account instrumental artifacts. You need to tell MOPEX not just the individual frames and errors, but also the individual mask files, including masks for permanently damaged pixels (which change with time as the telescope ages) and masks for instrumental artifacts (like cosmic rays and scattered light). This process has MANY MORE buttons and dials than the prior, simpler process - which is good once you're an expert, because you can remove the artifacts and leave a great image. But all these options also means you can really easily shoot yourself in the foot if you are not careful.

high-level overview, plus general cryptic notes put here until we have time to turn them into something more user-friendly.

run [sean's artifact mitigation software] on irac data. uses idl.

equires a flat directory structure that is just a nightmare!

after it finished: moved all log files to sean_intermed moved final files to sean_final 

       over_cor_sfx_SPITZER_I[1-4]..._bcd.fits
       cor_sfx_SPITZER_I[1-4]..._bunc.fits
       SPITZER_I[1-4]..._bimsk.fits

moved all other intermediate files to sean_intermed


mv over_cor_sfx_SPITZER_I1_*_bcd.fits sean_final mv over_cor_sfx_SPITZER_I2_*_bcd.fits sean_final mv over_cor_sfx_SPITZER_I3_*_bcd.fits sean_final mv over_cor_sfx_SPITZER_I4_*_bcd.fits sean_final mv cor_sfx_SPITZER_I1*bunc.fits sean_final mv cor_sfx_SPITZER_I2*bunc.fits sean_final mv cor_sfx_SPITZER_I3*bunc.fits sean_final mv cor_sfx_SPITZER_I4*bunc.fits sean_final mv SPITZER_I1*bimsk.fits sean_final mv SPITZER_I2*bimsk.fits sean_final mv SPITZER_I3*bimsk.fits sean_final mv SPITZER_I4*bimsk.fits sean_final

cp *list sean_final cp *list sean_intermed mv *log sean_intermed

mv *I1*fits sean_intermed mv *I2*fits sean_intermed mv *I3*fits sean_intermed mv *I4*fits sean_intermed


irac_preprocess is still subdir for all code, and make_corrected_bcds.tcsh is still in this main dir.

all *list files got copied to sean_intermed, sean_final.

  • list files in this dir need to be updated to point to sean_final.

awk '{printf("sean_final/%40s\n",$1)}' ch1234.list > ch1234.list.txt

awk '{printf("sean_final/%40s\n",$1)}' ch1_long.list > ch1_long.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch2_long.list > ch2_long.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch3_long.list > ch3_long.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch4_long.list > ch4_long.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch1_med.list > ch1_med.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch2_med.list > ch2_med.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch3_med.list > ch3_med.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch4_med.list > ch4_med.list.txt

awk '{printf("sean_final/%40s\n",$1)}' ch1_long_dmasks.list > ch1_long_dmasks.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch2_long_dmasks.list > ch2_long_dmasks.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch3_long_dmasks.list > ch3_long_dmasks.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch4_long_dmasks.list > ch4_long_dmasks.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch1_med_dmasks.list > ch1_med_dmasks.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch2_med_dmasks.list > ch2_med_dmasks.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch3_med_dmasks.list > ch3_med_dmasks.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch4_med_dmasks.list > ch4_med_dmasks.list.txt

awk '{printf("sean_final/%40s\n",$1)}' ch1_long_sigmas.list > ch1_long_sigmas.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch2_long_sigmas.list > ch2_long_sigmas.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch3_long_sigmas.list > ch3_long_sigmas.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch4_long_sigmas.list > ch4_long_sigmas.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch1_med_sigmas.list > ch1_med_sigmas.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch2_med_sigmas.list > ch2_med_sigmas.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch3_med_sigmas.list > ch3_med_sigmas.list.txt awk '{printf("sean_final/%40s\n",$1)}' ch4_med_sigmas.list > ch4_med_sigmas.list.txt

sscsci101% which mosaic.pl mosaic.pl: Command not found. sscsci101% source ../rebull/mopex/mopex-script-env.csh sscsci101% which mosaic.pl /ssc/sci/pleiades/rebull/mopex//bin/mosaic.pl

Generate the FIF as a separate step (not required to be separate, but i am getting strange errors now UNLESS i make it a separate step) if you do this, though, turn off FIF generation for individual mosaics, since it will save you time (and, evidently, errors). edit the file cdf/justfif_gui.nl

mosaic.pl -n justfif_gui.nl

make the mosaic: mosaic.pl -n mosaic_long_sr.nl > mosaic8.log &


maybe useful to make one just of the first year to compare with the mosaics you made mon/tues:

grep 13375232 ch1_long.list.txt > ch1_long.list235.txt grep 13375232 ch2_long.list.txt > ch2_long.list235.txt grep 13375232 ch3_long.list.txt > ch3_long.list235.txt grep 13375232 ch4_long.list.txt > ch4_long.list235.txt grep 13375232 ch1_med.list.txt > ch1_med.list235.txt grep 13375232 ch2_med.list.txt > ch2_med.list235.txt grep 13375232 ch3_med.list.txt > ch3_med.list235.txt grep 13375232 ch4_med.list.txt > ch4_med.list235.txt grep 13375232 ch1_long_dmasks.list.txt > ch1_long_dmasks.list235.txt grep 13375232 ch2_long_dmasks.list.txt > ch2_long_dmasks.list235.txt grep 13375232 ch3_long_dmasks.list.txt > ch3_long_dmasks.list235.txt grep 13375232 ch4_long_dmasks.list.txt > ch4_long_dmasks.list235.txt grep 13375232 ch1_med_dmasks.list.txt > ch1_med_dmasks.list235.txt grep 13375232 ch2_med_dmasks.list.txt > ch2_med_dmasks.list235.txt grep 13375232 ch3_med_dmasks.list.txt > ch3_med_dmasks.list235.txt grep 13375232 ch4_med_dmasks.list.txt > ch4_med_dmasks.list235.txt grep 13375232 ch1_long_sigmas.list.txt > ch1_long_sigmas.list235.txt grep 13375232 ch2_long_sigmas.list.txt > ch2_long_sigmas.list235.txt grep 13375232 ch3_long_sigmas.list.txt > ch3_long_sigmas.list235.txt grep 13375232 ch4_long_sigmas.list.txt > ch4_long_sigmas.list235.txt grep 13375232 ch1_med_sigmas.list.txt > ch1_med_sigmas.list235.txt grep 13375232 ch2_med_sigmas.list.txt > ch2_med_sigmas.list235.txt grep 13375232 ch3_med_sigmas.list.txt > ch3_med_sigmas.list235.txt grep 13375232 ch4_med_sigmas.list.txt > ch4_med_sigmas.list235.txt

grep 13375232 ch1234.list.txt > ch1234.list235.txt


make a mosaic just of the 235 files. update a (new copy of a) namelist and invoke it. mosaic.pl -n mosaic235.nl > m.log

mosaic.pl -n justfif_gui.nl > fif.log mkdir p235pbcd3.6long mkdir p235pbcd3.6short mkdir p235pbcd4.5long mkdir p235pbcd4.5short mkdir p235pbcd5.8long mkdir p235pbcd5.8short mkdir p235pbcd8long mkdir p235pbcd8short cp FIF.tbl p235pbcd3.6long cp FIF.tbl p235pbcd3.5short cp FIF.tbl p235pbcd4.5long cp FIF.tbl p235pbcd4.5short cp FIF.tbl p235pbcd5.8long cp FIF.tbl p235pbcd5.8short cp FIF.tbl p235pbcd8long cp FIF.tbl p235pbcd8short

mosaic.pl -n mosaic235i1l.nl > m1.log mosaic.pl -n mosaic235i2l.nl > m2.log mosaic.pl -n mosaic235i3l.nl > m3.log mosaic.pl -n mosaic235i4l.nl > m4.log


mosaic.pl -n justfif_gui.nl > fif2.log mkdir pbcd3.6long mkdir pbcd3.6short mkdir pbcd4.5long mkdir pbcd4.5short mkdir pbcd5.8long mkdir pbcd5.8short mkdir pbcd8long mkdir pbcd8short cp FIF.tbl pbcd3.6long cp FIF.tbl pbcd3.6short cp FIF.tbl pbcd4.5long cp FIF.tbl pbcd4.5short cp FIF.tbl pbcd5.8long cp FIF.tbl pbcd5.8short cp FIF.tbl pbcd8long cp FIF.tbl pbcd8short

mosaic.pl -n mosaicfulli1l.nl > m1lfull.log mosaic.pl -n mosaicfulli1s.nl > m1sfull.log mosaic.pl -n mosaicfulli2l.nl > m2lfull.log mosaic.pl -n mosaicfulli2s.nl > m2sfull.log mosaic.pl -n mosaicfulli3l.nl > m3lfull.log mosaic.pl -n mosaicfulli3s.nl > m3sfull.log mosaic.pl -n mosaicfulli4l.nl > m4lfull.log mosaic.pl -n mosaicfulli4s.nl > m4sfull.log



grep 13375232 ch1_long.list > ch1_long.list235 grep 13375232 ch2_long.list > ch2_long.list235 grep 13375232 ch3_long.list > ch3_long.list235 grep 13375232 ch4_long.list > ch4_long.list235 grep 13375232 ch1_med.list > ch1_med.list235 grep 13375232 ch2_med.list > ch2_med.list235 grep 13375232 ch3_med.list > ch3_med.list235 grep 13375232 ch4_med.list > ch4_med.list235 grep 13375232 ch1_long_dmasks.list > ch1_long_dmasks.list235 grep 13375232 ch2_long_dmasks.list > ch2_long_dmasks.list235 grep 13375232 ch3_long_dmasks.list > ch3_long_dmasks.list235 grep 13375232 ch4_long_dmasks.list > ch4_long_dmasks.list235 grep 13375232 ch1_med_dmasks.list > ch1_med_dmasks.list235 grep 13375232 ch2_med_dmasks.list > ch2_med_dmasks.list235 grep 13375232 ch3_med_dmasks.list > ch3_med_dmasks.list235 grep 13375232 ch4_med_dmasks.list > ch4_med_dmasks.list235 grep 13375232 ch1_long_sigmas.list > ch1_long_sigmas.list235 grep 13375232 ch2_long_sigmas.list > ch2_long_sigmas.list235 grep 13375232 ch3_long_sigmas.list > ch3_long_sigmas.list235 grep 13375232 ch4_long_sigmas.list > ch4_long_sigmas.list235 grep 13375232 ch1_med_sigmas.list > ch1_med_sigmas.list235 grep 13375232 ch2_med_sigmas.list > ch2_med_sigmas.list235 grep 13375232 ch3_med_sigmas.list > ch3_med_sigmas.list235 grep 13375232 ch4_med_sigmas.list > ch4_med_sigmas.list235

cp *list235 COMEGETME cp p235pbcd3.6long/Combine/mosaic.fits COMEGETME/p235i1long.fits cp p235pbcd4.5long/Combine/mosaic.fits COMEGETME/p235i2long.fits cp p235pbcd5.8long/Combine/mosaic.fits COMEGETME/p235i3long.fits cp p235pbcd8long/Combine/mosaic.fits COMEGETME/p235i4long.fits cp p235pbcd3.6long/Combine/mosaic_cov.fits COMEGETME/p235i1long_cov.fits cp p235pbcd4.5long/Combine/mosaic_cov.fits COMEGETME/p235i2long_cov.fits cp p235pbcd5.8long/Combine/mosaic_cov.fits COMEGETME/p235i3long_cov.fits cp p235pbcd8long/Combine/mosaic_cov.fits COMEGETME/p235i4long_cov.fits cp p235pbcd3.6long/Combine/mosaic_unc.fits COMEGETME/p235i1long_unc.fits cp p235pbcd4.5long/Combine/mosaic_unc.fits COMEGETME/p235i2long_unc.fits cp p235pbcd5.8long/Combine/mosaic_unc.fits COMEGETME/p235i3long_unc.fits cp p235pbcd8long/Combine/mosaic_unc.fits COMEGETME/p235i4long_unc.fits



cp pbcd3.6long/Combine/mosaic.fits COMEGETME/i1long.fits cp pbcd4.5long/Combine/mosaic.fits COMEGETME/i2long.fits cp pbcd5.8long/Combine/mosaic.fits COMEGETME/i3long.fits cp pbcd8long/Combine/mosaic.fits COMEGETME/i4long.fits cp pbcd3.6long/Combine/mosaic_cov.fits COMEGETME/i1long_cov.fits cp pbcd4.5long/Combine/mosaic_cov.fits COMEGETME/i2long_cov.fits cp pbcd5.8long/Combine/mosaic_cov.fits COMEGETME/i3long_cov.fits cp pbcd8long/Combine/mosaic_cov.fits COMEGETME/i4long_cov.fits cp pbcd3.6long/Combine/mosaic_unc.fits COMEGETME/i1long_unc.fits cp pbcd4.5long/Combine/mosaic_unc.fits COMEGETME/i2long_unc.fits cp pbcd5.8long/Combine/mosaic_unc.fits COMEGETME/i3long_unc.fits cp pbcd8long/Combine/mosaic_unc.fits COMEGETME/i4long_unc.fits


cp pbcd3.6short/Combine/mosaic.fits COMEGETME/i1short.fits cp pbcd4.5short/Combine/mosaic.fits COMEGETME/i2short.fits cp pbcd5.8short/Combine/mosaic.fits COMEGETME/i3short.fits cp pbcd8short/Combine/mosaic.fits COMEGETME/i4short.fits cp pbcd3.6short/Combine/mosaic_cov.fits COMEGETME/i1short_cov.fits cp pbcd4.5short/Combine/mosaic_cov.fits COMEGETME/i2short_cov.fits cp pbcd5.8short/Combine/mosaic_cov.fits COMEGETME/i3short_cov.fits cp pbcd8short/Combine/mosaic_cov.fits COMEGETME/i4short_cov.fits cp pbcd3.6short/Combine/mosaic_unc.fits COMEGETME/i1short_unc.fits cp pbcd4.5short/Combine/mosaic_unc.fits COMEGETME/i2short_unc.fits cp pbcd5.8short/Combine/mosaic_unc.fits COMEGETME/i3short_unc.fits cp pbcd8short/Combine/mosaic_unc.fits COMEGETME/i4short_unc.fits


tim found the overlap correction had gone bad in ch3. redoing, using the SSC's overlap corrector. overlap.pl -n overlap_long_sr.nl > overlap.log

Retrieved from "https://vmcoolwiki.ipac.caltech.edu/index.php?title=I%27m_ready_to_advance_to_a_highly_technical_and_in-depth_discussion_on_processing_Spitzer_data.&oldid=3207"