C-CWEL Source Matching Work

Big Picture Introduction -- an analogy that might be too fanciful?

When westerners first discovered the Americas, they had largely set out with the goal of finding gold or other treasure. They were coming at the new continent from the perspective of someone in a boat, with minimal information about what the landforms really were, except for what they could see with their own eyes. Their maps look strange to those of us used to seeing images of these landforms from space, but we have a whole lot more information now than they did then.

The first thing that these early Western explorers were able to attempt to map was the coasts, because that's what they had the most information about...and the most immediate need to know. They needed to know where coral reefs were that might damage their ships, and where the big rivers emptied into the sea so that they could take on more fresh water. They also could learn about more land faster when boating up the rivers rather than walking.

As more and more boats explored the coasts, the maps got better, but they still seem distorted compared to the landforms we know today. In some of these early maps of the Americas, when Europe and Africa were included, even the African coast on the Mediterranean side doesn't look all that realistic, compared to what we know now.

As the westerners pushed further and further into the land (in the Americas or for that matter Africa) from the coasts, their knowledge deepened about what the continent actually looked like, aided by improvements in technology (such as more accurate ways of measuring longitude). Their knowledge of the land started in clumps around the rivers, again because that's what they needed that best enabled them to travel the furthest. But, their knowledge expanded as fast as they could expand. And their goals changed too -- certainly some were still looking for treasure (or freedom from persecution, religious or otherwise), but more in the earliest years were just trying to survive (here I'm thinking of Jamestown or Roanoke). They explored to find more food to eat (critters or plants).

The Native Americans, of course, had a perfectly good understanding of what their land looked like, but even so, most likely, one tribe only knew the land near them -- my guess is that the Powhatan tribe (in VA) had no idea whatsoever what the Sioux tribe's lands looked like, even if trade routes were such that items could move from the Dakotas to Virginia. But the Native Americans were observing the land in a different way, having lived there for a while and having their own methods of exploration. Once westerners realized that they could learn from the Native Americans (here I'm thinking of Lewis and Clark), their knowledge could expand even faster once they paid attention to what was already known.

There were, at nearly every stage of these early maps, regions that were sort of hazy and unexplored, e.g., "here be dragons". Someone might have a vague idea of what was there, but maybe only on the edges; no one (at least no one known to the map maker) had any detailed knowledge of what was there.

SO NOW.... here is a perhaps tortured analogy... We would like to go exploring in a particular region, making note of where the big landforms are, and we have a specific goal of finding edible animals and plants to support our efforts in further explorations. Some people have spent some time exploring parts of this region before. It will make our knowledge expand faster if we pay attention to what is already known before setting off on our own journey. Others have mapped different parts of the region using different methods of exploration before -- some on foot and some on horseback, and still others in boats. Some people just ran through this region identifying the big trees and big rocks. Some people wrote down what they learned in Algonquin (what the Powhatan spoke) and we need to translate it before it makes sense in the language we speak. Some people cared a lot about their tiny camp next to the river and they know that region really well, but beyond the borders of their camp, "here be dragons." There is some information about the area around the camp that we can obtain from other people and from exploring on our own, but we can also come back to this very well-known region and check what we think we know about the rest of the area by testing it on the well-known region. The people who know the region right next to the river really well also know that this animal or that plant is good food and won't make us sick. When we go exploring out further from the camp, if we find a critter or a plant that we think is the same as the stuff that the people next to the river know is ok to eat, we can bring it back to that camp to compare it and see if it is, in fact, the same or something new. We can also, among the animals and plants we find, put them in groups of apparently similar things -- these are all 4-footed furry critters, and those have feathers and wings.

Explicitly drawing lines between that analogy and reality:

analogy	reality translation
We would like to go exploring in a particular region, making note of where the big landforms are, and we have a specific goal of finding edible animals and plants to support our efforts in further explorations.	We have a goal of exploring a region on the sky, specifically looking for young stars.
Some people have spent some time exploring parts of this region before. It will make our knowledge expand faster if we pay attention to what is already known before setting off on our own journey.	We need to read and understand the literature.
Other people have mapped different parts of the region using different methods of exploration before -- some on foot and some on horseback, and still others in boats.	Other people have used different wavelengths to explore this region before.
Some people just ran through this region identifying the big trees and big rocks.	Some people just identified the bright young stars, or the things bright in the wavelengths they were using.
Some people wrote what they learned down in Algonquin (what the Powhatan spoke) and we need to translate it before it makes sense in the language we speak.	Some people wrote down poorly constrained coordinates in epoch B1950 coordinates and we need to translate it to accurate J2000 coordinates. (Actually, does not seem to be the case for BRC 38! Lucky us! We still need to match objects from the literature, though.)
Some people cared a lot about a tiny patch next to the river and know that region really well, but beyond the borders of their camp, "here be dragons."	The Choudhury et al. group spent time sorting out the 5'x5' patch with 4-band IRAC coverage, but not the serendipitous data; we will have to use their results but also the rest of the data where possible.
There is some information about the area around the camp that we can obtain from other people and from exploring on our own...	We can comb the literature and use the 2MASS+WISE data to help guide us in exploring the region.
...but we can come back to this very well-known region and check what we think we know about the rest of the area by testing it on the well-known region.	We can use WISE to identify things with YSO-like colors in the region. Did we rediscover the YSOs that other people found, the Choudhury group using Spitzer data? If not, why not? Do the objects with YSO-like colors look like point sources in 2MASS or do they look like galaxies?
The people who know the region right next to the river really well also know that this critter or that plant is good food and won't make us sick. When we go exploring out further from the camp, if we find a critter or a plant that we think is the same as the stuff that the people next to the river know is ok to eat, we can bring it back to that camp to compare it and see if it is, in fact, the same or something new.	We have a region of space that is very well studied with Spitzer, and some limited serendipitous Spitzer data nearby. We will use WISE over the whole region to find things that we think might be YSOs. We can check our hunch that some of the objects are in fact YSOs by comparing what we get to the Spitzer data where we can, and including that data in our analysis.
We can also, among the animals and plants we find, put them in groups of apparently similar things -- these are all 4-footed furry critters, and those have feathers and wings.	Among the objects we find, we can put them in groups based on the shape of their SED, from 'really embedded' (class 0-I) down to 'not much of an IR excess' (class II-III).

We will not be able to get a comprehensive be-all-end-all understanding of the region (e.g., in the analogy, we will not go straight from Columbus or even Lewis and Clark to weather satellite views of the continent with a GPS in our car as we drive). We can, however, do the best that we can with the information we have, by learning from those who have gone before, learning as we go, and making intelligent guesses about what we don't know.

More specific introduction to source matching from the literature

Several people have done prior studies in BRC 38 before, but they have NOT found all the young stars! You tried to make sure we found all of these articles in the context of the proposal. Certainly investigator A working in BRC 38 in year X saw some of the same sources as investigator B working in that same region in year Y, as did investigator C in year Z. Now we actually have to do the work of figuring out which specific sources are which in all the papers - are the sources called out in paper 1 the same or different sources as paper 2?, etc., until all the papers are exhausted, and we have a complete catalog of all the previously studied sources in the region.

The thing that makes this somewhat complicated is that, even though everyone is reporting in RA and Dec, not everyone is using exactly the same system (though they are all J2000, there are some variations within that; WISE, 2MASS, Spitzer, and Chandra should all be well-matched), and not everyone has the same coordinate accuracies (some are working off of photographic plates, and some are working off large-format CCDs; Ogura+ 2002 is the toughest, but CWAYS already did this for you). And, what does it mean to have "the same" coordinates -- is within an arcsecond ok? 5 arcseconds? an arcminute? This is where it gets tricky, and where you have to apply your brain! Spitzer, WISE, and 2MASS are all using exactly the same, high-accuracy coordinate system -- it's all tied to 2MASS's J2000 coordinates -- but even then the position of the same object will not be EXACTLY the same in each image, in each catalog, because there is a limit to the precision with which we can identify the coordinates. Where possible, we need to update the old coordinates by comparing what the old papers say to the 2MASS data. Then we need to fold in the objects with newer coordinates into our collection of sources.

Part of the challenge here is bookkeeping -- writing down coordinates correctly, keeping track of which sources are which, and getting the correct data matched to the correct source.

In 2011, I thought this would be a relatively simple project that could be done before the summer visit. However, it turned out not to be the case. So, I pushed the 2012 team really hard, and they got through a tremendous amount of work prior to their visit. BRC 27 was far more of a complex problem than BRC 38, so it should be easier for us. I've pulled out and updated all of my best(?) explanations and descriptions here. IF IT DOESN'T MAKE SENSE, PLEASE ASK QUESTIONS. If this is done wrong, or only halfway done, it will make for a LOT of pain downstream. Trust me.

Venn Diagrams and Bookeeping

One of the difficulties we will have during this project is keeping all the source lists straight. It happens every year, and I don't know how to make it easier (in part because I am not sure exactly why it's so confusing), except for warning you that it will happen! Here is a Venn diagram explaining, roughly, the various source lists we will have before we are done, at minimum. This Venn diagram is meant to be a "big picture" sort of thing; this page on the source matching is meant to address just, well, the previously identified sources.

The source lists include:

• All "bright enough" sources seen in the WISE maps (a conceptual list only)
• Sources in the WISE catalog of photometry (to which we will add photometry from 2MASS, Haleakala, and Spitzer in the places where we have that data)
• Sources in this general direction studied by anyone else, ever (the majority of those reported are also YSOs, but not all of them) -- this is the list we are trying to assemble here.

Out of those sets, our ultimate scientific goals mean that we are striving to identify:

• YSO candidates we select from IR excess
• YSOs that others identify that do not appear to have an IR excess.

The Venn diagram is even trying to correctly represent the relative sizes of the circles in that "all bright enough sources" ought to be darn close to "sources in the catalog" and that there will be some "sources in this general direction..." not in the regions we care about, and some of those sources that do not have IR excesses.

NOW we are going to work on the list of "Sources in this general direction studied by anyone else, ever".

For this diagram, I tried to spatially represent the concepts behind what we're doing now (on this page), but I admit the circles are not as carefully constructed/laid out as the first one!

Each of the 7 papers studying things in the region of BRC38 (Ogura et al. 2002, Getman et al. 2007, Beltran et al. 2009, Choudhury et al. 2010, Chauhan et al. 2009, Barentsen et al. 2011, Nakano et al. 2012) looked in the direction of BRC38. Surely, then, they saw some of the same sources as each other, and as what we are seeing. For example, the Getman survey with Chandra and the Choudhury Spitzer data covered comparable regions but not exactly the same. Ogura saw some of the same sources that Getman did, but not all of them -- even within the same area, they did not see the same sources, because one survey was X-ray driven, and one was Halpha driven. They *will* see different sources, not only because they're looking at different wavelengths, but also because each survey is not infinitely deep -- the sensitivity of the surveys is limited, and as such will not see every source in this direction. (For example, Beltran et al. is working in JHK but deeper than 2MASS.) Same for each other pair of papers, and our survey.

The Goal

The goal here is to construct a list that is as clean as possible for each of the objects that these other folks studied, identifying which objects are truly the same between surveys, and identifying which of these objects are ones that those authors thought were actually young stars (as opposed to, e.g., background giants). We also want to carry along each of the relevant bits of information that these other authors provided -- the object is a lot easier to identify as clearly a young object or a contaminant if there is optical data, so if the other authors reported any optical measurements, we should keep track of those and tie them to the correct object in our analysis. We should also make note of any spectral types or other relevant information. The aim of this part of the project is thus:

• Which objects from paper x are also seen in paper y?

and then, the next step we will take is

• Which of these objects are seen in the WISE data? (for those that are missing, why are they missing?)

The Challenges

This would be an easy task if:

• everyone provided their original images, either as a figure or as a fits file
• everyone worked in the same coordinate system, by which i mean not just "J2000" vs "B1950" but "J2000 tied to 2MASS" as opposed to "J2000 tied to the pulsars seen by NRAO" or "J2000 as calibrated as best I can based on the HST Guide Stars I happen to see in my image".
• the objects were all greater than 5 arcseconds apart from each other on the sky, such that each source that is detected was cleanly and uniquely detected in each survey.
• and, of course, if we were guaranteed a match between surveys.

Working backwards up that list...

We've already talked above about how we are not guaranteed a match between surveys, because stars are different brightnesses at different bands, and because the surveys have limited sensitivity.

There are plenty of sources that are very close together. Even among just the YSO candidates, some are very close to each other, closer than 5 arcseconds.

If we had fabulous coordinates for everything, we could let the computer match them all up and not worry about it. It turns out that for most of these literature surveys in BRC 38, they are pretty well-matched, and the computer's automatic merging gets us a long way. But it's always good to remember that just because the computer says it doesn't mean it's right.

If we had images, we could line them up by eye and identify the same objects in each frame. I don't necessarily mean "line them up in ds9" (which would be the ideal case). But also, you can identify the objects simply by comparison between images they publish and images to which you have access (IRAC, 2MASS, POSS). This is what we are going to have to retreat to, in the tough cases.

Here are some notes on the 7 BRC 38 papers, in chronological order:

• Ogura et al. 2002 - COORD NEED UPDATING, but CWAYS did this for you. Probably good to at least take a quick look at these, just in case. there are finding charts.
• Getman et al. 2007 - x-rays. got all tables. includes all relevant results from Nisini et al. coord ok because they are from Chandra. Good to remember that Chandra's PSF changes the further you go off-axis - objects on the edge of the Chandra field are much poorer resolution than near the center.
• Beltran et al. 2009 - NIR. got all tables. coord should be ok.
• Choudhury et al. 2010 - IRAC, MIPS, optical. got all tables. coord should be ok.
• Chauhan et al. 2009 - BVI, NIR, IRAC. coord ok. though some inconsistencies. NEED TO CHECK FOR BRC 38. got all tables.
• Barentsen et al. 2011 - optical (r,i,Ha). coord ok. got all tables. Would be nice to get table of everything in region we care about, not just that which they are reporting on. Need to go to main IPHAS archive.
• Nakano et al. 2012 - optical (r,i,Ha); IR (AKARI). coord ok. got all tables, i think. Might as well go and get table from AKARI of everything in the region, though I think WISE will be more powerful.

The mechanics of what we need to do

Links of interest:

• http://irsa.ipac.caltech.edu/applications/Gator/ - Gator
• http://irsa.ipac.caltech.edu/applications/finderchart/ - Finder Chart

For each of these papers, we need a machine-readable (read as "plain text file that the computer can parse into individual numbers rather than images of numbers or gobbledegook from microsoft") version of the relevant data tables. This is either:

• obtained from the journal itself, in which case the data table is typically much longer than we need
• obtained by typing in the coordinates of the objects in our fields from these older papers and then getting updated coordinates.

STOPPED UPDATING HERE

Here is a video I made for my 2011 team on getting updated coordinates. This approach SHOULD work in MOST cases but did not work in every case; more on this momentarily. What we need to do is :

• type the 1950 coordinates into the 2mass point source archive, making sure that it knows that it is 1950 coordinates; or type in the 2000 coordinates (and it will assume that they are 2000 coordinates).
• look at what comes back, and take the closest *bright* object.

Then we will have 5 lists of UPDATED, HIGH QUALITY coordinates, one per paper, and we can let the computer run through the list, finding the matches between papers. We then can generate one file that purports to have one line per literature object, with all the relevant data on that line. The difficulty comes in that inevitably, a few sources during this process end up tied to the same object, or identified in other ways as duplicates or incorrect matches.

The approach above to get updated coordinates for targets works ON THE ASSUMPTION THAT THERE ARE NOT "TOO MANY" SOURCES NEARBY, AND THAT THE PRIOR SURVEYS, BEING SHALLOW, ARE MOST LIKELY TO MATCH TO THE BRIGHT 2MASS SOURCE. As I say, this should work in MOST cases but not ALL of them.

If the original authors are reporting more than one source within one paper, we should not consolidate them into one unless we are REALLY sure that the original authors were wrong. (This does happen; see my North American Nebula paper for examples.) Within each paper, this is what I would do:

• Go do the 2MASS archive search (Go here: http://irsa.ipac.caltech.edu/applications/Gator/ -- pick 2MASS then on the next page pick 2mass point source catalog (PSC), then use that search page, making sure to specify which coordinate system in which you are working. Use the coordinates from the original paper to avoid transcription errors, as opposed to any subsequent notes or xls files. Be VERY careful to copy the coordinates exactly; that's the most common problem.)
• Look at the sources returned. Is there just one within 5-10 arcsec? that's probably your match; go on to the next source. If there is more than one, is there just one BRIGHT one within 5-10 arcsec? Is there another source from that same paper within 5-10 arcsec? If the answer to either of those questions is yes, go on to the next step.
• Go get the images in another window. Compare the images (if provided) from the original papers. We can use Finder Chart (http://irsa.ipac.caltech.edu/applications/FinderChart/) to get DSS and 2MASS images of the region where there is a confusing match. Compare this to the images provided in the papers. Identify which object is the one(s) in the paper.
• With that information, then go back to your PSC, and then use the 2MASS point source catalog to find the actual high-precision coordinates of that specific dot you have identified in the images as the match to the literature object.

UPDATE 31 May 12: while the OUTPUT of this process has to be in plain ASCII text, you can work in Excel or Google Docs and spit it out as plain text as the last step. Here is a video on getting plain text (including IPAC table format) files into Excel.

Here is a list of all the papers we have assembled as part of the lit review, as well as a list of their coordinate list status. First priority items (e.g. those for you to tackle first in May/June) are in bold in the last column:

paper	notes	coordinate list status	merging status and pending items
Getman et al 2007	X-rays. coordinates likely just fine.	got all tables from journal. LMR has merged t1,t2,t3 together.	need to merge it to rest of catalogs. does it really have all of Nisini et al. (2001) already in t3? - YES, Getman Table 3 contains K, 2MASS and Spitzer data for all 20 Embedded Young objects from Nisini Table 2 Getman Tables 1 and 2 contain X-Ray and MIR data for Nisini stars 2, 3, 10, 11 --Peggy Piper 20:56, 10 June 2012 (PDT) NOW MERGED INTO BIG ARRAY-7/3/12 --LMR NB: all sources detected are tabulated; the globule sources are the only ones clearly asserted as likely young, so i've just tagged those as YSOs.
Beltran et al. 2009	NIR. coordinates likely just fine.	got all tables from journal. (link)	individual tables have been merged together. merged into big array --7/3/12 --LMR. NB: all sources detected are tabulated! text talks a lot about how there are YSOs there, but THERE IS NO TABLE OF JUST YSOs. Sigh. Merged photometry in but cannot of course tag the ones they think are young.
Choudhury et al. 2010	MIR. coordinates likely just fine.	got all tables from journal. (link)	tables from paper have been merged together. merged into big array --7/3/12 --LMR. NB: not entirely clear if all objects are being asserted as YSOs or not. Includes stuff in later table with a spectral type for an object that I can't find in Ogura (Ogura-BRC38-17). Merged photometry in but cannot of course tag the ones they think are young.
Chauhan et al. 2009	NIR, MIR (IRAC only). 27 and 38. coordinates ... odd. look at carefully.	got all tables from journal. (link)	need to merge their tables together (will take some care! - do this if the rest of the bold stuff here is done and you're chomping at the bit for more), then merge that to rest of catalogs. Chauhan table 6 has been merged with table 3. Sources within .13 seconds for right ascension and 2 arcseconds for declination were considered matches (2 asec = ~.13 sec). All 2MASS table 3 to ra/dec table 6 were much smaller than this. Three 2MASS table 3 to ra/dec table 3 had slightly larger mismatch (0.15 and 0.17 sec mismatch). Also checked 2MASS names against Ogura list and all are correct. Origin/meaning of non matching? Megeath or Allen et al. 2004? Media:ChauhanTableMerged1.xls Media:ChauhanTableMerged1.txt NOW MERGED INTO BIG ARRAY-7/3/12 --LMR. NB: just YSOs tabulated, though for 27 there are indeed sources that appear in the IRAC table that do not appear in the JHK table.
Barentsen et al. 2011	iprime,rprime, halpha.	got all tables from journal.	LMR has already merged it to rest of catalogs. -7/3/12 --LMR. NB: just YSOs tabulated
Nakano et al. 2012	iprime, halpha, some AKARI.	got all tables from journal. (link)	LMR has already merged it to rest of catalogs. -7/3/12 --LMR. NB: just YSOs tabulated
Ogura et al. 2002	there are finding charts!!!	38 done at least first pass by cways. link for data files as downloaded from journal	COORD NEED **UPDATING** need to obtain list of objects as published, find match in 2mass, assemble list of matches betw this and 2mass. you probably will need the finding charts in the paper to sort out the matches. Ogura list updated 20120625 (sorry, ignore the date in the name) text File:CWAYSBRC273438Oguraids 20120614.txt excelFile:CWAYSBRC273438Oguraids 20120614.xls coordinates entered and matched, Peggy's groups cross checked within group, with Rebull 2012,and with Jackie's group. waiting Lauren and Jackie's groups cross check. MERGED INTO BIG ARRAY-7/3/12 --LMR. NB: just YSOs tabulated

Last (next) steps

Once we have updated high-precision lists of coordinates from each paper, we can merge them together, and come up with a new-and-improved list of all the previously identified objects in this region in one place, one line per object with all the relevant information, in a plain text file.

Then we will compare this list to our WISE catalog. There will be some with IR excesses, and some without IR excesses. I expect that we will probably detect them all, but there might be some we do not detect.

Then we will also compare this list to where there is Spitzer data, and obtain Spitzer data for those sources where we can.

Then we will also compare this list to the optical data, where possible. There WILL be some we WILL NOT DETECT in the optical data.