hi, when trying to seperate SN1a, from truth summary table i got 445209 results(redshifts), and from diaobject i got 70976 results(fluxes), so how do i allocate redshift and flux/magnitude corresponding to each SN1a, as i want magnitude vs redshift of all the SN1a (not evolution of one SN1a), on a single plot.
and no SN1a id from truth summary table or matches truth table is present in any of the DiaObject, DiaSource, and ForcedSourceOnDiaObject tables , am i doing something wrong?
Hi, Akshita! I am the Community Engagement Team forum watcher for this week. I am just replying to let you know I am looking into your question. I note that the MatchesTruth table has a match_objectID column, which is the objectId of the object within the Object table, but I have found no equivalent column in the DiaObject , DiaSource , or ForcedSourceOnDiaObject tables that matches the DiaObject/DiaSource id back to either the Object table or the MatchesTruth table. Let me look further into this and consult others.
More info soon (I hope)!
OK, I have some additional info now.
It appears that, although the DP0.2 MatchesTruth table has a match_objectId column, which is the objectId of the object within the Object table, there is nothing corresponding for the DiaObject , DiaSource , or ForcedSourceOnDiaObject tables. For that matter, there is no spatial match between, say, the Objects table and the DiaObjects table – at least not for DP0.2 (but there should be in the future). Thus, to do what I think you want, one must do one’s own matching between the entries in the TruthSummary table and (say) the DiaObjects table. Unfortunately, speedy spatial matching can be a big pain. If you don’t mind an imperfect – but not unreasonable – quick-and-dirty method, I recommend using healpixels with a sufficiently high NSIDES value that you do no risk either too much contamination or too many missing matches. Along these lines, I wrote a simple Jupyter notebook, which you can find attached to this message. (For the time being, you can also find a copy in my scratch area of https://data.lsst.cloud – in /scratch/douglasleetucker/notebooks/TrueSNeIa.ipynb.
I hope this helps!
Sorry for the delay!
TrueSNeIa.ipynb (166.4 KB)
thank you for the detailed solution.
i tried the healpy code to get match both the tables , so for around 6000 SNe in diaobject , only 1000 out of those 6000 matched with the truth table Sne, i didn’t understand that?
i had used the diaobject tutorial code to get Sne between redshift range 0.1 to 0.3, but the matched 1000 Sne have redshifts way beyond the expected range?
i have attached the jupyter notebook containing the code and plots.
Untitled5.ipynb (86.1 KB)
Good questions! I have run your notebook (many thanks for supplying it!), but I don’t have an immediate answer. Of course, not all the transients found by the diaobject notebook will be SNe: some will be variable stars. According to Sectoin 5.3 of the DC2 Release Paper, about 10% of the simulated stars were made variable. But it still seems surprising that only 1000 of the diaObjects had matches with the SNe in the Truth Summary table. Maybe you can try some spot checks of the light curves of some of the matched and unmatched diaObjects, something like what was done for the first 20 matches in Section 3.4 of tutorial notebook 07a_DiaObject_Samples.ipynb ?
I am also not sure why the 07a_DiaObject_Samples.ipynb notebook seems to be netting so many z>0.3 SNe Ia. Could they just be mismatches (i.e., the healpix matching method is yielding too many mismatched objects)? Could there be an issue with the SN-finding algorithm in 07a_DiaObject_Samples.ipynb? Are other types of SNe (besides SNe Ia) also being represented in the simulated data? But I don’t think the DC2 Release Paper mentioned other SN types?
Might need to think about this more…
Re. “Could there be an issue with the SN-finding algorithm in 07a_DiaObject_Samples.ipynb?”
That is not an SN-finding algorithm, not by any means. The notebook 07a contains a very rough, back-of-the-envelope way to identify a bunch of DiaObjects with parameters that suggest they are SNIa-like. The resulting sample is not scientifically useful (not pure, not complete) – it is just a few simple cuts that will get you a bunch of DiaObjects for which we can demonstrate how to plot light curves using the DiaSource table. Real SN-finding algorithms are photometric classifiers that fit the light curves with templates, rigorously, and provide probabilistic classifications. The text in Section 1 of the notebook describes this.
Re. “Are other types of SNe (besides SNe Ia) also being represented in the simulated data?”
The DC2 simulation only included Type Ia supernovae. But there are also variable stars, and the simple cuts being made on the DiaObject light curve summary parameters in notebook 07a could permit DiaObjects that are variable stars into the sample.
Hi, Akshita! I am just writing down some (I hope) final thoughts; so we can declare this issue resolved.
I see 2 related but opposite concerns:
For both cases, it is necessary to match the catalogs from the DiaObject/DiaSource tables with catalogs from the DC2 TruthSummary table. I showed some of my analysis in previous replies above. Since those replies, I have created a cleaned-up notebook (dia_match_truth.ipynb), which I have pushed to the DP0.2 delegate contributions GitHub repository .
For the first concern ("[A]re the candidates found in the DP0.2 DiaObject and DiaSource tables using the back-of-the-envelope method identify the True SNe Ia from the DC2 Truth Summary table?"), it was found that 313 of the 331 DiaObject/DiaSource candidate SNe Ia identified by this method are indeed well matched (to much better than a 1-arcsec match radius) with SNe Ia from the TruthSummary table. This was done by finding for each DiaOIbject/DiaSource candidate SN Ia its best match in the TruthSummary table; to do this quickly, the astropy match_coordinates_sky function was used. Here is the histogram of separations for 331 DiaObject/DiaSource candidate SNe Ia:
There are clearly some bad matches. Let’s look more closely; further, let’s plot separately the histograms for DiaOIbject/DiaSource candidate SNe Ia whose best match was a TruthSummary variable star and for those whose best match is a TruthSummary SN Ia:
OK, the DiaOIbject/DiaSource candidate SNe Ia whose best match was a TruthSummary variable star are all over the place. Clearly these are overwhelmingly mis-matches. Those whose best match was a TruthSummary SN Ia are pretty heavily clustered near 0 arcserc. One can make ever more zoomed in histogram plots, but it soon becomes apparent that a match radius of 1.0 arcsec is a pretty reasonable value for the cutoff for a “good match.” Here is the zoomed in histogram plots out to a match radius of 1.0 arcsec:
Looks pretty good! There are three good matches to TruthSummary table variable stars, with the overwhelming majority of the good matches being to TruthSummary table SNe Ia. If we count the number of these good matches to TruthTable SNe Ia, we find 313, out of a total number of 331 DiaObject/DiaSource candidate SNe Ia. In other words, about 95% of the candidate SNe Ia identified from the back-of-the-envelope method described DP0.2 Tutorial Notebook 7a turned out to be TruthSummary SNe Ia.
One final plot for this first concern is a plot of the redshifts of these 313 verified candidate SNe Ia:
As expected from the NB07a methodology, these are all relatively low redshift SNe Ia.
Now for the second concern ("[D]o the true SNe Ia and variable stars in the DC2 Truth Summary have matches in the DP0.2 DiaObject/DiaSource tables?"). To address this concern, the DP02. delegate contribution notebook dia_match_truth.ipynb downloads all 11,325 TruthSummary entries classified as variables within a ~0.5 sq deg patch (61.0<=RA<=62.0, -33.0<=DEC<=-32.), downloads all 100,643 objects within that same sky area from the DiaObject table, and then finds the best match for each entry the downloaded TruthSummary catalog within the downloaded DiaObject catalog. The resulting matched catalog has 11,325 (since the search was for the best match for each TruthSummary entry).
Since the TruthSummary table is much deeper than the DiaObject catalog, we expect a lot of poor matches. As before, we will denote a good match as one with a separation of less than 1.0 arcsec. Let’s first look at teh TruthSummary variable stars. We will plot the r-band magnitudes from the TruthMatch table for all matches (blue) and for just good matches (orange):
Note that the mag_r of good matches peaks at about mag_r=21, whereas the distribution of mag_r's for all matches continues up to around mag_r=28.
The TruthSummary table unfortunately does not record the values of mag_r for SNe Ia. The Truth apparent magnitudes for the SNe can be found by other ways, but, to keep to just the TruthSummary table, we can plot the histogram of redshifts, instead. Here, we plot the redhifts from the TruthMatch table for all matches (blue) and for just good matches (orange):
The plot for the SNe Ia is not as definitive as for the variable stars, but it is still suggestive. The good matches (separation radius less than 1.0 arcsec) peak at a lower redshift than do the full sample of matches, presumably because the good matches have brighter apparent magnitudes.
To get another look at the True SNe Ia, we can consult the additional Truth table, truth_sn_summary_v1-0-0.parquet, provided on the RSP for DP0 Delegate use by Jim Chiang. We access this table on the DP0 RSP in the following manner:
pfnm_sn_sum = '/project/jchiang/Run2.2i/truth/SNe/truth_sn_summary_v1-0-0.parquet'
result_sn_sum = pd.read_parquet(pfnm_sn_sum)
We can then plot not only the redshifts of all 405293 SNe Ia in the DESC DC2 catalog, we can also plot the apparent B-band magnitudes (very roughly comparable to the r_mag mags):
Note the B-band magnitudes (mB) go well below the roughly mag_r = 21 seen for the good matches for TruthSummary variable stars plotted earlier.
Finally, we can also plot the apparent B-magnitude vs. redshift for all the True SNe Ia in truth_sn_summary_v1-0-0.parquet:
The dramatically reduced number density of SNe Ia beyond a redshift of about z=0.95 is feature of the truth catalogs used to generate the DESC DC2, the generation of which is described in LSST DESC et al. (2021).
