Survey Strategy and Solar System Science

Tags: #<Tag:0x00007fb37f5b7978>

Some thoughts and metrics related to the survey strategy and solar system science.

Context: general solar system science metrics include discovery metrics (with various discovery criteria) and characterization metrics (ie. how many colors for objects can we measure, and can we determine a light curve or even shape measurement from the lightcurve). The most important metric is discovery; if we can’t find the objects, then we can’t do anything else with them (or follow them up at other observatories). Secondary metrics are all of the characterization metrics. There are multiple populations of small bodies throughout the solar system, which move with different apparent velocities and cover varying amount of the sky (generally centered on the ecliptic, but some more widely distributed than others).
As such, survey strategy can have different effects on different populations, even when considering just one metric. Secondary metrics can also be important, especially if the primary discovery metric isn’t a strong differentiator between strategies.

With that, we can look at various aspects of the survey strategy.

First up - intranight strategy
The relevant simulations here include visits = 2x15s snaps or visits = 1x30s snap, whether we have single visits per night, double visits, or triples (for some fraction of each night).
The relevant simulations (due to an oversight on the project side) include runs from FBS 1.4 and FBS 1.5 … the metric outputs for each run are normalized by the relevant baseline_vX_10yrs output.
The list of runs included for consideration are:

'baseline_2snaps_v1.5_10yrs', 'baseline_2snapsv1.4_10yrs',
 'baseline_v1.5_10yrs', 'baseline_v1.4_10yrs',
 'third_obs_pt15v1.5_10yrs', 'third_obs_pt15v1.4_10yrs',
 'third_obs_pt30v1.5_10yrs', 'third_obs_pt30v1.4_10yrs',
 'third_obs_pt45v1.5_10yrs', 'third_obs_pt45v1.4_10yrs',
 'third_obs_pt60v1.5_10yrs', 'third_obs_pt60v1.4_10yrs',
 'third_obs_pt90v1.5_10yrs',  'third_obs_pt90v1.4_10yrs',
 'third_obs_pt120v1.5_10yrs',  'third_obs_pt120v1.4_10yrs']

The punchline is the following plot, showing the discovery completeness at bright and 50% values, for various populations, across all of these runs.

It may be surprising that the ‘no pairs’ simulation results in any objects being found at all - that we do find any means that fields have not-insignificant overlaps within a night, that sometimes objects move between fields, and that sometimes we take pairs of visits in a night anyway.
This can be seen in the intranight histogram as well … no pairs and standard baseline histograms below, which show that the number of repeat visits within a night drops significantly for the no-pairs simulation, but not to 0.
It should be noted that the existence of overlaps of visits in the ‘no-pairs’ state is fragile … if the scheduler algorithm was changed to reduce overlaps, then these would likely disappear.

This shows that the favored strategy would be taking pairs of visits, in the same filter within the night (about a 3% improvement in the discovery fraction of NEO and PHAs). Strongly disfavored strategies would be single visits per night (an over 10% hit for main belt asteroids), due to the requirement of two observations per night for MOPS linking. The 2x15s strategy (2snaps) vs. 1s30s strategy (1snap per visit) involves about a 5% decrease in completeness for Trojan asteroids, with varying decreases among other populations – this is generally linked to the overall fewer number of visits possible when doing 2snaps vs. 1. Observing with a triple visit near the end of the night is not a significant issue until the amount of time devoted to triples is large (the pt90 and pt120 runs), at which point it can result in about 4% fewer NEOs being discovered. This is most likely due to the decreased amount of sky observed in each night.

It’s worth noting that observing each pair in the same filter, while it has a benefit for discovery, has a small cost for characterization, at least when looking at the number of objects which can be measured in 4 colors (primarily for Jovian Trojans).

Next: footprint

The footprint generally means “what does the survey footprint look like”, and in particular, tends to refer to what does the WFD or high-numbers-of-visits portion of the footprint look like. There can be variations in the filter distribution, but primarily we’ve investigated what part of the sky the WFD (or which part of the sky tends to get 500-1200 visits) portion of the survey looks like.

The relevant footprint runs can all be pulled from the FBS 1.5 release. The sets of runs with relevant information include the “footprint” series, the “bulge” series (because these include heavy coverage through the galactic plane in various ways), and the “filter_dist” series (because these include variations in filter coverage, but also - importantly - they feature a restricted survey area that includes no visits in the North Ecliptic Spur). The specific runs are:


To visualize the difference in these footprints, here are plots of the distribution of total visits for each of the above runs (using the same scale for each simulation):

Limited WFD footprint types …

  • all of the filterdist runs have this same map, which covers a declination region similar to the standard survey WFD, with no variation across the galactic plane, and no NES or SCP coverage. The variations on filterdist have different filter distributions.

Standard survey variations

Extended N/S WFD coverage

  • footprint_big_sky_dust uses an extended N/S region for the WFD (going about 10 degrees further north and south), with the galactic plane boundaries delinated by dust extinction. There is extended coverage with fewer visits to the north, but no SCP or galactic plane coverage.
  • footprint_bigsky uses a similar extended N/S region for the WFD, with the galactic plane boundaries defined by a galactic latitude (=20deg) only. footprint_bigsky_nouiy is similar, but without u, i or y bands. Like in the footprint_bigsky_dust, there is extended coverage with fewer visits to the north, but no SCP or galactic plane coverage.
  • bulges_bs and bulges_cadence_bs use the same footprint map, although with different timing for the galactic plane coverage. Note that the WFD is extended N/S beyond the ‘standard’ footprint, so this is like the ‘big sky’ footprint, but with bulge, northern, and SCP coverage.
  • bulges_bulge_wfd and bulges_cadence_bulge_wfd, and bulges_i_heavy and bulges_i_heavy_cadence all use this survey footprint map, although with different timings and filter distributions. Note that it is similar to the big sky basis, but with galactic plane and SCP coverage, and these variations also have a band of WFD-level coverage through the galactic bulge.
  • footprint_bigwfd is sort of a hybrid between the big sky and the standard survey; it has an extended WFD region, going further north than in the big sky but not as far south. This footprint includes SCP and GP coverage, with a small extension for the NES.
  • footprint_newA includes an extended N/S WFD (big sky style), with galactic plane region defined by galactic latitutde (l=20). The bulge direction is covered to slightly fewer visits than the full WFD; the anti-center is covered to normal WFD depths. The SCP and NES (and extended northern coverage) is included, to a more limited depth. The WFD region only achieves about 771 visits per pointing.
  • footprint_newB is similar to footprint_newA, but the galactic anti-center is covered to much fewer visits in an attempt to redistribute these into WFD and add to NES. The WFD region here achieves a median of 842 visits per pointing.

In terms of solar system object discoveries, we might expect the footprint to be a significant contributor. This is true particularly for TNOs, as they move very slowly across the sky. NEOs cover large amounts of the sky, thus are less sensitive to footprints. The MBAs are similar to the NEOs, although more constrained toward the ecliptic plane. The Trojan asteroids are also sensitive to footprint, as they are concentrated into limited sky locations due to their resonant nature.

We can again look at the discovery completeness at bright and 50% values, for various populations, across these footprint-related runs: (sorted by completeness for bright TNOs)

It’s clear that the filter_dist* runs, where the NES was not covered, have a severe impact on TNO discovery (-30%). The other populations are not impacted as strongly, although Trojans suffer an almost 20% drop in some of these filter_dist runs (the difference is due to the exact filter distribution — filter_dist 1 and 4 are ‘uniform filter distribution’ and ‘heavy u’ distribution, respectively … the standard survey puts only about 30% of total visits per pointing into u band). None of these filter_dist runs improve solar system object discovery, even though the WFD region will have more visits within its footprint.

Removing the filter_dist runs and replotting to look at more subtle differences in the other runs:

In these runs, the trend for TNOs follows from the big_sky runs (which did not cover the galactic plane region), then the bluer footprint (which is unfavorable due to filter distribution), then through the relatively ‘standard’ survey footprint (footprint_add_mag_clouds, baseline, footprint_no_gp_north, footprint_standard_goals being very similar), picks up at footprint_bigwfd and then increases further with the bulges_* series, which have a ‘big_sky’ WFD footprint with galactic plane coverage. The overall winner here is the newB footprint, which did slightly better than the newA footprint (most likely due to the slightly larger number of visits per pointing), but the general adoption of a big-sky style footprint that includes galactic plane coverage would be ideal*.
*it’s worth noting that difference imaging for TNO discovery may be difficult near the galactic plane - however, transient and variable studies will also rely on this difference imaging, so it may be reasonable to assume it will work to some level (and thus benefit TNO discovery to some level, even if not as far as implied in these trends).

We should also look at the discovery for other populations - their variations are not as strongly dependent on footprint (only 96-102%, compared to the TNO range of ~92%-107%, after discounting the filter_dist runs which produced a -30% completeness for TNOs). The bright PHA and NEO population completeness is better for the ‘big-sky’ style runs (both big_sky* and the bulges*related variants) … presumably due to sky coverage again. However, the fainter PHA and NEO population completeness falls slightly in these big_sky style surveys, likely due to the fewer number of visits per pointing (as these smaller objects reach bright apparent magnitudes for a shorter amount of time, you have to have more visits to ‘catch’ them).

Checking in with characterization, we find more ties with filter distribution across the footprint than the footprint map itself, although the increase in TNO observations in the big_sky style runs is again visible, with a huge increase with newB (which has additional observations through the NES, in comparison to most of the other big_sky footprints).

Next: rolling cadence

When looking at the effect of rolling cadence, the best examples are from the FBS 1.6 release. These rolling cadence runs included a new ‘footprint class’ which calculates more precisely the season start and end for each point in the sky and then tracks the number of visits at that point compared to the time available to observe it (instead of just tracking the number of visits at each point). This means that rolling seasons start and end more smoothly, instead of rushing to acquire visits at the start of a particular part of the 'rolling sky’s availability.

The FBS 1.6 ‘rolling_fpo’ runs (rolling footprint from FBS 1.6) have 1x30s visits, so should be compared against the FBS 1.6 baseline_1nexp_v1.6_10yrs run (which also uses the new footprint).
These runs include:


When we look at the variation of discovery completeness over these runs (normalized to the completeness values in baseline_1nexp_v1.6_10yrs), we don’t see any clear wins, across all populations. The Jovian Trojans tend to have the hardest time with rolling cadence, perhaps because the Trojans move slightly between the declination bands in each year so by swapping sky coverage around we end up missing some objects (the Trojans being the most concentrated on the sky). The losses are worst for the 6-band rolling cadences, which split the sky into 6 separate declination bands and cover these in successive years. Most other populations do slightly better than baseline with the 2-band rolling cadence, (and some even do slightly better with 3-band rolling cadence) probably because there are slightly more visits so slightly better chances of having a group of visits which work for discovery. The second number in the name of each run refers to how much weight the rolling band received compared to the background visits – 0.8/0.9/1.0 imply increasing emphasis on the rolling region compared to the ‘background’ (although there are other basis functions which require some minimum number of visits per year, for all-sky template generation, which limit the effect of these weights).

Solar system science does not favor these kind of rolling cadences for all populations; most populations gain (<2%) from a limited 2-band rolling cadence, but faint MBAs lose ~1% completeness for all rolling cadences and faint Jovian Trojans lose 1-5% depending on the exact rolling cadence chosen.

Once again checking in with characterization -

There are no clear wins in terms of characterization either, which is somewhat surprising as more visits in a given time period would seem to make it more likely to receive enough visits to measure colors and/or lightcurves.
If we look specifically at the likelihood of getting enough visits for lightcurve inversion (which requires a large number of high SNR visits over a wide range of phase angles), rolling cadence looks much more favorable for some populations (although still bad for Trojans) -

Given that characterization is a secondary metric behind discovery however, this does not seem like enough to choose rolling cadence as a preference.

Next: twilight NEO surveys

To evaluate the effect of a targeted NEO twilight survey, we look at the ‘twilight_neo_mod[1,2,3,4]_v1.5_10yrs’ series:


In each of these simulations, an additional twilight time NEO-targeted survey was added. The NEO-targeted visits consist of 1 second exposures in r, i, or z band (bandpass used depends on which filter was previously in use in the camera, which means it is loosely tied to lunar phase but also survey progress). For each night where the twilight survey was triggered, there were attempted observations in the evening twilight and morning twilight, typically about 440 visits per night. The observations within a given night required triplets for each field pointing. The fields chosen for the NEO survey were within 40 degrees of the ecliptic, at high airmasses toward the sun.
The “modX” indicates how frequently the twilight_survey was triggered; 1 = every night*, 2 = every other night*, 3 = every third night*, 4 = every fourth night*.
(*where possible … there were some nights where weather would have not permitted visits in one or both of the twilight periods, etc.)

Number of visits per night for the twilight survey, in the twilight_neo_mod1 and twilight_neo_mod4 simulations:
and statistics (mean/median/percentiles) on the number of twilight NEO survey visits in each night and the number of nights the twilight survey was executed (“Count”).

Sky coverage patterns for an individual night in the twilight_neo surveys (night chosen based on the 'first night that had more than 500 visits in the twilight_neo survey):

After the full ten years of the survey, the sky coverage included in the twilight survey looks like:

After this time, the field altitude and solar elongation histograms for the twilight_neo survey fields looks (similar for each different simulation)

Ok, description of the twilight survey characteristics out of the way - what was the impact?

In general it looks like a win or slight neutral for the bright/large end of the populations but a loss at the faint end of the populations, with this effect being strongest for the case when the twilight_neo survey executed every possible night (mod1). When the twilight survey exectued every night, large NEOs and PHAs showed a ~5% increase in completeness, but small NEOs and PHAs had a ~6% loss in completeness. Other fainter populations had even stronger losses - smaller Trojans and TNOs lost 15% in completeness. This can be understood when we look at the overall number of standard (longer exposure) visits in the WFD in each of these twilight_neo cases compared to the baseline:

The twilight time taken by executing the twilight NEO survey appears to be necessary for standard operations. In the case of twilight_neo_mod1, the number of visits for “other purposes” drops by 10% – a significant loss, that is likely the cause of the decreased completeness for smaller/faint objects (which don’t benefit so much from the 1second visits in the twilight neo survey).
(Edit to add here, as well as below - ‘twilight’ in this case involves observations starting as 12 degree twilight. This time is used for both the start of regular operations and the start of ‘twilight’ survey observations).

Again checking on the effect on characterization, we almost the opposite effect … the smaller populations have an increase in the number of objects which could have 4 colors measured, with only small (~3%) decrease in color opportunities for the TNOs.

1 Like

Then: Any other effects (from other survey strategy choices)

The families of simulations above cover the most obvious questions related to solar system science. It’s worth asking if any of the other simulations created for other purposes (such as adding high airmass visits for DCR measurement, or adding short exposures over the sky for bright star measurements) impact solar system objects significantly, either in a positive or negative way.

If we look at all of the runs available in FBS 1.5, and take out runs that we’ve included in our analysis above (or are equivalent to these runs) we see that there are some runs which are worth commenting on.

The WFD-depth series are a family of simulations which varied the weight given to the WFD region, looking at some metrics related to SRD quantities. While there is only a small variation with the fraction of time spent on WFD, a heavy emphasis on WFD results in slightly lower completeness.

On the other hand, the dcr_[x] family of runs as well as the short_exp[x] series of runs added either (single) high airmass or (single) short exposures over the sky, which do not seem to add to the general completeness for solar system objects, with a trend that the more time is spent in their special observations - the lower the completeness for solar system objects.

It’s worth commenting on the footprint_stuck_rolling run, which stands out as neutral or an improvement for completeness in all solar system populations. This is somewhat surprising, as this simulation was intended as a test case for ‘breaking’ science: it is a rolling cadence, where the ‘rolling’ portion of the sky doesn’t actually swap as intended. The final footprint looks like
While most populations have neutral or slightly better than baseline results with this simulation, the small Trojan population enjoys an 8% boost.

The notion that what is called “twilight time” might be subsumed into standard operations was perhaps not clear at the time the white papers were written, rather the notion for this variation of a twilight survey and perhaps others was to make use of time before the WFD survey starts and after it ends each night. What was the definition here of when twilight begins and ends? Presumably no simulation of the effect of mega-constellations was included?

Presumably bright means H≤16.0 and faint H≤22.0 for NEOs? If the “most important metric is discovery” these detection statistics may need to be weighted by the 90+ % (perhaps 99%) completion for H≤16.0 at the start of the Survey and ~43% completion for H≤22 at the start. Many detections of all of these objects will not represent new discoveries. Have any of the simulations tagged the known populations? This may have a significant effect on near-sun discoveries since Rubin’s significantly fainter limiting magnitude is anticipated to offer a disproportionate benefit, which is to say that the trade-off may be between discoveries near-sun and re-detections near opposition.

Hi @RobSeaman,

Yes, I think the concept of ‘twilight time’ was maybe not well communicated at the time of the white paper call. In the twilight_neo simulations, we start observations at a sun altitude of -12 deg (astronomical twilight) and in the standard simulations we did also. At the time of writing the call for white papers, we thought this 12-18 degree twilight time was not required for WFD visits … with the more pessimistic weather conditions we use now, and the number of other minisurveys in play, we do now need this time to meet WFD requirements.

I think we could look into doing the twilight survey during nautical twilight, but it wasn’t clear (to me anyway) whether that was what people were talking about.

No, we are not including the effects of mega-constellations of satellites. I suspect if we have a true mega-constellation, it will not be worth doing twilight surveys for NEOs anyway, but I could very well be wrong. We don’t know enough about it yet, but I am a bit pessimistic perhaps.

I did not include ‘previously known’ objects in the metric … it’s something we could do. If most of the H=16 objects are known at the start of the survey, I’m not sure that I follow that the twilight surveys would be helpful … adding the twilight surveys decreases completeness for the H=22 NEOs.

I got some questions / requests for clarification via email, which I will answer here for everyone:

1x30s vs. 2x15s visits – where are these? These two kinds of simulations are in the 'Intranight strategy" in the first post. The ‘baseline_2snaps’ runs have 2x15s visits, but everything else has 1x30s visits. In terms of the overall number of visits in the simulation, going from 2x15s visits to 1x30s visits lets us cram in about 7% more observations overall (due to smaller overheads from the readout and shutter open/close). But the impact on population completeness when going from 2x15s (the current default still) to 1x30s is smaller: (these are % changes in overall completeness)

3 pairs in 15 nights detection loss NEO H=16.0       0.850044
3 pairs in 15 nights detection loss NEO H=22.0       3.458009
3 pairs in 15 nights detection loss MBA H=16.0       0.000000
3 pairs in 15 nights detection loss MBA H=21.0       4.170132
3 pairs in 15 nights detection loss Trojan H=14.0    0.000000
3 pairs in 15 nights detection loss Trojan H=18.0    6.143443
3 pairs in 15 nights detection loss TNO H=4.0        0.240449
3 pairs in 15 nights detection loss TNO H=8.0        2.117227

(with base completeness levels in the 2x15s simulation of

3 pairs in 15 nights detection loss NEO H=16.0        91.760000
3 pairs in 15 nights detection loss NEO H=22.0        58.046157
3 pairs in 15 nights detection loss MBA H=16.0       100.000000
3 pairs in 15 nights detection loss MBA H=21.0        56.690961
3 pairs in 15 nights detection loss Trojan H=14.0    100.000000
3 pairs in 15 nights detection loss Trojan H=18.0     47.059684
3 pairs in 15 nights detection loss TNO H=4.0         74.860000
3 pairs in 15 nights detection loss TNO H=8.0         47.579377

At the bright end, we see small effects while at the faint end, the effects are larger. This reflects the fact that large/bright objects get more opportunities for discovery anyway, so are less sensitive to having more observations in the survey.

What does the FracPop5 metric do? Sorry, I should have clarified this. There are a number of characterization metrics run for each population – the inner solar system (everything out to TNOs) and the outer solar system (the TNOs) have slightly different metrics for these. You can see the full results for each simulation at (for the FBS 1.5 runs).
For the outer solar system, characterization is evaluated using the LightcurveColor_OuterMetric which looks for a primary bandpass that has enough visits to determine a lightcurve, and then secondary bandpasses that would have enough supplemental observations to determine a color. The result of the metric tells you how many bandpasses it could have a color (with another bandpass having a lightcurve) in – for the outer solar system then “FracPop5” = the fraction of the population which had a lightcurve in one bandpass and then four other bandpasses with enough for color information … yielding four colors and a lightcurve.
For the inner solar system, characterization is evaluated with the Color_AsteroidMetric for colors, and separately for lightcurve inversion potential with LightcurveInversion_AsteroidMetric. The asteroid color metric looks for enough SNR-weighted visits in various bandpasses to establish a color; the colors are required to be in particular filters. For the inner solar system populations, ‘FracPop5’ indicates that the asteroid was observed with enough visits in grizy to obtain colors with those bandpasses.

There were some more questions about the twilight survey, but I think those are handled above.

I realize the plots are somewhat hard to read. I did a relatively quick job plotting these using pandas plotting functions directly, so the colors and linestyles aren’t assigned … I can work on it, but you can also access all of this information directly (see CSV files for discovery and characterization for each simulation release at (*csv), the list of runs in each subset of simulations is in the thread above, and you can recreate this work using the notebooks on the epyc Jupyterhub server (which SSSC members have access to) - they are the *ipynb files in the common/2020_June directory (Intra-night Strategy.ipynb, Footprint Strategy.ipynb, Rolling Strategy.ipynb, and Twilight NEO surveys.ipynb).

Is plotting H=4 TNOs diagnostic for the Kuiper belt/TransNeptunian region? H=4 is the dividing line for dwarf planets. Would plotting H=6 TNOs be more reflective of TNO discovery statistics or does it look the same as H=8?

Good point, in general H=4 and H=6 would look the same; H=8 show more variation because it is in the area of the completeness curve that is dropping more rapidly. For TNOs completeness does not drop as fast as it does for other populations. I suppose there may be some simulations where completeness drops faster or earlier if the images are shallower, so using H=6 might be a good change.
The completeness curves look like this:
and for all of the 1.5 simulations they have similar ratios at H=4 and H=6 …
and for the other populations I was just using the brightest objects so used H=4 here.
I could see swapping to H=6 instead.

Since it’s relevant:

here are the completeness curves for each of the populations for all of the FBS 1.5 runs.






Using a quick by-eye, really rough estimate of where the completeness drops from about the max down to about 10% … looks like PHAs and NEOs drop over about 6 magnitudes in H, the MBAs drop this far over about 4 magnitudes in H, the Trojans drop over about 3.5 magnitudes, and the TNOs take about 5 or 6 magnitudes to make this drop (although it looks to me like completeness is still rising slowly even at the bright end). I think this is reflecting the wider (relative) distance distribution within the populations that drop more slowly.

Thanks, Lynne, these are really helpful.
I have it right that twilight NEO survey (sans megaconstellations) hasn’t yet been run with an ISO population? I see PHA/NEO/MBA/Trojan/TNO on the discovery variation metric plot.

That is correct - not run with an ISO population (or a Vatira population). Comet populations are also one we’ve talked about, but has not been implemented for these runs.

Thanks for all this. There are a few populations (resonant KBO islands) and L4/L5 Neptune Trojans in the repo from the white paper. Would it be easy to include these as well. Many of them highlight specific needs for the NES that aren’t necessarily identified by the KBO detection numbers. Maybe that’s something that can be a task for participants at the November workshop?

I think these are not unreasonable things to consider.
Let’s identify what populations would be good to add, I’ll double-check the available info, and we can get those run too.

I’m going to go through and replace the figures here with updated versions where I have them. I learned a thing or two about plotting with pandas dataframes since I made these, and added some extra features (for when comparing the lots of runs for the SCOC report).