Scheduler not working for given RA,Dec and MJD start

Support
#1

Hello,

I am trying to run scheduler for the following parameters:

def generate_observations_rough(self, conditions):
obs = empty_observation()
obs[‘RA’] = np.radians(275.08)
obs[‘dec’] = np.radians(7.1853)
obs[‘exptime’] = 30. # Seconds
obs[‘nexp’] = 2
obs[‘filter’] = ‘r’
# If it’s been 12 hours or more, don’t try to execute the observation anymore
obs[‘flush_by_mjd’] = conditions.mjd + 0.5
obs[‘note’] = ‘mysurvey’ # Always good to set the note to which survey generated the observation
# Make it a list of 3
result = [obs] * 3
return result

for a MJD start of 59560.

The error I am getting is:
RuntimeError: Scheduler has failed to provide a valid observation multiple times.

I was wondering if someone can help me fix this problem. Thank you

#2

@yoachim @ljones – any thoughts on this? Thanks very much…

1 Like
#3

Looks like you have asked the scheduler to observe a point on the sky that has an altitude of -8 degrees. The telescope has a minimum altitude limit of +20 degrees.

You can change the starting MJD to a time when the target is up, or include a filler survey object that has available targets it can observe until the higher priority targets are available.

1 Like
#5

Hi Peter, Aaron,

Thank you for the reply. I have now solved that issue - it was indeed caused by the altitude limit.

However, now I have ran into a new issue with the scheduler: Trying to run the example in lsst/rubin_sim_notebooks/scheduler/1-Scheduler_intro.ipynb , I semi-consistently get an error when the declination is >-77. The notebook runs perfectly fine when the RA is 228 deg or more and DEC is < -77, but seems to break once the RA and DEC are outside these limits.

I have posted the detailed issue at Error running Scheduler_intro notebook in lsst/rubin_sim_notebooks · Issue #383 · lsst/rubin_sim · GitHub

Any help would be appreciated.

#6

Hi @yoachim , @ameisner ,
Just following up with you two on @Susmita’s question about the scheduler issues when running outside of the coordinate limits mentioned above.

#7

Thanks for the ping.

So, I think the problem is that a couple of different things are converging to make it hard to see what’s going on.

First of all, it’s not always obvious, but re-running the cell in the notebook which calls sim_runner does NOT actually do the same thing each time, UNLESS you reset the model observatory. This is because the model observatory is updated during the sim_runner execution, so that “time passes”. In other words, check mo.mjd before and after you try to run sim_runner, to make sure it’s trying to obtain observations when you think it is. A good practice would be to reset your model observatory to a known time at the start of the cell.

Secondly, at least when I tried this – when sim_runner “crashes out” due to not being able to obtain observations, it doesn’t return anything! So your “observations” array from sim_runner is empty even though the model observatory is updated inside sim_runner. This makes it look like nothing was observed … however, it actually was. If you notice the time on the model observatory after the error, it’s 60218.33631284 instead of 60218, when it started. So sim_runner ran (successfully) for 0.3 days or so, and if you try running sim_runner with a survey length of 0.3 days, you do see the successful observations returned.

So the question really is – what happened at mjd = 60218.33631284 ?
And then, if I look at the observations which were acquired if I ran for 0.3 days I see the final altitude of the field is 0.377 radians … which is 21 degrees, and the pointing limit of the telescope is again +20 degrees.
So the observatory model is refusing to observe pointings which are outside the range of where it can safely observe, and then sim_runner says it cannot schedule observations (because this one target is its only choice, and the target has now set).

Does this answer the question?

Now, I will also add this —
the simulated example survey in the notebook is really just a very simple example. Fixing only a few target observations and insisting on these in “generate_observations_rough()” without any consideration of whether the targets are visible or not results in these kind of problems – not all fields are visible all times.
It’s not how any of our actual surveys in use work … every survey we actually use with the CoreScheduler in full simulations includes a check on whether the field is at a usable altitude, in the “calc_reward_function”. And then, because the reward is “infeasible” (-Infinity), the requested observation will never show up as being requested – and so sim_runner doesn’t crash from this effect.

The RA/Dec in the simulated simple example were chosen so that the pointing is visible for the length of time shown in the sim_runner example, but this is not how you should try to program any real survey.

If you are trying to program something which will acquire observations of a particular point on the sky, but does account for visibility, you might want to have a look at the FieldSurvey. You might still want to be able to propose valid targets for times when your fields are not available, but I think the error/crash is only triggered by repeatedly requesting invalid targets (not just not proposing targets).

1 Like
#8

Hi @Susmita ! I just wanted to follow up on this question and see whether we were able to answer your question. Did this last post from Lynne solve the second issue you were having? If not, perhaps we can resolve it with more information. Thanks!

#9

Hello,

I would like to first thank @ljones for such a descriptive and helpful reply and the community for being so supportive. This has helped me in understanding the cause of the failure. At the moment, I have been working on incorporating FieldSurvey into my scheduler such that similar errors are avoided. I am not fully successful in doing so yet but I think we may be onto something. If you happen to have some resources that may help in understanding and incorporating FieldSurvey, it will be very helpful.

Thank you again
Susmita

#10

I take it you’re trying to use the rubin_scheduler software to generate a scheduler for a different purpose? It may be helpful to know a little bit more about your goals.

#11

Hello, Thank you for your reply. The main objective of the work is to generate and schedule a series of observations over a period of ~months/years for a list of source positions using rubin_scheduler. Previously, the test source in our case was below the altitude limit when the survey length was 1 day, changing which to 0.3 days produced a list of observations. The suggestion was to use FieldSurvey to calculate the reward function and then completely remove the observations from the list that was nit feasible. I have been working on incorporating FieldSurvey, however, it seems to give every RA and DEC a calc reward function of 0 which makes every observation feasible. I was wondering what is the best way to deal with generating a series of observations and then using scheduler to schedule only the observations (over a period of a few months/years) that was feasible such that we do not get any errors from scheduler due to the multiple calls to an observation that cannot be observed at all.

#12

Thanks for the follow up info, @Susmita. @ljones , would you happen to have a response to Susmita’s post?

#13

Hi Susmita,

You need to actually change the BasisFunctions you’re using, so that one of them returns np.nan when the field is not visible. FieldSurvey as-is is only suitable for a single pointing, so I’m not sure about having a list of RA/Dec values within the FieldSurvey (if that’s what you’re doing), although you could have multiple FieldSurveys.

For a bit more information on calculating feasibility and rewards, you might find the information here useful: Scheduler — rubin_scheduler

For a more complex example that uses FieldSurvey with basis functions that will account for whether the fields are visible or not, you might find this notebook helpful – https://github.com/lsst-sitcom/ops_rehearsal_commissioning_2024/blob/main/scheduler/ops_rehearsal_scheduler.ipynb

And we also have a new kind of survey class that might be interesting if you are indeed trying to schedule a list of RA/Dec pointings, if you have a look here - https://github.com/lsst/rubin_sim_notebooks/blob/main/scheduler/pointings_example.ipynb
(the PointingsSurvey doesn’t currently use separate basis functions, but instead will need to be subclassed to include the equivalents of these basis functions … examples are given in the notebook).

Do any of these help? If you’d like further feedback, I’d be happy to have a look at a notebook where you’re developing this for use, if you can post a link.

Lynne

#14

Hello Lynne,

Thank you so much for your reply. I have been working with ops_rehearsal_scheduler and it seems to be doing what I wanted it to do. Thank you so much for your help.

Regards
Susmita

1 Like