Hello all, I generated fake catalog of stars with random coordinates (raColName, decColName in radian) and magnitude (zmagVar) within the patch 5,5 corner, tract 9813 of HSC.
I followed this page to create the catalog [New Tasks for Fake Source Insertion].
I set all other columns concerning galaxy parameters to zero as I am not interested to add galaxies.
I then used this catalog with processCcdWithFakes: processCcdWithFakes.py /datadec/cppm/Manal/HSC/DATA/rerun/20200903 --rerun 20200903_NoConditionfakes --clobber-config --config insertFakes.fakeType="/datadec/cppm/Manal/HSC/Logs/processCcdWithFakes/fakecatalogFullNoCondition.csv" --id visit=1168 ccd=0..8^10..103 filter=HSC-Z tract=9813
I got no errors, however it seems that no stars have been added to the exposure (visit 1168) as the output of this command line for all ccds was:
processCcdWithFakes.insertFakes INFO: Making 0 fake galaxy images
processCcdWithFakes.insertFakes INFO: Making 0 fake star images
Can you please let me know what is wrong in the procedure I followed?
I attach the distribution of the fakecatalog coordinates (red plus), and the sources in the calexp 1168 in blue. 
Thank you for you answer
I suspect that the names of the RA/dec columns in your input catalog (fakecatalogFullNoCondition.csv) need to be raJ2000 and decJ2000 (unless you change them with config options). Also make sure that the sourceType column is present and zero or star.
It looks like anything recognizable by astropy.table.Table.read() is acceptable as an input catalog, with the type able to be inferred from the file suffix if present, so your .csv file should be OK, but, after further perusal of the code, it seems that the length of your fake catalog is zero so something must not be getting read in properly.
Hello, thank you for your answer. Yes, the ra, dec columns have the right names (raJ200, …) and they are in radian. Here is an extract of the catalog I am using
Do you get a log line saying “Removing [...] rows with HLR = 0 for either the bulge or disk” with a number equal to the number of rows in your catalog?
If so, I think you need to add --config insertFakes.doCleanCat=False to your command line as a workaround for a bug in https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/insertFakes.py#L683
In the log file I have:
processCcdWithFakes.insertFakes INFO: Removing 0 rows with HLR = 0 for either the bulge or disk
Ok, I will add --config insertFakes.doCleanCat=False to my command line.
Thank you
If that log line says “0”, then something else must be wrong.
Is the output of these lines non-zero?
from astropy.table import Table
print(len(Table.read("/datadec/cppm/Manal/HSC/Logs/processCcdWithFakes/fakecatalogFullNoCondition.csv").to_pandas()))
The output of this command is 1000 (it is the number of objects I generated). The cvs file is ok, I could read it, plot the ra, dec. So I guess the issue comes from other place.
I added the option insertFakes.doCleanCat=False, but seems does not change anything (it is still running but I can see the two lines:
processCcdWithFakes.insertFakes INFO: Making 0 fake galaxy images
processCcdWithFakes.insertFakes INFO: Making 0 fake star images
). I attach the log file I got before adding this option. ProcessFakeOutputNocondition.py (243.7 KB)
One more thing: which versions of pipe_tasks and the rest of the Science Pipelines are you using?
I am using DM_RELEASE=v20.0.0
Ah, that may be part of it. It looks like in that version you need to specify sourceType as “star” rather than 0.
Ah ok, from which weekly version I can use 0 instead? I will try with the version w_2020_28
Looks like w_2020_29 is the earliest, but it may be safer to use w_2020_34 which has some bug fixes.
Thank you for your fast answers. Ok, I will try with 34 version then. Thanks
I got lots of other kind of errors when using version 34. I then changed the sourceType column to star instead of 0 with stack v20, and so it works.
I seen some outputs like:
processCcdWithFakes.insertFakes INFO: Making 0 fake galaxy images
processCcdWithFakes.insertFakes INFO: Making 4 fake star images
Thank you again for your kind help
Manal
What errors did you get with 34? Could you give the table file to me in a readable format so that I can try and get it to work?
Oh sorry, I did see your request before now. Yes, let me please reproduce the error (as I did not keep it) and I will send it to you with the table I am using
Hi @mchbib, I’m following up on this topic because it is not yet marked as “resolved”. It sounds like @ktl’s suggestions were working for you, and based on the topic “imageDifferenceDriver with fakes”, that you were able to add fake stars to the image?