BIGINTs/Longs in CSV file uploaded to the RSP

Support
#1

Hi

If I upload a CSV file of bigints/longs to the RSP (eg generated from

SELECT top 1000 objectId FROM dp02_dc2_catalogs.Object

=>

objectId
1248675773246603272
1248675773246603273
1248675773246603274
1248675773246603275
1248675773246603276
1248675773246603277
...

the values get interpreted as floats? and the 1000 unique objectIds end up as multiple occurences of the same values (ended up with 9 unique values).

=>

|objectId              |
|double                |
|                      |
|                      |
 1.24867577324660326E18 
 1.24867577324660326E18 
 1.24867577324660326E18 
 1.24867577324660326E18 
 1.24867577324660326E18 
 1.24867577324660326E18 
 1.24867577324660326E18 
 1.24867577324660326E18 
...

I realise I should probably use a different format that specifies datatypes and it’s not necessarily trivial to auto-detect types but it threw me for a a while. There’s a type box on the upload window so maybe it’s possible to declare it there but I’m not sure what to put there.
Thanks
Mike

image
image964Ă—981 42.2 KB

image
image1398Ă—981 58.7 KB

image
image1398Ă—981 61.4 KB

#2

Hi @MRead – interesting. OK, so just to check, you’re using the Portal to run the following query:

SELECT top 1000 objectId FROM dp02_dc2_catalogs.Object

And then downloading it as a csv and re-uploading it in the portal, and it’s interpreting the type as “null”?

If so, I just tried this and it’s properly interpreting the objectIDs as longs, so the entire objectID is there.

Screen Shot 2024-10-22 at 12.56.04 PM
Screen Shot 2024-10-22 at 12.56.04 PM942Ă—469 25 KB

#3

Hi @ryanlau

Yes that’s what I’m doing. Just gone through the steps again to confirm. Screenshots attached.

Odd, I must be doing something wrong.

Mike

csv1
csv11593Ă—981 279 KB

cvs2
cvs21593Ă—981 159 KB

csv3
csv31593Ă—981 206 KB

csv4
csv41593Ă—981 132 KB

csv5
csv51593Ă—981 177 KB

#4

Hi @MRead ,
OK, I think I’ve figured out why we’re getting different results. I was running a different instance of the rsp, but double checked that I am able to reproduce your issue on data.lsst.cloud – I’ll see if I can raise this issue to the proper channels

Screen Shot 2024-10-22 at 2.24.57 PM
Screen Shot 2024-10-22 at 2.24.57 PM1638Ă—1604 174 KB

1 Like
#5

Hi @ryanlau , thanks for confirming. Maybe it’s something that’s already fixed if it’s working in a test RSP?

#6

Yes, I think that’s correct. I’m not sure when an updated version of the RSP will be pushed, but I’m asking around.

#7

@MRead I took a look, as this thread remains without a solution. I cannot reproduce the issue at data.lsst.cloud by following your steps. I use the Chrome browser and the process works the same as what Ryan experienced.

One thing we could check is whether the problem happens at the time of download or upload. If you look at the .csv file, do the contents look like long integers (indicating that the download worked fine, and perhaps the conversion happens during upload) or are they in exponential notation (vice versa).