BIGINTs/Longs in CSV file uploaded to the RSP

Hi

If I upload a CSV file of bigints/longs to the RSP (eg generated from

SELECT top 1000 objectId FROM dp02_dc2_catalogs.Object

=>

objectId
1248675773246603272
1248675773246603273
1248675773246603274
1248675773246603275
1248675773246603276
1248675773246603277
...

the values get interpreted as floats? and the 1000 unique objectIds end up as multiple occurences of the same values (ended up with 9 unique values).

=>

|objectId              |
|double                |
|                      |
|                      |
 1.24867577324660326E18 
 1.24867577324660326E18 
 1.24867577324660326E18 
 1.24867577324660326E18 
 1.24867577324660326E18 
 1.24867577324660326E18 
 1.24867577324660326E18 
 1.24867577324660326E18 
...

I realise I should probably use a different format that specifies datatypes and it’s not necessarily trivial to auto-detect types but it threw me for a a while. There’s a type box on the upload window so maybe it’s possible to declare it there but I’m not sure what to put there.
Thanks
Mike



Hi @MRead – interesting. OK, so just to check, you’re using the Portal to run the following query:

SELECT top 1000 objectId FROM dp02_dc2_catalogs.Object

And then downloading it as a csv and re-uploading it in the portal, and it’s interpreting the type as “null”?

If so, I just tried this and it’s properly interpreting the objectIDs as longs, so the entire objectID is there.

Hi @ryanlau

Yes that’s what I’m doing. Just gone through the steps again to confirm. Screenshots attached.

Odd, I must be doing something wrong.

Mike





Hi @MRead ,
OK, I think I’ve figured out why we’re getting different results. I was running a different instance of the rsp, but double checked that I am able to reproduce your issue on data.lsst.cloud – I’ll see if I can raise this issue to the proper channels

1 Like

Hi @ryanlau , thanks for confirming. Maybe it’s something that’s already fixed if it’s working in a test RSP?

Yes, I think that’s correct. I’m not sure when an updated version of the RSP will be pushed, but I’m asking around.

@MRead I took a look, as this thread remains without a solution. I cannot reproduce the issue at data.lsst.cloud by following your steps. I use the Chrome browser and the process works the same as what Ryan experienced.

One thing we could check is whether the problem happens at the time of download or upload. If you look at the .csv file, do the contents look like long integers (indicating that the download worked fine, and perhaps the conversion happens during upload) or are they in exponential notation (vice versa).

Hi @MelissaGraham , thanks for following up. It looks like the updates predicted by Ryan have propagated to the production RSP as it is now working for me.

Great news, thanks for confirming and marking the solution @MRead, we appreciate it.