I am currently implementing the TAP protocol layer for our Lite IDAC, and I would like to get some advice on the best TAP service for the system. Specifically, I am evaluating whether to use GAVODaCHS, PyVO,
another TAP implementation or even making our own.
Could you please share your experience or recommendations regarding the following:
Which TAP service would you recommend for a project like Lite IDAC?
Are there any known challenges or advantages with GAVODaCHS, PyVO, or other TAP services that are particularly relevant to LSST data or similar large-scale astronomical projects?
If there are any other services or tools that you think would be more suitable for managing and querying large datasets from LSST or similar observatories, I would appreciate your input.
Does this mean that you will be serving a subset of Rubin catalogs through this TAP service? Or is this TAP service just for supplementary data that’s unique to your IDAC?
You should be aware that if you do not do this the way that we do in the RSP, there are aspects of the normal RSP service of catalogs that will require extra work from you in order to duplicate – if this is a goal of your IDAC. Not all TAP server implementations are interchangeable in these respects.
HI @SantiagoArroyoL - Yes, as Frossie says, CADC has a TAP server implementation that should fit your purposes. The deployment documentation and process is improving but may require some help from us, which we’d be happy to provide. Probably the forum mentioned above, or the IDAC slack channel is best for those questions: Slack ?
Just for clarity: in the RSP we are currently running a fork of CADC TAP with a major feature added (DataLink service descriptor annotation). This feature is quite important to the overall user-facing feature set of the RSP.
We are working with CADC to get this upstreamed, but I’m not sure how long this will take.
Yes, thanks for making that clear Gregory. I pointed to the YouCat image and source documentation because that one is the most “out of the box ready” in OpenCADC. When we do upstream the RSP TAP, it will also be in the form of a configurable image, and will come with those extra DataLink features.
I should also point out that YouCat is simply a TAP server but with the option feature of allowing end users to upload their own persistent tables. It’s an implementation of what will likely become the next TAP standard.
Also, when the RSP TAP is upstreamed, YouCat will inherit the DataLink service descriptor support as well.
Thank you all for your replies.
I’ll be trying the YouCat implementation, seems very complete and whenever you upstream the RSP TAP it’ll be easier to migrate.
I’ll be making further questions through Slack or the IDAC though this was very helpful and I’m happy to see it has helped others.
I’m quite sure the Docker image name/tag is not meant to be retrieved as a URL; it’s meant to be provided to docker pull or as a Kubernetes image name.
I’m here once again. Which youcat documentation would you recommend me to read? I’m having trouble finding a more streamlined documentation and what are the next steps
Hi @SantiagoArroyoL
What’s not explicitly stated in these youcat docs is that the service (the docker image) requires a postgresql database with the pgsphere extension installed. You could get started with that if you don’t have it already. Meanwhile, I’ll put you in touch with one of our developers to help you work through those docs. If you have any specific questions please open a github issue in that repository and the conversation can happen there.
Brian
@bmajor let me know you are trying to use this. I can help with configuration.
First: back end database. This currently requires a PostgreSQL database with pgsphere extension (due to some “static config” inside the image); we are working to make it more configurable in the future.
We do have things available to help with the database setup (pgsphere rpms, recommended account config).
Second: AAI system. To use youcat effectively, you will need an AAI system. We have an in-house system but we also support OIDC (token auth) with a suitable deployment config. This hasn’t been tested much and I will need to fix at least one minor limitation before that actually works - I am working on that right now so fixed later this week). What do you have in mind for this?
Hi @pdowler , a pleasure.
Yes, we’ve already got a distributed PostgreSQL database. I’ll start reading about pgSphere shortly and install it this week. If you could share documentation about recommended configurations, that would be great.
What I was originally trying to accomplish was the TAP implementation to connect to our front end (mainly Firefly and Jupyter, mimicking the RSP Notebook and Portal aspects). I am using Youcat as sugested with the image @bmajor provided, yet I still have many doubts about which way to go with the configuration and what would be best for a final implementation, hence my original question.
And lastly about an AAI system we do already have an in house system developed by the university. This would mainstream the authentication process from members of the university that would have acces to the LSST portal and such we have a web portal mimicking again the RSP. This site is inaccesible without proprer credentials. We already have the Notebook aspect deployed with kubernetes allowing multiple users. We have some questions regarding this last step, wether we should use Phalanz or stay with our in house kubernetes solution, etc, etc. But one step at atime, first full backend.
TL;DR:
We already have PostgreSQL, will install pgshpere
Any suggestions on TAP implementation and YouCat usage would help
We could plan a meeting or a more direct communication channel to talk about AAI and deployment