Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Set up initial "main" shared data repository at NCSA

Description

This ticket captures setting up the "main" data repository (containing all non-simulated data) at NCSA, including:

  • Instrument registration for HSC, DECam, and all relevant Rubin data (at least LATISS).

  • Raw ingest for at least some data from each of those instruments (at least RC2 for HSC).

  • Gen2to3 conversion of the latest PS1 and Gaia reference catalogs.

  • Registration of the main skymaps in use for HSC processing.

  • Gen2to3 conversion of the current default HSC calibration repository.

  • Gen2to3 conversion of a few recent HSC RC2 reruns.

This is blocked by NCSA work to set up the filesystem and database appropriately (tickets pending), and would ideally be done after and any follow-on tickets from that as well, so we can have DB-side access controls from the beginning.

Similar tickets will be created for the other shared data repositories.

Checklist

Lucidchart Diagrams

Issue Matrix

hide

Activity

Jim Bosch April 19, 2021 at 5:23 PM

Paul reports elsewhere that he doesn't have time review the lsst-dm repo right now, so I've just merged it and I'm calling this ticket done.

Jim Bosch April 6, 2021 at 7:38 PM

I've merged the daf_butler, obs_base, and obs_lsst branches, since they (of course) don't depend on the gen3_shared_repo_admin branch.

Tim Jenness April 5, 2021 at 7:57 PM

I have a couple of comments on daf_butler and obs_base:

  • Log messages reporting information when doing a dry run

  • Feeling a bit squeamish about obs_base making assumptions about whether Datastore is going to want expanded DataIds or not.

I have no opinions on the obs_lsst config change.

Jim Bosch April 5, 2021 at 7:07 PM

I think it's finally time to get this reviewed; I still have to ingest DECam data, but I'll do that on .

The vast majority of the code here is at https://github.com/lsst-dm/gen3_shared_repo_admin/pull/2. Yes, that's an lsst-dm repo, so it doesn't really need to be reviewed or satisfy much of anything as part of that review (I think the code and docs are good, but there is a complete and utter lack of unit tests). I'm also not sure it'll really be useful after we set up the initial repos at a few places and then retire Gen2, but I certainly hope it'll be useful for setting up repos at places other than NCSA. With that in mind, I'm going to ask to take an "as superficial as he likes" look at it, in preparation for us pair-programming adapting it for the Princeton HSC repositories in the near future.

, could you look at the smaller changes in daf_butler, obs_base, and obs_lsst? These are all incidental minor fixes I did in the course of the work, though the daf_butler/obs_base change to allow advanced users to not always expand data IDs might be something that merits a follow-up ticket.

Done
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Labels

Reviewers

Paul Price
Tim Jenness

Story Points

RubinTeam

Parent

Checklist

Created February 3, 2021 at 3:51 PM
Updated April 19, 2021 at 5:23 PM
Resolved April 19, 2021 at 5:23 PM