Enable PP upload.py for LSSTCam-ImSim

Description

Pick a few exposures from /repo/dc2 runnable by ApPipe, put them to pp-dev’s "unobserved" curation, prepare the pp-dev central repo with needed refcats/calibration/skymap/template data, setup the necessary butler collections, and all other things not yet done in  DM-48194 to make this work: upload.py LSSTCam-imSim 2 (or a higher number up to ). Update the playbook accordingly.

For this ticket, use s3://rubin-pp-dev-users/central_repo_2 as the central repo. 

Issue Matrix

hide

Activity

Hsin-Fang Chiang March 12, 2025 at 5:03 PM

`upload.py LSSTCam-imSim` is now part of the test suite

Hsin-Fang Chiang February 19, 2025 at 12:21 AM

Using 2.2i/defaults chain directly as I chained previously meant trouble about calibration collection 2.2i/calib versus LSSTCam-imSim/calib in the local repo, and it needed an extra calib collection export. Following the discussions at I re-chained the LSSTCam-imSim/defaults in the dev repo:

butler collection-chain s3://rubin-pp-dev-users/central_repo_2 LSSTCam-imSim/defaults LSSTCam-imSim/calib,skymaps,refcats,LSSTCam-imSim/templates,pretrained_models
This is more consistent with other instruments.

Hsin-Fang Chiang February 13, 2025 at 9:28 PM

With the ticket branch from here and , `python upload.py LSSTCam-imSim 2` uploads 3 detector images from exposure 496960 and 496989, and all 3 images can finish ApPipe and generate alerts.

please review
https://github.com/lsst-dm/next_visit_fan_out/pull/21 and https://github.com/lsst-dm/prompt_processing/pull/255 . Thank you!

LSSTCam-imSim is weird because it's an LSST instrument but does not follow most LSST conventions in their headers. Forcing it to be really like LSST means changes in `obs_lsst` and may invalidate existing data, which we don't want. On the other hand PP will only process newly uploaded imsim data and we can overwrite headers for those.

Hsin-Fang Chiang February 11, 2025 at 9:44 PM

Adding notes here as this could bite us later… As of 2025-02-11

 

The DC2_cells_v1 in /repo/dc2 is named lsst_cells_v1 in other repos; it has the same hash.

TheDC2_cells_v1 in all other repos, including s3://rubin-pp-dev-users/central_repo_2 , have a different hash. Those have a different tractOverlap in their config.

While imsim templates are generated in /repo/dc2 and now seems too late to correct it for . I think we can live with it somehow, just that it will likely generate confusion.

Hsin-Fang Chiang February 10, 2025 at 9:14 PM

imsim DC2 /repo/dc2 has a different convention and have defaults collections under 2.2i/x , not LSSTCam-imSim/x . PP relies on those collection names, so I added chains in pp-dev repo:

Done
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Labels

Reviewers

Krzysztof Findeisen

Story Points

RubinTeam

Components

Checklist

Created December 17, 2024 at 9:56 PM
Updated April 24, 2025 at 10:25 PM
Resolved March 12, 2025 at 5:03 PM