Enable PP upload.py for LSSTCam-ImSim
Description
blocks
is blocked by
relates to
Issue Matrix
hideActivity
Hsin-Fang Chiang March 12, 2025 at 5:03 PM
`upload.py LSSTCam-imSim` is now part of the test suite
Hsin-Fang Chiang February 19, 2025 at 12:21 AM
Using 2.2i/defaults
chain directly as I chained previously meant trouble about calibration collection 2.2i/calib
versus LSSTCam-imSim/calib
in the local repo, and it needed an extra calib collection export. Following the discussions at I re-chained the LSSTCam-imSim/defaults
in the dev repo:
butler collection-chain s3://rubin-pp-dev-users/central_repo_2 LSSTCam-imSim/defaults LSSTCam-imSim/calib,skymaps,refcats,LSSTCam-imSim/templates,pretrained_models
This is more consistent with other instruments.
Hsin-Fang Chiang February 13, 2025 at 9:28 PM
With the ticket branch from here and , `python upload.py LSSTCam-imSim 2` uploads 3 detector images from exposure 496960 and 496989, and all 3 images can finish ApPipe and generate alerts.
please review
https://github.com/lsst-dm/next_visit_fan_out/pull/21 and https://github.com/lsst-dm/prompt_processing/pull/255 . Thank you!
LSSTCam-imSim is weird because it's an LSST instrument but does not follow most LSST conventions in their headers. Forcing it to be really like LSST means changes in `obs_lsst` and may invalidate existing data, which we don't want. On the other hand PP will only process newly uploaded imsim data and we can overwrite headers for those.
Hsin-Fang Chiang February 11, 2025 at 9:44 PM
Adding notes here as this could bite us later… As of 2025-02-11
The DC2_cells_v1
in /repo/dc2
is named lsst_cells_v1
in other repos; it has the same hash.
TheDC2_cells_v1
in all other repos, including s3://rubin-pp-dev-users/central_repo_2
, have a different hash. Those have a different tractOverlap
in their config.
While imsim templates are generated in /repo/dc2
and now seems too late to correct it for . I think we can live with it somehow, just that it will likely generate confusion.
Hsin-Fang Chiang February 10, 2025 at 9:14 PM
imsim DC2 /repo/dc2
has a different convention and have defaults collections under 2.2i/x
, not LSSTCam-imSim/x
. PP relies on those collection names, so I added chains in pp-dev repo:
Pick a few exposures from
/repo/dc2
runnable by ApPipe, put them to pp-dev’s "unobserved" curation, prepare the pp-dev central repo with needed refcats/calibration/skymap/template data, setup the necessary butler collections, and all other things not yet done in DM-48194 to make this work:upload.py LSSTCam-imSim 2
(or a higher number up to ). Update the playbook accordingly.For this ticket, use
s3://rubin-pp-dev-users/central_repo_2
as the central repo.