Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

fix pandaDistributionEndpoint to support different protocols

Description

Currently in bps_panda_DF.yaml, fileDistributionEndPoint: "file://${LSST_RUN_TEMP_SPACE}/{operator}/panda_cache_box/{payloadFolder}/{uniqProcName}/" is forced to be 'file' protocol. It will raise errors for cases like LSST_RUN_TEMP_SPACE="https://ceph-svc16.gridpp.rl.ac.uk:1094/lsst:datadisk/job_outputs/users/$USER"

to fix it:

(1) set fileDistributionEndPoint: "${LSST_RUN_TEMP_SPACE}/{operator}/panda_cache_box/{payloadFolder}/{uniqProcName}/"

(2) in here , add lines to check if LSST_RUN_TEMP_SPACE contains protocols like “://“(or urlparse). If it doesn’t contain protocol part, add “file://“ to it.

Issue Matrix

hide

Activity

Michelle Gower August 13, 2024 at 7:52 PM

Still missing the doc/changes file. And need a docstring updated. Otherwise good to merge.

Zhaoyu Yang August 13, 2024 at 1:03 PM

Hi Michelle, I’ve made the new commits and the tests at all sites look good. Please help review it.

Michelle Gower August 12, 2024 at 7:17 PM

Let me know when you are ready for me to look at it. It also needs a doc/changes file.

Zhaoyu Yang August 12, 2024 at 3:41 PM

Thanks Tim! I’ll work on that and do more tests.

Tim Jenness August 12, 2024 at 3:22 PM

I’ve had a quick look at the PR and I think it needs to be modified to:

  • Use ResourcePath immediately without any special scheme checks.

  • Change the copy utility function to take a ResourcePath and fix that function so that it doesn’t use os.path.join on the URI but uses the .join() method on the ResourcePath.

Done
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Reviewers

Michelle Gower

RubinTeam

Data Facility

Components

Checklist

Created August 6, 2024 at 2:08 PM
Updated August 14, 2024 at 2:18 PM
Resolved August 14, 2024 at 2:18 PM