Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

arrow_to_numpy function drops mask information

Description

The Astropy VOTable interface requires a numpy MaskedArray. Currently in dax_obscore I’m creating astropy masked columns, creating an astropy table, and then extracting the array and mask to make a MaskedArray. Ideally this could be done without using astropy Table.

It was suggested that I try arrow_to_numpy in daf_butler. This returns a recarray that drops mask information:

whereas the version I want needs to be masked:

It seems that numpy recarray does not support masking at all. The mask information is available in arrow_to_numpy_dict.

 

Additionally I see there is a bug if there are no valid values in a string column and I needed the following diff:

Issue Matrix

hide

Activity

Tim Jenness September 30, 2024 at 9:19 PM

Turned out to be trivial in the end. Code coverage indicates that we are now getting masked and unmasked numpy records out although I'm still looking at the test code trying to work out where the unmasked ones are coming from and what tests we want (other than an `isinstance`).

Done
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Labels

Reviewers

Eli Rykoff

Story Points

RubinTeam

Ops Middleware

Components

Parent

Checklist

Created September 30, 2024 at 6:29 PM
Updated January 31, 2025 at 11:38 PM
Resolved October 1, 2024 at 3:25 AM