Thank you everyone for the reviews and testing. I’ve now merged this very large branch to main. Next I have to put on PyPI and then add support for ID.

Looks great, few comments on PR.

I think the code is ready for review for real now. I’ve made many changes since the last review and added support for validating all SIAv2 parameters even if they aren’t all used. I think the code is at least usable now and we should merge it so we can start putting it on PyPI.

• I still batch, pending a decision on what batch size we should use is MAXREC is not given.

• COLLECTION and ID are not handled since I think they might need to be handled initially by the server code (since both those parameters contain information on which butler to use but the dax_obscore API is given a single butler). I also need to add a butler URI parser so that somebody can reliably extract the butler label and UUID.

• I did rewrite the VOTable construction code so it’s much faster and uses arrow_to_numpy (there is a related butler ticket which I will merge tomorrow).

• I have added a bit of caching (eg parsing the felis data model) but we might need to add caching for the instruments and band/filter information so that we don’t continue to ask butler for them every single time (but maybe butler will cache those internally).

• The biggest overhead is now in the queries themselves and the RecordBatch creation.

• I correctly report OVERFLOW if MAXREC is specified and I started adding the query-builder warning messages to the VOTable INFO section.

Right - I think there are multiple clean but disruptive solutions that involve changing a lot of code at many layers, but also much more pragmatic solutions that do a bit of special-casing in the service layer only (taking advantage of the fact that the number of actual cameras we need to support in those services is small and bounded), and I just don’t see us being able to prioritize the former.

can comment further, but I don’t think he believes that we can get “instrument” in as a dimension for coadds any time soon, especially if you need the timespan issue resolved with a butler infrastructure change.