ENH: Add VOParquet support for TAP queries

[stvoutsin]

I am wondering if it would be worth adding support for VOParquet (https://www.ivoa.net/documents/Notes/VOParquet/20250116/NOTE-voparquet-1.0-20250116.html)?

This is supported as far as I can see in astropy, see:

job = AsyncTAPJob.create(
    service.baseurl, query,
    RESPONSEFORMAT="application/vnd.apache.parquet",
    session=session
)
job = job.run().wait()
response = session.get(job.result_uri, stream=True)
table = Table.read(io.BytesIO(response.content),
                   format="parquet.votable")

At Rubin we do currently support VOParquet output and are generally eager to start advertising that option more because of the noticeable performance improvements it gives us, so it would be nice if this was possible through PyVO.

If folks think this is worth adding I'm happy to work on this if no one else already was planning to do so.

[bsipocz]

At Rubin we do currently support VOParquet output and are generally eager to start advertising that option more because of the noticeable performance improvements it gives us, so it would be nice if this was possible through PyVO.

What aspects would you think should live in pyvo as opposed in upstream? As I see this is more analogous to the xml parsing we do in astropy, so ideally all of the components should live there.

(should we chat about it 1)here, 2) on slack or 3) during the interop?)

[stvoutsin]

I will be attending the Interop, so happy to chat about it there.
A preview of what I'm thinking is that indeed the parsing is handled by Astropy, and PyVO handles the transport layer:

• Detecting the content-type of the response and routing to the right method
• If content type is Parquet, route to Table.read(..., format="parquet.votable") instead of votableparse.
• Changing run_sync/run_async to allow setting the right responseformat

[andamian]

It is a sensible thing to do IMO even if VOParquet is not a REC since the actual implementation is upstream.

[stvoutsin]

I've done a quick dive into the relevant parts of PyVO and there are at least three options worth considering, plus the approach that I'm drafting on this branch (main...stvoutsin:pyvo:voparquet-support).

Option 1 return a different type from execute()

The simplest approach is to return a VOParquetResults subclass of TAPResults for Parquet and TAPResults as usual for VOTable.
Each class stays simple and TAPResults doesn't need to know Parquet exists.
The problem with this one is that execute() now has two possible return types, and callers handling both formats need to branch on isinstance, and shared behaviour like broadcast_samp has to live somewhere both classes can reach.

Option 2 - unified TAPResults with internal dispatch

Keep TAPResults as the single return type and handle the dispatch internally.
TAPResults.__init__ detects whether it's been given an astropy Table (Parquet path) or a VOTableFile and routes accordingly.
DALResults.__init__ gets a bypass keyword for supplying a pre-built table, skipping _findstatus/_findresultstable/_findinfos.
AdhocServiceResultsMixin probably needs a guard to skip the adhoc service scan for the non-VOTable path.
This one is not perfect either, there's an isinstance check in a constructor and DALResults gains a branch which slightly conflicts with how it is designed around VOTable.

Option 3 - formal backend abstraction

DALResults takes a backend object (_VOTableBackend or _ParquetBackend) that encapsulates all discovery logic, and __init__ just reads from it.
Adding a new format means just writing a new backend class.
The catch here is that _findstatus, _findresultstable, _findinfos are currently overridable instance methods, so moving them into a backend breaks that contract for subclasses.
AdhocServiceResultsMixin would also need restructuring. This seems cleaner but is a larger refactor that seems riskier.

What I've gone with (Which I'm not loving either):

The VOParquet spec requires the Parquet file to carry a valid VOTable document in its metadata (IVOA.VOTable-Parquet.content). Since it must be schema-valid VOTable, it can carry INFO elements and RESOURCE elements including DataLink service descriptors (though the spec isn't explicit about this, so I need to confirm).

We parse that embedded VOTable ourselves and pass it straight to DALResults.__init__, so _findstatus, _findinfos, and AdhocServiceResultsMixin all work fine on it without any changes.
The only addition to TAPResults is a _findresultstable override that returns a _TableResultsAdapter.This uses the FIELD elements from the embedded VOTable's TABLE directly for column metadata, and wraps the Parquet astropy Table in a small array adapter to handle the row-access interface that DALResults and Record want.

DALResults and AdhocServiceResultsMixin require no changes as far as I can see with this approach.

Side note: astropy's VOParquet reader (parquet.votable format) doesn't currently do antyhing with INFO elements or DataLink service descriptors from the embedded VOTable metadata.

In my approach I work around this by reading the embedded XML directly from the Parquet file metadata via pyarrow and parsing it with votableparse.
However if astropy is patched to expose the full embedded VOTable (assuming the VOParquet note allows for this), then the helper methods I have for this go away.

This approach introduces some isinstance branching however in TAPResults which makes me think it is not the cleanest option.

Update: A (probably) improved approach to the one linked here that removes the need for the adapters, by adding internal variables to store the table and fields in DALResults:
main...stvoutsin:pyvo:voparquet-support
Note that there is some increased complexity in the parsing stage, because the astropy parsed Table does not give us the parsed metadata-only VOTableFile so we have to reparse it. If astropy was patched to include it in some internal store, than at least we can remove that additional layer of complexity.

We can discuss this in more detail during the interop.

[msdemlei]

On Thu, Jun 04, 2026 at 05:58:45PM -0700, stvoutsin wrote: In my approach I work around this by reading the embedded XML directly from the Parquet file metadata via pyarrow and parsing it with `votableparse`. However if astropy is patched to expose the full embedded VOTable (assuming the VOParquet note allows for this), then the helper methods I have for this go away.

I think that's the right thing to do. Let's keep assuming regular VOTable responses, perhaps with some benign hacks when the VOTable metadata comes in unusual places. *If* we really want first-class non-VOTable responses (and I don't think we want that), I'd regard that as a matter of TAP 2.0. There's so much we'd then need to think about, and I have a hard time imagining fully working TAP on top of CSV, say.

We can discuss this in more detail during the interop.

Has that happened? I, for one, am in favour of this going ahead if this actually interoperates with what the services (I know AIP is doing something like this, too) do.