`symlink_path` has wrong arrow type

[nolta]

The VAST Catalog table declares parent_path, name, and symlink_path as Arrow String. For the first two that type is accurate, because VAST rejects non-UTF-8 filenames at the filesystem layer. For symlink_path it is not: symlink targets are accepted verbatim, whatever bytes they contain.

The result is that table.select() can return a pyarrow.string() array whose contents are not UTF-8, which violates the Arrow columnar specification. Nothing in the read path notices, because validation is off by default, so the invalid data can propagate silently into whatever the consumer writes and can fail much later, in an unrelated process.

Test

import os
import time

import duckdb
from ibis import _
import vastdb

# create a symlink with non-UTF-8 target
BAD = b"\x89HBF\r\n\x1a\n\x02\x08\x08"
os.symlink(BAD, b"/scratch/<user>/bad.symlink")

# wait for catalog to be updated
time.sleep(1000)

# search catalog and write symlink_path to parquet
session = vastdb.connect(...)
with session.transaction() as tx:
    predicate = (_.search_path == '/scratch') & (_.parent_path == '/scratch/<user>/')
    batches = tx.catalog().select(columns=['symlink_path'], predicate=predicate)
    con = duckdb.connect()
    con.execute("COPY (SELECT * FROM batches) TO 'catalog.parquet' (FORMAT parquet)")

# raises Invalid Input Error if any symlink_path in the catalog is not valid UTF-8
duckdb.connect().execute(
    "SELECT count(symlink_path) FROM read_parquet('catalog.parquet')"
).fetchone()

Workaround

Cast symlink_path to binary:

import pyarrow as pa

def fix_symlink_path_type(reader: pa.RecordBatchReader) -> pa.RecordBatchReader:
    schema = pa.schema([
        f.with_type(pa.binary()) if f.name == 'symlink_path' else f
        for f in reader.schema
    ])
    return pa.RecordBatchReader.from_batches(
        schema, (b.cast(schema) for b in reader)
    )

batches = fix_symlink_path_type(table.select(...))