Skip to content

HIVE-29878: Support pagination for list endpoints in HMS REST Catalog and remove redundant type casting in REST Catalog - #6750

Open
Aggarwal-Raghav wants to merge 4 commits into
apache:masterfrom
Aggarwal-Raghav:HIVE-29878
Open

Aggarwal-Raghav wants to merge 4 commits into
apache:masterfrom
Aggarwal-Raghav:HIVE-29878

Conversation

@Aggarwal-Raghav

@Aggarwal-Raghav Aggarwal-Raghav commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR updates HMSCatalogAdapter to correctly extract pageToken and pageSize query parameters for the listNamespaces, listTables, and listViews REST routes.
Additionally, this PR performs several architectural cleanups to modernize the adapter:

  • Removed the redundant castResponse method, leveraging Java generics.
  • Replaced the deprecated NAMESPACE_SPLITTER and decodeNamespace(String) with modern Iceberg 1.11.0 equivalents (namespaceFromQueryParam and decodeNamespace(..., "%1F")).
  • Extracted 'pageToken', 'pageSize', and 'parent' string literals into standard constants.

Why are the changes needed?

Iceberg Rest Catalog Spec supports it as follows and this helps in catalog contains large number of tables.

  1. https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L250-L269
  2. https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L525-L538

Does this PR introduce any user-facing change?

NO

How was this patch tested?

Added TestHMSCatalogAdapterPagination JUnit test

@Aggarwal-Raghav

Copy link
Copy Markdown
Contributor Author

CC @deniskuzZ , can you help with review?

@Aggarwal-Raghav

Copy link
Copy Markdown
Contributor Author

gentle ping for review @deniskuzZ , have handled your review comments.

Class<T> responseType,
Supplier<Object> unpaginatedTask,
BiFunction<String, String, Object> paginatedTask) {
String pageToken = PropertyUtil.propertyAsString(properties, PAGE_TOKEN, null);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only listNamespaces, listTables, and listViews and listFunctions supports pagination based on iceberg spec.

@Aggarwal-Raghav Aggarwal-Raghav changed the title HIVE-29878: HMS REST Catalog ignores pagination for listTables and listNamespaces HIVE-29878: Support pagination for list endpoints in HMS REST Catalog and remove redundant type casting in REST Catalog Sep 23, 2026
@Aggarwal-Raghav

Copy link
Copy Markdown
Contributor Author

This is ready from my side!

namespace = Namespace.empty();
}
return castResponse(ListNamespacesResponse.class, CatalogHandlers.listNamespaces(asNamespaceCatalog, namespace));
return paginateIfRequested(

@deniskuzZ deniskuzZ Sep 24, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we simplify it further?

/**
 * Paging parameters of a list request. Without pageSize the listing is unpaged, expressed as a
 * single unbounded first page. This relies on CatalogHandlers.paginate treating a null token as
 * the first page and returning a null next-page-token once the list is exhausted.
 */
private record PageRequest(String token, String size) {
  private static final PageRequest UNPAGED =
      new PageRequest(null, String.valueOf(Integer.MAX_VALUE));

  static PageRequest from(Map<String, String> vars) {
    String size = vars.get(PAGE_SIZE);
    if (size == null) {
      return UNPAGED; // token ignored, as upstream; keeping it would overflow token + MAX_VALUE
    }
    Preconditions.checkArgument(NumberUtils.toInt(size, 0) > 0,
        "Invalid %s: %s, must be a positive integer", PAGE_SIZE, size);
    return new PageRequest(vars.get(PAGE_TOKEN), size);
  }
}

private ListNamespacesResponse listNamespaces(Map<String, String> vars) {
  Namespace parent = vars.containsKey(PARENT)
      ? RESTUtil.namespaceFromQueryParam(vars.get(PARENT))
      : Namespace.empty();
  PageRequest page = PageRequest.from(vars);
  return CatalogHandlers.listNamespaces(asNamespaceCatalog, parent, page.token(), page.size());
}

private ListTablesResponse listTables(Map<String, String> vars) {
  Namespace namespace = namespaceFromPathVars(vars);
  PageRequest page = PageRequest.from(vars);
  return CatalogHandlers.listTables(catalog, namespace, page.token(), page.size());
}

private ListTablesResponse listViews(Map<String, String> vars) {
  Namespace namespace = namespaceFromPathVars(vars);
  PageRequest page = PageRequest.from(vars);
  return CatalogHandlers.listViews(asViewCatalog, namespace, page.token(), page.size());
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Thanks for your repeated guidance on this

Assertions.assertNotNull(res1, "Response should not be null");
Assertions.assertEquals(5, res1.namespaces().size(), "Should return all 5 unpaginated");

// 2. Both pageToken and pageSize (should call paginated and slice without errors)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as in prev PRs, please split into individual test-cases

@deniskuzZ deniskuzZ left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, but please address the test refactor comment

…atalog

- Removed redundant castResponse method and BadResponseType exception
- Replaced deprecated NAMESPACE_SPLITTER with RESTUtil.namespaceFromQueryParam
- Extracted 'parent' string literal into PARENT constant
@sonarqubecloud

Copy link
Copy Markdown

@Aggarwal-Raghav

Copy link
Copy Markdown
Contributor Author

@deniskuzZ
CI has degraded a lot over last few weeks, lot of pods are getting killed and time taken to run CI has increased a lot. Is it just parallelism due to large numeber of PR's?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants