Skip to content

Browser side ES client - sends request through new general purpose proxy endpoint - #246293

Open
mattkime wants to merge 61 commits into
mainfrom
es_proxy
Open

mattkime wants to merge 61 commits into
mainfrom
es_proxy

Conversation

@mattkime

@mattkime mattkime commented Dec 14, 2025 •

Copy link
Copy Markdown
Contributor

Summary

Provides a client side elasticsearch client. The client uses a generic Elasticsearch proxy exposed as the internal API /internal/es_proxy/{path*}, and moves Index Management's index list calls onto it to verify its functionality.

esClient plugin

  • Browser: the contract offers getClient(), which lazily loads an elasticsearch-js API client that sends every call through the proxy.
  • Server: /internal/es_client/<es path>?<query> is sent to /<es path>?<query> on Elasticsearch as the current user, and the response is streamed back. Kibana authorization is disabled on the route; Elasticsearch enforces index and cluster privileges, so any authenticated user can call any Elasticsearch API their role allows. Writes are never retried, requests time out after 120 seconds (504), and only a small allowlist of headers is forwarded. Kibana's request id reaches Elasticsearch as x-opaque-id.

Examples to try in Dev Tools:

GET kbn:/internal/es_client

PUT kbn:/internal/es_client/my-index-000001/_create/1
{
  "@timestamp": "2099-11-15T13:12:00",
  "message": "GET /search HTTP/1.1 200 1070000",
  "user": {
    "id": "kimchy"
  }
}

Incompatible use cases

  • Any time user permissions are insufficient. The proxy assumes normal user permissions and there's no way to change this.
  • Streaming request bodies to kibana or sending large payloads. This could potentially be addressed but I've decided to limit the scope of this PR as this is a less common use case.

Index Management

These index list calls (12 of them) now go through the es_proxy client instead of Index Management server routes:

  • the index actions: close, open, delete, refresh, flush, clear cache and forcemerge;
  • creating an index;
  • loading mappings, settings and stats;
  • updating mappings;
  • the document sample, the document counts (an ES|QL query) and the user privileges check;
  • the browser index stats enricher.

The server routes those calls used are removed, along with their helpers and tests. These stay on the server: the indices list and indices_get (serverless metering), reload, single-index load, vector count, data streams, GET mapping (used by other plugins) and PUT settings.

Bundle size

The async bundle size is large at 480k but compresses down to about 30k over the wire which is acceptable.

@elasticmachine

elasticmachine commented Dec 14, 2025 •

Copy link
Copy Markdown
Contributor
🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

  • Click to trigger kibana-pull-request for this PR!
  • Click to trigger kibana-deploy-project-from-pr for this PR!
  • Click to trigger kibana-deploy-cloud-from-pr for this PR!
  • Click to trigger kibana-entity-store-performance-from-pr for this PR!
  • Click to trigger kibana-storybooks-from-pr for this PR!
  • Click to trigger kibana-static-checks for this PR!

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

🔍 Preview links for changed docs

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!


The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

@jbudz jbudz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pnpm-workspace.yaml LGTM

Comment on lines +56 to +58
// Without a client value, the empty header clears the scoped client's Kibana-origin default;
// Elasticsearch treats it as no origin.
[PRODUCT_ORIGIN_HEADER]: getHeader(request.headers, PRODUCT_ORIGIN_HEADER) ?? '',

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We cannot trust the client to send x-elastic-product-origin on an endpoint such as this. The header controls what Elasticsearch allows the caller to do, and we do not want arbitrary consumers to set this. Requests from this API should not set this header, since they are not originating from Kibana.

Specifically, specifying this header:

  • Disables the .-prefixed block on indices and index templates
  • Grants external system-index access
  • Omits associated deprecation logs from the upgrade assistant

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first two items would mean that this couldn't be used for anything index or index template related.

The last item seems like it simply won't work for any use case.

There's much I don't understand about this but relying on x-elastic-product-origin set to kibana seems like keeping api users honest rather an actual security measure. I guess I'm unaware of how much we rely on kibana as a security layer in from of ES.

Comment on lines +57 to +58
const hasDotSegment = (path: string): boolean =>
path.split('/').some((segment) => DOT_SEGMENT.test(segment));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be bypassed as written. I recommend using the WHATWG parser instead. Something like this:

Suggested change
const hasDotSegment = (path: string): boolean =>
path.split('/').some((segment) => DOT_SEGMENT.test(segment));
const PROXY_PATH_ORIGIN = 'http://kibana.invalid';
/**
* Returns true when the browser would request a different path than the one given,
* for example by resolving "." or ".." segments (even percent-encoded), treating "\" as "/",
* or stripping tabs and newlines.
*/
const isAlteredByUrlParsing = (path: string): boolean => {
const requested = `${ES_CLIENT_BASE_PATH}${path}`;
return new URL(requested, PROXY_PATH_ORIGIN).pathname !== requested;
};

And then down below in request():

if (isAlteredByUrlParsing(path)) {
  throw new Error(
    `The Elasticsearch proxy cannot send ${path}: the browser would request a different path.`
  );
}

return headers;
};

export const createProxyHandler =

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will need to add some bespoke CSRF handling to this endpoint. GET and HEAD requests are considered "safe" in the standard, but we do not want to support cross-origin requests to Elasticsearch in this manner. So you can't rely on our platform controls at this point.

Please add a check to this handler which rejects the request if it lacks the x-elastic-internal-origin header:

if (!('x-elastic-internal-origin' in request.headers)) {
  return response.badRequest({ body: { message: 'Missing x-elastic-internal-origin header' } });
}

const method = request.route.method.toUpperCase();
const body =
Buffer.isBuffer(request.body) && request.body.length > 0 ? request.body : undefined;
const path = getEsPath(request.url);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to see us land this initial version with an explicit list of paths/patterns that we need to support. This will reduce the blast radius when an inevitable security scanner flags this endpoint. I have no objection to this becoming a widely used endpoint, but I would prefer that we start with a known set of paths, and expand it as the need arises. This has a side benefit of softening the gap this introduces with respect to the Dev Tools feature privilege.

I also want an explicit deny list as well that can work in tandem. For example, I wouldn't want the /_security/* endpoints to be made available here. There just isn't a need for it, as we need plugins to use Kibana's own security abstractions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but I would prefer that we start with a known set of paths, and expand it as the need arises

What's the process for expanding? What are the guidelines regarding what is allowed vs what isn't?

@jesuswr jesuswr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from a core pov. My only concern was already raised in #246293 (comment)

mattkime and others added 3 commits November 19, 2026 23:19
…306623

Restore the index_doc_count and documents sample routes and their callers so
this PR only contains the es_client changes, and commit the user status route
files restored when reverting the security API usage.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
*/
export type EsClient = EsClientApi & {
// The generated API methods require `this.transport` to be an elasticsearch-js `Transport`.
readonly transport: EsClientTransport & Transport;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: P3 (Low)

The public contract exposes the entire Node Transport, but the browser implementation only provides request. A consumer can use members such as diagnostic without a type error and then fail at runtime. Keep the Node transport cast private to the generated API adapter.

Generated by Libra. React with 👍 or 👎 to give feedback on this comment.

Comment on lines +31 to +34
const description =
'Forwards the request method, path, query string and body to Elasticsearch as the current user ' +
'and streams the response back. GET and HEAD requests cannot carry a body; use POST for ' +
'search-style APIs such as `_search`, `_count` and `_msearch`.';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: P3 (Low)

The route description tells callers to use POST for _search and _msearch, but neither path is in ALLOWED_PATHS, so the proxy rejects both with a 403 before calling Elasticsearch. The description is published as the route's OAS summary, so anyone reading it will try requests that can't succeed. Mention only allowed APIs like _count, or say that paths are limited to the allowlist.

Generated by Libra. Specialist: libra-complexity. React with 👍 or 👎 to give feedback on this comment.

Comment on lines +173 to +177
return response.custom({
statusCode: error.statusCode ?? 500,
body: error.body,
// The transport may have read and parsed the error body, so its original framing no longer applies.
headers: getResponseHeaders(error.headers ?? {}, proxyBasePath, {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: P2 (Medium)

If credential refresh cannot recover a streamed Elasticsearch 401, passing the parsed ES error object to response.custom turns it into a 500. Kibana’s error adapter expects an object with message, but ES returns { error, status }. Preserve the upstream body with bypassErrorFormat so Kibana’s browser session interceptor receives the 401.

Generated by Libra. React with 👍 or 👎 to give feedback on this comment.

Comment on lines +20 to +21
export const FORBIDDEN_INDEX = 'es-client-forbidden';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: P3 (Low)

These fixed index names make Scout setup and teardown delete indices they may not own on a shared deployment. Overlapping runs can also delete one another’s fixtures. Use a run-unique prefix for both indices and derive the scoped role and cleanup names from it.

Generated by Libra. React with 👍 or 👎 to give feedback on this comment.

await esClient.indices.delete({ index: INDEX_NAME }, { ignore: [404] });
});

apiTest('fetches an index settings', async ({ apiClient }) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: P3 (Low)

Keep a check that a fresh index returns default settings through the new read path on both stateful and Serverless. The replacement proxy tests check index names and response format, but not defaults, which the settings editor uses to populate editable fields. A regression that drops those values could leave the editor incomplete while the Scout tests still pass.

Generated by Libra. Specialist: scout. React with 👍 or 👎 to give feedback on this comment.

@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

💔 Build Failed

Failed CI Steps

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
esClient - 63.0KB +63.0KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
esClient - 723.0B +723.0B
kbnUiSharedDeps-srcJs 5.5MB 5.5MB -17.0B
shared-plugins 12.4MB 12.4MB +1.9KB
vendors-heavy 2.0MB 2.0MB -34.0B
total +2.5KB
Unknown metric groups

@kbn/rspack-optimizer bundle module count

id before after diff
esClient - 2 +2
shared-plugins 4088 4089 +1
total +3

async chunk count

id before after diff
esClient - 1 +1

shared async chunks total size

id before after diff
all 16.8MB 16.8MB +193.0B

shared chunks total size

id before after diff
all 7.8MB 7.8MB +131.0B

total optimizer output size

id before after diff
all 69.1MB 69.2MB +65.9KB

History

cc @mattkime

This branch was successfully deployed

1 active deployment
docs-preview — 5fffb1ae Deployed Nov 20, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting Feature:Index Management Index and index templates UI release_note:skip Skip the PR/issue when compiling release notes Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more t//

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants