feat(api): run the knowledge base and file copies from the migration endpoints

[ogabrielluiz]

Stacked on the database copy PR. relocate-kb --json (#15527) and relocate-files --json (#15525), which it needs, have merged.

This adds the other two copies to the routes that run a copy step: "Copy knowledge bases" and "Copy files". Both can be tried first with a test run. The decisions an admin can make about a copy come in the next PR.

What each step runs. POST /api/v1/migration/steps/{id}/runs now also takes:

• copy_knowledge_bases, which runs langflow relocate-kb --to postgres --json with --verify-skipped or --no-verify-skipped (see below). The vectors go into the destination database as pgvector tables.
• copy_files, which runs langflow relocate-files --json --bucket <bucket> --prefix <prefix>, with the bucket and the prefix the record holds.

{"dry_run": true} adds --dry-run. A test run says what a copy would do and never completes the step. The database copy still has none.

Where they write. Both commands work on the database the new instance will run on, which the database copy filled: LANGFLOW_DATABASE_URL is the destination database, and PGVECTOR_CONNECTION_STRING is the same one. So this instance's own rows stay as they are, and it goes on serving its knowledge bases and files from its own disk.

An instance already on PostgreSQL. It keeps its database, and the two copies work on it. The knowledge bases go to the store the server's own PGVECTOR_CONNECTION_STRING names: the command is given that value unchanged, so it writes where this instance already reads pgvector knowledge bases from, whichever database that is. The copy changes the knowledge_base rows of the database the instance is still serving from, and from then on the instance reads each knowledge base from that store. So before this step the admin starts the server with PGVECTOR_CONNECTION_STRING set, to its own database unless the vectors are to live somewhere else. Without the variable the step reads blocked with pgvector_env_missing, and starting the copy answers 409 with the same code, whatever the step reads. A rehearsal on a real PostgreSQL instance showed what this prevents: a knowledge base that answered 200 before the copy answered 500 after it, turning changes back on did not bring it back, and a restart with the variable set did.

The destination's address is read with the driver its test and the database copy read it with. Taken as it was typed, an address spelled postgresql+asyncpg:// passed the test of the destination and the database copy, and then failed both later copies with a misleading schema_mismatch.

A knowledge base the copy does not read. A copy from SQLite gives relocate-kb the destination as its pgvector store. A knowledge base that this server already keeps in its own pgvector store has a row that says pgvector, so in the copy it names the destination, which holds none of its chunks. The command skipped it as already on the target, and the step read done. relocate-kb now counts first: a knowledge base that would be skipped has its chunks counted in its store, unless its row records none. Fewer chunks than the row records fails it with kb_target_short, and a store that cannot be read fails it with kb_target_unreachable, so the step ends blocked. --no-verify-skipped turns the count off, and the copy always says which it wants. From SQLite it passes --verify-skipped. On an instance on PostgreSQL it passes --no-verify-skipped: there the copy reads the server's own store, and the check step has counted every such row in it.

Such a knowledge base is still not copied, because the copy is given no way to read the server's own store. In this PR the step stays blocked on it, and the decision that gets past it comes with the next PR. An instance with no knowledge base on its own disk has no such step, so one whose knowledge bases are all in pgvector still moves without them. Run by hand, the command counts as well, so it lists such a knowledge base as failed and ends with exit code 1. The page is behind a flag in this release, so the command line is how an instance gets moved, and with the count off by default it skipped that knowledge base with exit code 0. A knowledge base that is in place and whose row records too many chunks fails the same way, and --no-verify-skipped is the way past it. Under a short count the command prints a hint line that names the option. Only the text output has the hint: the page gets the code and the reason, and nobody there types the command.

The file copy and AWS. It signs in with the keys the admin gave: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, plus AWS_ENDPOINT_URL and AWS_CA_BUNDLE when they were given. It gets none of the server's own AWS_ variables, and AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE point at an empty file. So a profile, a session token or an endpoint the server has for its own work cannot redirect the copy. No key is on the command line.

The record. A report's failed items are kept under report.attention, the first hundred of them. report.counts still says how many failed. Every item stays in the run's events. A step whose report has a failed item is blocked with that item's code, for example kb_metric_change or file_conflict.

Order. Each copy is locked until the one before it is done or skipped. Copying the database again writes every row of the destination again, and with them what the other two copies changed there: where each knowledge base is kept, and how chat history names its attachments. So a knowledge base or file copy counts only when it started after the database copy on record. When the database is copied again, the other two read current again.

How to try it

Same $BASE, $M, $H, $J and follow as in the database copy PR, on a SQLite instance with a local knowledge base and an uploaded file, after the database copy. The destination has a database with the vector extension turned on and a bucket, saved through PUT /destinations.

RUN=$(curl -s -X POST -H "$H" -H "$J" -d '{"dry_run": true}' $M/steps/copy_knowledge_bases/runs | jq -r .run_id)
follow copy_knowledge_bases $RUN | jq -c 'if .event == "item" then {event, item: (.item | {kb_name, status, code}), seq} else . end'
# {"event":"item","item":{"kb_name":"handbook","status":"would_relocate","code":null},"seq":1}
# {"event":"report","ok":true,"dry_run":true,"counts":{"would_relocate":1},"attention":[],"seq":2}
# {"event":"end","status":"done","exit_code":0,"seq":3}
curl -s -H "$H" $M | jq -c ".record.steps.copy_knowledge_bases | {status, dry_run, counts: .report.counts}"
# {"status":"done","dry_run":true,"counts":{"would_relocate":1}}
# The step still reads current: a test run completes nothing.

RUN=$(curl -s -X POST -H "$H" $M/steps/copy_knowledge_bases/runs | jq -r .run_id)
follow copy_knowledge_bases $RUN | jq -c 'if .event == "item" then {event, item: (.item | {kb_name, status, copied, source_count}), seq} else . end'
# {"event":"progress","phase":"copying","done":12,"total":12,"unit":"chunks","subject":"4064a0ae-81dd-4cb1-af43-42eb1801e062","seq":1}
# {"event":"item","item":{"kb_name":"handbook","status":"relocated","copied":12,"source_count":12},"seq":2}
# {"event":"report","ok":true,"dry_run":false,"counts":{"relocated":1},"attention":[],"seq":3}
# {"event":"end","status":"done","exit_code":0,"seq":4}

RUN=$(curl -s -X POST -H "$H" -H "$J" -d '{"dry_run": true}' $M/steps/copy_files/runs | jq -r .run_id)
follow copy_files $RUN | tail -2
# {"event": "report", "ok": true, "dry_run": true, "scope": "all users", "counts": {"would_copy": 1}, "bytes": 0, "attention": [], "seq": 5}
# {"event": "end", "status": "done", "exit_code": 0, "seq": 6}
# The bucket is still empty.

RUN=$(curl -s -X POST -H "$H" $M/steps/copy_files/runs | jq -r .run_id)
follow copy_files $RUN | tail -3
# {"event": "progress", "phase": "copying", "done": 1, "total": 1, "bytes": 5, "unit": "files", "seq": 4}
# {"event": "report", "ok": true, "dry_run": false, "scope": "all users", "counts": {"copied": 1}, "bytes": 5, "attention": [], "seq": 5}
# {"event": "end", "status": "done", "exit_code": 0, "seq": 6}

curl -s -H "$H" $M | jq -c '.steps[] | select(.id | startswith("copy") or endswith("target")) | [.id, .state, .reason]'
# ["connect_target","done",null]
# ["copy_database","done",null]
# ["copy_knowledge_bases","done",null]
# ["copy_files","done",null]
# ["start_target","locked","not_available"]
# ["check_target","locked","not_available"]

In the destination the knowledge_base row now says postgres, and the bucket holds files/<user id>/cat.txt. In this instance's own database the same row still says sqlite.

A knowledge base whose vectors are not unit length is not copied until the admin accepts the change of ranking, which the next PR adds. Until then its step reads blocked:

# {"event":"report","ok":false,"dry_run":false,"counts":{"relocated":1,"failed":1},"attention":[{"kb_name":"notes","code":"kb_metric_change","flag":"--allow-metric-change", ...}],"seq":4}
# ["copy_knowledge_bases","blocked","kb_metric_change"]
# ["copy_files","locked","earlier_step"]

Copy the database again, and the two copies after it are to be made again:

RUN=$(curl -s -X POST -H "$H" $M/steps/copy_database/runs | jq -r .run_id); follow copy_database $RUN > /dev/null
curl -s -H "$H" $M | jq -c '.steps[] | select(.id | startswith("copy")) | [.id, .state, .reason]'
# ["copy_database","done",null]
# ["copy_knowledge_bases","current",null]
# ["copy_files","locked","earlier_step"]

How I tested it

The two new files hold 69 cases at this layer. At the top of the stack, with the fixes of every layer in place, the six test files of the migration (routes, pause, copy runs, copies, preflight and knowledge base relocation) give 417 passed and 16 skipped with no service configured, and the copy runs file gives 61 passed and 3 skipped against a local PostgreSQL with pgvector and no S3 keys. I did not start an S3 server for this rebuild.

Every copy in the tests is the real command, run by the route as a child process against a real destination: a PostgreSQL database with pgvector and an S3 bucket of the test's own. The knowledge bases are made the way the app makes them, as SQLite stores with their chunks and vectors.

• Knowledge bases: a test run, then the copy. The destination's row says postgres and its vector table holds every chunk. This instance's row still says sqlite. Then the database is copied again: the destination's row says sqlite once more, the step reads current, and copying the knowledge bases again puts it right.
• Files: a test run that leaves the bucket empty, then the copy. For the copy the server's environment is given a profile that does not exist, a config file with another endpoint, a session token, an endpoint that leads nowhere and LANGFLOW_STORAGE_TYPE=s3. The file still lands in the bucket the admin named. A chat attachment recorded by its path on this instance's disk is renamed in the destination's message.files, and this instance's own row is unchanged.
• A knowledge base that cannot be copied, and a file whose name the bucket already holds with other content, each block their step with the item's code, and the record keeps the item.
• A copy that dies without a report (the destination database is gone) blocks the step as crashed, with the last lines of stderr as the message. This one needs no driver, so it runs in the unit job too.
• The search for secrets now covers all three copies: the database password, the S3 secret key and this instance's secret key are on no command line, in no response, in no file under CONFIG_DIR and in no log.
• What lets a copy in, for the two copies added here: a file copy is refused with secrets_missing when the worker's keys are for another bucket than the saved one, and a knowledge base copy that was made is refused with locked while the database copy before it is blocked again. Both started with a 202 on the published commit.
• A knowledge base kept in this server's own pgvector store: its row says postgres and records 36 chunks, and the destination has no table for it. The test run and the copy both list it as failed with kb_target_short, the step reads blocked, and a second copy skips the knowledge base that was copied and lists the other again. Without the change the test run reported ok with the knowledge base skipped. The unit job covers the command line the page builds for both kinds of source (read by the command's own parser too), what the command does when no option is given, both places where the command skips, a store that cannot be read, the count rule on a SQLite store, and which codes get the hint line. Typed by hand against the same setup, langflow relocate-kb --to postgres --dry-run lists that knowledge base as failed, prints the hint under it and ends with exit code 1, and with --no-verify-skipped it lists it as skipped, prints no hint and ends with 0. That pgvector counts a table that is not there as 0 is checked only by the tests that need PostgreSQL.
• An instance on PostgreSQL, with the environment set by hand and no PostgreSQL server needed. With PGVECTOR_CONNECTION_STRING naming another database than the instance's own, the copy starts, and the environment of the real child, read from the process, carries that value for the vectors and the instance's own address for the database. Without the variable the step reads blocked with pgvector_env_missing, starting it answers 409 and no command starts. The same 409 answers when the step already reads done.

The tests that need PostgreSQL or S3 skip when the driver or the server is missing. The S3 ones carry the api_key_required marker, like the other object storage tests.

I saw every new test fail before the code existed. The rule about copying the database again came from a real run: I copied the database a second time on a real server and found the destination's knowledge base rows pointing back at local disk while the step still read done. The test for it failed in the same way before the fix.

Summary by CodeRabbit

• New Features
  • Migration runs can now copy knowledge bases and file storage, alongside databases, using the selected destination’s settings.
  • Dry runs are available for knowledge-base and file copies. Database dry runs remain unsupported.
  • Relocation can verify knowledge bases already marked as moved. The command enables verification by default; use --no-verify-skipped to skip it.
• Bug Fixes
  • Migration reports now flag knowledge bases with missing or inaccessible chunks and identify blocking failures, helping prevent incomplete copies from being treated as complete.

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Repository: langflow-ai/langflow/.coderabbit.yaml
• Review profile: CHILL
• Plan: Advanced
• Run ID: 6d55907b-18cd-46c8-ae4c-618ebad9d1bd

---

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Walkthrough

Migration runs now support knowledge-base and file copies alongside database copies. The changes add dry-run handling, step-specific destination configuration, copy-step gating, and optional verification of knowledge bases already routed to the target.

Changes

Migration copy workflow

Layer / File(s)	Summary
Knowledge-base relocation verification src/backend/base/langflow/api/utils/knowledge_base_relocation.py, src/backend/base/langflow/__main__.py, src/backend/tests/unit/base/knowledge_bases/test_knowledge_base_relocation.py	Relocation can verify chunks for skipped knowledge bases. It reports unreachable or short target storage, and the CLI provides verification controls and result hints.
Step-specific copy commands and environments src/backend/base/langflow/api/utils/migration_copies.py, src/backend/tests/unit/api/utils/test_migration_copies.py	Copy commands and environments now use step-specific destinations and dry-run settings. File-copy environments use destination S3 credentials, and reports retain up to 100 attention items.
Migration run orchestration and validation src/backend/base/langflow/api/v1/migration.py, src/backend/tests/unit/api/v1/test_migration_copy_runs.py	Migration runs apply dry-run and pgvector checks, track step status and ordering, and execute knowledge-base and file copies. Tests cover copy outcomes, failures, and credential isolation.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant MigrationAPI
  participant copy_environment
  participant copy_command
  participant CopyProcess
  participant Destination
  MigrationAPI->>copy_environment: Build environment for selected step
  MigrationAPI->>copy_command: Build command with destinations and dry-run setting
  MigrationAPI->>CopyProcess: Start copy process
  CopyProcess->>Destination: Copy knowledge-base or file data
  CopyProcess-->>MigrationAPI: Return copy report
  MigrationAPI->>MigrationAPI: Update run and step status

Loading

Merge Risk: 🟡 Moderate · up to e9976

File migration can fail before copying begins when the server uses temporary AWS credentials. Clear the inherited session token before merging.

🚥 Pre-merge checks | ✅ 8 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 30.49% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 82 functions across 7 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (8 passed)

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Test Coverage For New Implementations		The PR includes substantial tests for each new implementation. test_migration_copies.py covers command construction, dry runs, destination environments, PostgreSQL and S3 handling, verification flag…
Test Quality And Coverage		The PR adds substantial pytest coverage for the new migration copy functionality. Unit tests validate command construction, dry-run handling, destination-specific environments, PostgreSQL driver norma…
Test File Naming And Structure		All changed tests use the backend test_*.py pattern under src/backend/tests/unit, with valid pytest functions, fixtures, parametrization, and async test support. Test names are descriptive. The su…
Excessive Mock Usage Warning		PASS: The changed tests do not show excessive mock usage. test_migration_copies.py adds no mocks. The migration endpoint tests run real child processes and exercise PostgreSQL, S3, database rows, fi…
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly summarizes the main change: adding knowledge base and file copy operations to the migration endpoints.

---

---

---

✨ Finishing Touches 💡 2

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

---

---

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

---

---

🛠️ Fix failing CI checks 💡

• Commit to this branch
• Create a new PR

---

---

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

✅ Test Coverage Advisor

No source changes detected without accompanying tests. Thanks for keeping coverage up! 🎉

Advisory check only — never blocks merge.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 68.97%. Comparing base (4435a6a) to head (ca2aff4).

Additional details and impacted files

@@                     Coverage Diff                     @@
##           feat/migration-runs-api   #15611      +/-   ##
===========================================================
- Coverage                    69.88%   68.97%   -0.92%     
===========================================================
  Files                         2777     2775       -2     
  Lines                       299516   299488      -28     
  Branches                     40374    40392      +18     
===========================================================
- Hits                        209325   206572    -2753     
- Misses                       87697    90422    +2725     
  Partials                      2494     2494              

Flag	Coverage Δ
backend	78.70% <100.00%> (+0.02%)	⬆️
frontend	65.46% <ø> (-1.59%)	⬇️
lfx	67.49% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/backend/base/langflow/__main__.py	59.54% <100.00%> (+0.21%)	⬆️
...se/langflow/api/utils/knowledge_base_relocation.py	85.81% <100.00%> (+4.34%)	⬆️
...ackend/base/langflow/api/utils/migration_copies.py	100.00% <100.00%> (ø)
src/backend/base/langflow/api/v1/migration.py	96.94% <100.00%> (-0.10%)	⬇️

... and 310 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[erichare]

Reviewed 1d90561 against the preceding stack PR. The per-step environments isolate the destination credentials, dry runs remain incomplete, and rerunning the database invalidates later copies. No blocking findings introduced by this delta. Focused migration checks across the stack passed (151 tests, 12 skipped and 6 deselected). The live PostgreSQL/S3 cases were not rerun locally.

CI remains an independent gate: Python 3.14 Group 1 failed the unchanged native first-start storage test with chroma/needs_attention instead of sqlite/ready. The affected test and conversion paths are unchanged here. Parallel process-check interference is plausible but unconfirmed. Please resolve that failure before merging.

[erichare]

Thanks, @ogabrielluiz, for keeping copies pointed at the destination database and covering dry runs and rerun invalidation. Approved for this stack delta. The outstanding Python 3.14 native-storage CI failure still needs attention, and the separate copy-admission finding is tracked on #15610.

[github-actions]

Frontend Unit Test Coverage Report

Coverage Summary

Lines	Statements	Branches	Functions
	59.49% (97538/163942)	73.98% (15125/20443)	54.61% (2413/4418)

Unit Test Results

Tests	Skipped	Failures	Errors	Time
7402	0 💤	0 ❌	0 🔥	21m 12s ⏱️

[ogabrielluiz]

Added in ea98973: a hint: line under a kb_target_short item in the text output, out of reason and flag. It also says that the option skips every knowledge base already on the target, since it applies to the whole run. Thanks for the probe.