Remove structured-output data mapping pipeline and strengthen data-map skill#2377
Open
VellummyilumVinoth wants to merge 1 commit into
Open
Conversation
Contributor
|
Warning Review limit reached
More reviews will be available in 9 minutes and 36 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more credits in the billing tab to continue. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (23)
📝 WalkthroughWalkthroughThis PR removes the legacy data-mapper AI flow, switches record and requirements processing to attachment-based prompts, and updates the ChangesData-mapper orchestration removal
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related issues
Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/orchestrator.ts`:
- Line 17: Remove the unused `keywords` symbol from the import statement that
imports from "`@wso2/ballerina-core`". The `keywords` import is not referenced
anywhere in the file and should be deleted from the destructured import list
alongside DMModel, SourceFile, Command, ProcessContextTypeCreationRequest, and
CodeContext.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 16523103-8c93-4fee-9d69-c36332300f5b
📒 Files selected for processing (11)
workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/context-api.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/index.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/orchestrator.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/mapping-prompt.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/repair-prompt.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/schema.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/types.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/code-generation.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/model.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/temp-project.tsworkspaces/ballerina/ballerina-extension/src/rpc-managers/ai-panel/utils.ts
💤 Files with no reviewable changes (6)
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/code-generation.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/schema.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/index.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/repair-prompt.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/model.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/context-api.ts
VellummyilumVinoth
force-pushed
the
refactor-datamap-old
branch
from
f7a162b to
3578771
Compare
VellummyilumVinoth
changed the title
VellummyilumVinoth
changed the base branch from
release/ballerina-5.12.1
to
release/ballerina-5.12.x
VellummyilumVinoth
force-pushed
the
refactor-datamap-old
branch
from
3578771 to
9b3d1da
Compare
RNViththagan
left a comment
RNViththagan
left a comment
Member
There was a problem hiding this comment.
Shall we move the prompt content to a data-mapper/skill.md file? It's mostly a big text block now, so a markdown file would be a bit cleaner to work with.
Also, shall we clean up any leftover bits in the data-mapper/ folder as a follow-up?
VellummyilumVinoth
force-pushed
the
refactor-datamap-old
branch
3 times, most recently
from
d7b698f to
de97766
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/context-api.ts (1)
26-27: 🎯 Functional Correctness | 🟠 MajorUse UTF-8-safe base64 encoding for text input.
btoa(request.text)throws on non-Latin-1 text, so pasted Unicode requirements can fail before processing. UseBuffer.from(request.text, 'utf8').toString('base64')instead.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/context-api.ts` around lines 26 - 27, The text-input branch in context-api.ts uses btoa(request.text), which is not UTF-8 safe and can fail on Unicode content. Update the request.text handling in the context-api flow to base64-encode using a UTF-8-safe approach such as Buffer.from(..., 'utf8').toString('base64') before passing the file payload to processFiles, so non-Latin-1 text is handled correctly.
🧹 Nitpick comments (1)
workspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/registry.ts (1)
64-72: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winValidate
skill-creatorfrontmatter before registration.
parseSkillMdreturns empty strings when required fields are missing; unlikedata-map, this registers an unusable built-in skill instead of failing fast.Proposed fix
const skillCreator = parseSkillMd(skillCreatorMd); +if (!skillCreator.name || !skillCreator.description) { + throw new Error(`[skill-creator] SKILL.md is missing required frontmatter fields (name="${skillCreator.name}", description="${skillCreator.description}")`); +} export const skillCreatorSkill: Skill = {
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/types.ts`:
- Around line 44-51: `AttachmentProcessRequest` currently exposes both
`processType` and the optional `isRequirementAnalysis`, which lets requirements
requests fall through to the wrong parser. Update the flow in `processFiles` and
the related types in `types.ts` so parsing is derived directly from `processType
=== "requirements"` instead of relying on `isRequirementAnalysis`. Make sure the
requirement-specific extraction path is selected wherever the request is
handled, especially in `context-api.ts`, and remove or stop depending on the
redundant flag.
---
Outside diff comments:
In
`@workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/context-api.ts`:
- Around line 26-27: The text-input branch in context-api.ts uses
btoa(request.text), which is not UTF-8 safe and can fail on Unicode content.
Update the request.text handling in the context-api flow to base64-encode using
a UTF-8-safe approach such as Buffer.from(..., 'utf8').toString('base64') before
passing the file payload to processFiles, so non-Latin-1 text is handled
correctly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 39b11183-cceb-4ae4-981c-61dbe6f60b73
📒 Files selected for processing (21)
workspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/constants.tsworkspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/context.tsworkspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/data-map/SKILL.mdworkspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/index.tsworkspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/registry.tsworkspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/utils.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/constants.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/context-api.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/index.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/orchestrator.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/attachment-prompts.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/mapping-prompt.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/repair-prompt.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/schema.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/types.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/code-generation.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/model-optimization.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/model.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/temp-project.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/types-generation.tsworkspaces/ballerina/ballerina-extension/src/rpc-managers/ai-panel/utils.ts
💤 Files with no reviewable changes (10)
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/constants.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/repair-prompt.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/code-generation.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/types-generation.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/temp-project.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/index.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/model-optimization.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/schema.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/mapping-prompt.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/model.ts
✅ Files skipped from review due to trivial changes (3)
- workspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/constants.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/data-map/SKILL.md
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/attachment-prompts.ts
VellummyilumVinoth
force-pushed
the
refactor-datamap-old
branch
from
de97766 to
9577559
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@workspaces/ballerina/ballerina-visualizer/src/views/AIPanel/components/AIChatInput/index.tsx`:
- Around line 345-354: The template insertion branch in AIChatInput is
incorrectly gated on updatedContent.tagParams, which prevents template-capable
skills from using insertSkill when params are missing. Update the conditional
around the skillEntry.commandTemplates check so template-capable skills activate
based on commandTemplates alone, and only pass tagParams into insertTextAtCursor
when updatedContent.tagParams is present; use the insertSkill and
insertTextAtCursor flow in AIChatInput to locate the fix.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 9b649527-d115-4d8b-82dc-65b96c9a6c28
📒 Files selected for processing (23)
workspaces/ballerina/ballerina-core/src/rpc-types/ai-panel/interfaces.tsworkspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/constants.tsworkspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/context.tsworkspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/data-map/SKILL.mdworkspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/index.tsworkspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/registry.tsworkspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/utils.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/constants.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/context-api.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/index.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/orchestrator.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/attachment-prompts.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/mapping-prompt.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/repair-prompt.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/schema.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/types.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/code-generation.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/model-optimization.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/model.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/temp-project.tsworkspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/types-generation.tsworkspaces/ballerina/ballerina-extension/src/rpc-managers/ai-panel/utils.tsworkspaces/ballerina/ballerina-visualizer/src/views/AIPanel/components/AIChatInput/index.tsx
💤 Files with no reviewable changes (10)
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/schema.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/repair-prompt.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/model-optimization.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/temp-project.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/types-generation.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/mapping-prompt.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/constants.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/model.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/index.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/utils/code-generation.ts
✅ Files skipped from review due to trivial changes (2)
- workspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/constants.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/context.ts
🚧 Files skipped from review as they are similar to previous changes (9)
- workspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/data-map/SKILL.md
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/prompts/attachment-prompts.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/registry.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/index.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/types.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/context-api.ts
- workspaces/ballerina/ballerina-extension/src/rpc-managers/ai-panel/utils.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/agent/skills/utils.ts
- workspaces/ballerina/ballerina-extension/src/features/ai/data-mapper/orchestrator.ts
VellummyilumVinoth
force-pushed
the
refactor-datamap-old
branch
from
9577559 to
1c06785
Compare
VellummyilumVinoth
force-pushed
the
refactor-datamap-old
branch
from
1c06785 to
19b9ed0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The old data mapping pipeline used a series of direct Claude API calls with a structured-output prompt (getDataMappingPrompt) to generate and repair mappings. This has been fully superseded by the agent-based /data-map skill, which handles mapping generation directly in the editor using getDataMappingSkillContent.
This PR removes all dead code from the old pipeline — generation, repair, and file-based hint extraction — and strengthens
the skill instructions so the agent reliably produces correct Ballerina data mapper functions (expression body =>, correct placement under "Data Mappers", proper let usage for sub-mappings, and full coverage of the technical rules the old prompt enforced).
It resolves: wso2/product-integrator#1763
Goals
Approach
UI Component Development
npm run storybookfrom the root directory to view current components.Manage Icons
User stories
Release note
Documentation
Training
Certification
Marketing
Automation tests
Security checks
Samples
Related PRs
Migrations (if applicable)
Test environment
Learning
Summary by CodeRabbit
Summary by CodeRabbit