[feat] add rsLoRA support to MultiLoRA

[xichengpro]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

Write the detail information belongs to this PR.

Experiment results

Paste your experiment result here(if needed).

[gemini-code-assist]

Code Review

This pull request implements Rank-Stabilized LoRA (rsLoRA) scaling logic across the linear, embedding, and Megatron forward methods. The reviewer recommends simplifying the scaling calculation with a ternary operator and abstracting the logic to reduce code duplication. A bug was also identified in the embedding forward pass where the LoRA configuration is initialized using an incorrect reference.

[tastelikefeet]

Hi, there are some gemini comments to be resolved

[xichengpro]

Hi, there are some gemini comments to be resolved

Thanks for the feedback. I've refactored the rsLoRA scaling logic to use a ternary operator.

[tastelikefeet]

Can you please run pre-commit run --all-files to lint the code please? After this, this PR can be merged. Run pip install pre-commit to install pre-commit.

[xichengpro]

Can you please run pre-commit run --all-files to lint the code please? After this, this PR can be merged. Run pip install pre-commit to install pre-commit.

Thanks for the feedback. All pre-commit checks now pass locally.