Skip to content

[v2] Move Content-Length validation from urllib3 to botocore - #10622

Draft
aemous wants to merge 3 commits into
v2from
urllib3-v2-content-length-regression
Draft

[v2] Move Content-Length validation from urllib3 to botocore#10622
aemous wants to merge 3 commits into
v2from
urllib3-v2-content-length-regression

Conversation

@aemous

@aemous aemous commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Notes:

  • In [v2] Update urllib3 to 2.6.3 #9971, we upgraded our urllib3 bundled dependency to the next major version. This introduced a regression where urllib3 performs Content-Length validation instead of botocore, causing a different error to be raised instead of botocore's IncompleteRead. As of urllib3 v2, Content-Length validation is enforced by default, and there is a config option to opt-out (Enable enforce_content_length by default urllib3/urllib3#2514). botocore's Content-Length validation was implemented in 2013 when urllib3 had no support for Content-Length validation, so botocore built its own Content-Length validation. This PR opts out of urllib3's Content-Length validation so this check is handled by botocore (as it was pre-urllib3-v2). This fixes a bug in S3 streaming to stdout when an IncompleteRead is encountered.

Description of changes:

  • Pass enforce_content_length=False to urllib3, to opt-out of content-length validation. This brings Content-Length validation back to urllib3. This fixes a bug with encountering an IncompleteRead while streaming an S3 object to stdout.

Description of tests:

  • Added a blackbox test locally to test the S3 streaming download scenario. Verified that the expected retry request is made by the AWS CLI. Without this change, the retry is not made, and the streaming download terminates and surfaces an incomplete read error. Note: this blackbox test is not included in this PR, but it will be added in a future PR as part of the same stack as [v2] Add blackbox tests #10591.
  • Successfully ran all test suites and CI (see GitHub Actions).

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@aemous
aemous requested a review from a team as a code owner September 8, 2026 14:01
@aemous aemous added s3 v2 customization Issues related to CLI customizations (located in /awscli/customizations) labels Sep 8, 2026
@aemous aemous changed the title [v2] Do not let urllib3 enforce content-length validation [v2] Move Content-Length validation from urllib3 to vendored botocore Sep 8, 2026
@aemous aemous changed the title [v2] Move Content-Length validation from urllib3 to vendored botocore [v2] Move Content-Length validation from urllib3 to botocore Sep 8, 2026
@aemous
aemous marked this pull request as draft September 9, 2026 14:17
@aemous

aemous commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

Keeping this in draft mode until we finalize relevant internal discussions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

customization Issues related to CLI customizations (located in /awscli/customizations) s3 v2

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant