Skip to content

added logic to pass the firewall#1983

Open
niveditasing wants to merge 2 commits intodatacommonsorg:masterfrom
niveditasing:code_fix_to_pass_firewall
Open

added logic to pass the firewall#1983
niveditasing wants to merge 2 commits intodatacommonsorg:masterfrom
niveditasing:code_fix_to_pass_firewall

Conversation

@niveditasing
Copy link
Copy Markdown
Contributor

The HUD website's firewall (Amazon WAF) started blocking the default Python requests library. Instead of returning the file,
it returned a security challenge (HTTP status 202), causing the script to crash.
The fix bypasses the firewall so all files can download successfully. I haven't added any code to skip already-downloaded files yet—the script just successfully downloads all of them now without crashing.

Differ has no deletions as all files are now getting download
PR checklist: https://docs.google.com/spreadsheets/d/1BzweR9Sj58j0H2_BweGTmfE4Z1lrjPZL8u1FS0kzCeg/edit?pli=1&gid=0#gid=0

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the download_with_retry function in scripts/us_hud/income/process.py to include a custom User-Agent header and implement error handling with response.raise_for_status(). A review comment correctly identifies a security risk due to disabled SSL verification and suggests adding a timeout to the request to prevent potential hangs.

Comment thread scripts/us_hud/income/process.py Outdated
@niveditasing niveditasing requested a review from saanikaaa April 29, 2026 05:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant