Skip to content

Add Russian temporal period rules#2767

Open
dimonnld wants to merge 1 commit into
vectorize-io:mainfrom
dimonnld:russian-temporal-periods
Open

Add Russian temporal period rules#2767
dimonnld wants to merge 1 commit into
vectorize-io:mainfrom
dimonnld:russian-temporal-periods

Conversation

@dimonnld

Copy link
Copy Markdown
Contributor

Problem

Russian temporal queries mostly don't reach the period table and fall through to dateparser, which does not resolve Russian declension. Verified by calling the library directly:

search_dates("что обсуждали в мае", settings={"RELATIVE_BASE": ref})  # -> None
search_dates("что обсуждали в май", settings={"RELATIVE_BASE": ref})  # -> [('в май', ...)]
search_dates("на прошлой неделе",   settings={"RELATIVE_BASE": ref})  # -> None
search_dates("в июне 2026",         settings={"RELATIVE_BASE": ref})  # -> [('2026', ...)]  # month lost

Russian months inflect: май (nominative) → в мае (prepositional) → 13 мая (genitive). dateparser only resolves the nominative — but the prepositional is the only correct form after в, so the way the language is actually spoken is exactly what fails. Passing languages=["ru"] explicitly does not help; this is dictionary coverage, not language detection.

This mirrors what chinese_temporal_periods.py already documents:

dateparser's Chinese coverage is uneven for period expressions: it often returns None, a single-day...

Change

Russian alternatives added to the existing rules in temporal_periods.py — no new module. Chinese needs one because ideographs lack whitespace boundaries; Russian is whitespace-delimited like German, so the existing rules extend naturally. Declension is handled by enumeration (ма[йяе], июн[ьяе], январ[ьяе]).

Covered: вчера, позавчера, сегодня, (на) прошлой неделе, (в) прошлом месяце, (в) прошлом году, (на) прошлых выходных, пару|несколько дней|недель|месяцев назад, and all twelve months in three cases.

Also fixes an existing precision bug — affects every language

The month+year rule runs before dateparser, so 13 июля 2026 — and equally 13 July 2026 — collapsed to the whole month instead of that day. The month rule now skips when a day number precedes the month, letting exact dates fall through to dateparser, which resolves them correctly.

# Skip when a day number precedes the month ("13 июля 2026", "13 July 2026"):
# that is an exact date, and collapsing it to the whole month loses precision.
if re.search(rf"\b\d{{1,2}}\s+({pattern})\b", query, re.IGNORECASE):
    continue

Tests

22 new cases across three parametrized tests, following the test_query_analyzer_chinese_periods pattern: periods, no-false-positives (вчерашние новости, майонез, мартовские тезисы must not match — Python's \b is Unicode-aware), and day+month+year precision including the English case.

Full suite: 404 passed. ruff check and ruff format --check clean.

Also verified live against a 13k-unit bank:

query window
что обсуждали 13 июля 2026 2026-07-13 .. 07-13
что обсуждали в мае 2026 2026-05-01 .. 05-31
что было в прошлом месяце 2026-06-01 .. 06-30
что обсуждали на прошлой неделе 2026-07-06 .. 07-12
what did we discuss last week 2026-07-06 .. 07-12 (unchanged)
what about may 2026 2026-05-01 .. 05-31 (unchanged)

Two adjacent issues, not addressed here

  1. "we" parses as a date. search_dates("what did we discuss in May") returns [('we', ...), ('in May', ...)], and query_analyzer.py takes valid_results[0] — so the false positive wins over the real date. The blacklist ({"do","may","march","will","can","sat",...}) doesn't include we. Happy to file separately if useful.
  2. в мае / in May without a year matches in no language — the month rule requires (\d{4}).

@benfrank241

Copy link
Copy Markdown
Member

Thanks @dimonnld — this is excellent work, and the analysis (dateparser only resolving the nominative, the \b Unicode-aware false-positive tests, the live 13k-unit verification) is exactly the rigor we like to see. I verified locally: test_query_analyzer.py is 404/404, and I confirmed the day-precision bug fails-before on main for the English case too (13 July 2026 collapses to 2026-07-01 .. 07-31 on main, and correctly falls through to dateparser as the exact day with your fix).

One request before we merge: could you split this into two PRs? They are really two independent changes with different review surfaces:

  1. The day-precision fix (the continue guard so 13 <month> <year> falls through to dateparser). This is a language-agnostic correctness bug that affects English today, so it can land quickly on its own once it is isolated — just the guard + the day+month+year precision tests.
  2. The Russian temporal-period coverage. This is a language-coverage enhancement; it extends recall behavior for a new language and is worth reviewing on its own merits (declension enumeration, the no-false-positive cases, etc.).

Keeping them separate lets us fast-track the bug fix without coupling it to the new-language review, and keeps each PR’s diff and test intent focused. The precision half in particular shouldn’t have to wait on the Russian review.

No changes needed to the actual code — same commits, just two PRs. Happy to review both as soon as they are up.

@dimonnld

Copy link
Copy Markdown
Contributor Author

@benfrank241 done — split as requested. The day-precision fix is now its own PR: #2791 (the language-agnostic continue guard + an English precision test, verified on both dateparser 1.2.2 and 1.4.1).

This PR now stacks on #2791: the first commit is the precision fix, the second (Add Russian temporal period rules) is the language-coverage change to review here — declension enumeration, month+year, relative periods, and the no-false-positive cases. Once #2791 lands I'll rebase and this diff will reduce to just the Russian change.

Full test_query_analyzer.py is green on both dateparser versions (404 passed).

nicoloboschi pushed a commit that referenced this pull request Jul 20, 2026
extract_period() runs before dateparser and matches "<month> <year>", so
"meeting on 13 July 2024" was widened to 2024-07-01..2024-07-31 and the day
was lost. Skip the month-table match when a day number precedes the month,
letting dateparser resolve the exact date instead.

Language-agnostic: affects every language in the period table (English shown
in the test). Split out of #2767 per review so the correctness fix can land
independently of the Russian-coverage change.
Extend the non-Chinese period table with Russian relative expressions
(вчера/позавчера/сегодня, «пару|несколько дней|недель|месяцев назад»,
прошлой неделе|месяце|году, прошлых выходных) and Russian month names in
their inflected forms, so Russian time queries get the same deterministic
extraction as English.

Russian months inflect: dateparser only resolves the nominative ("май"),
while "в мае" (prepositional) and "мая" (genitive, in explicit dates) are
the forms that occur. Enumerated per month with word-boundary guards so
stems inside longer words (майонез, мартовские) must not match.
@dimonnld
dimonnld force-pushed the russian-temporal-periods branch from da2725b to 46058c3 Compare July 20, 2026 17:08
@dimonnld

Copy link
Copy Markdown
Contributor Author

@benfrank241 rebased onto main now that #2791 has landed. This PR is down to the single Add Russian temporal period rules commit (temporal_periods.py + test_query_analyzer.py) — the day-precision fix is gone from the diff since it merged separately. Full test_query_analyzer.py stays green (404 passed) on both dateparser 1.2.2 and 1.4.1. Ready for review whenever you have a chance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants