Add Russian temporal period rules#2767
Conversation
|
Thanks @dimonnld — this is excellent work, and the analysis (dateparser only resolving the nominative, the One request before we merge: could you split this into two PRs? They are really two independent changes with different review surfaces:
Keeping them separate lets us fast-track the bug fix without coupling it to the new-language review, and keeps each PR’s diff and test intent focused. The precision half in particular shouldn’t have to wait on the Russian review. No changes needed to the actual code — same commits, just two PRs. Happy to review both as soon as they are up. |
b2b31a3 to
da2725b
Compare
|
@benfrank241 done — split as requested. The day-precision fix is now its own PR: #2791 (the language-agnostic This PR now stacks on #2791: the first commit is the precision fix, the second ( Full |
extract_period() runs before dateparser and matches "<month> <year>", so "meeting on 13 July 2024" was widened to 2024-07-01..2024-07-31 and the day was lost. Skip the month-table match when a day number precedes the month, letting dateparser resolve the exact date instead. Language-agnostic: affects every language in the period table (English shown in the test). Split out of #2767 per review so the correctness fix can land independently of the Russian-coverage change.
Extend the non-Chinese period table with Russian relative expressions
(вчера/позавчера/сегодня, «пару|несколько дней|недель|месяцев назад»,
прошлой неделе|месяце|году, прошлых выходных) and Russian month names in
their inflected forms, so Russian time queries get the same deterministic
extraction as English.
Russian months inflect: dateparser only resolves the nominative ("май"),
while "в мае" (prepositional) and "мая" (genitive, in explicit dates) are
the forms that occur. Enumerated per month with word-boundary guards so
stems inside longer words (майонез, мартовские) must not match.
da2725b to
46058c3
Compare
|
@benfrank241 rebased onto main now that #2791 has landed. This PR is down to the single |
Problem
Russian temporal queries mostly don't reach the period table and fall through to
dateparser, which does not resolve Russian declension. Verified by calling the library directly:Russian months inflect:
май(nominative) →в мае(prepositional) →13 мая(genitive). dateparser only resolves the nominative — but the prepositional is the only correct form afterв, so the way the language is actually spoken is exactly what fails. Passinglanguages=["ru"]explicitly does not help; this is dictionary coverage, not language detection.This mirrors what
chinese_temporal_periods.pyalready documents:Change
Russian alternatives added to the existing rules in
temporal_periods.py— no new module. Chinese needs one because ideographs lack whitespace boundaries; Russian is whitespace-delimited like German, so the existing rules extend naturally. Declension is handled by enumeration (ма[йяе],июн[ьяе],январ[ьяе]).Covered:
вчера,позавчера,сегодня,(на) прошлой неделе,(в) прошлом месяце,(в) прошлом году,(на) прошлых выходных,пару|несколько дней|недель|месяцев назад, and all twelve months in three cases.Also fixes an existing precision bug — affects every language
The month+year rule runs before dateparser, so
13 июля 2026— and equally13 July 2026— collapsed to the whole month instead of that day. The month rule now skips when a day number precedes the month, letting exact dates fall through to dateparser, which resolves them correctly.Tests
22 new cases across three parametrized tests, following the
test_query_analyzer_chinese_periodspattern: periods, no-false-positives (вчерашние новости,майонез,мартовские тезисыmust not match — Python's\bis Unicode-aware), and day+month+year precision including the English case.Full suite: 404 passed.
ruff checkandruff format --checkclean.Also verified live against a 13k-unit bank:
что обсуждали 13 июля 2026что обсуждали в мае 2026что было в прошлом месяцечто обсуждали на прошлой неделеwhat did we discuss last weekwhat about may 2026Two adjacent issues, not addressed here
"we"parses as a date.search_dates("what did we discuss in May")returns[('we', ...), ('in May', ...)], andquery_analyzer.pytakesvalid_results[0]— so the false positive wins over the real date. The blacklist ({"do","may","march","will","can","sat",...}) doesn't includewe. Happy to file separately if useful.в мае/in Maywithout a year matches in no language — the month rule requires(\d{4}).