Foonote-references flavour #1325

Draft

lfoppiano wants to merge 56 commits into

feature/literature-law-flavour

lfoppiano commented Aug 6, 2025 •

edited

Loading

Member

This flavor aims to develop a special flavour for Law and certain Humanities articles using footnotes for reference/footnotes/mixed content.

Guidelines are here: https://github.com/cboulanger/grobid-footnote-flavour/blob/main/docs/guidelines.md

coveralls commented Aug 6, 2025 •

edited

Loading

coverage: 40.466% (+0.2%) from 40.279%
when pulling 7d7e0b6 on feature/literature-law-flavour
into 1ad5901 on master.

lfoppiano changed the title ~~DH and LAW Literature flavour~~ Foonote-references flavour

lfoppiano force-pushed the feature/literature-law-flavour branch from 7d7e0b6 to a13f0d5 Compare

January 30, 2026 16:14

coveralls commented Jan 30, 2026 •

edited

Loading

Pull Request Test Coverage Report for Build 21991403140

Details

0 of 0 changed or added relevant lines in 0 files are covered.
109 unchanged lines in 4 files lost coverage.
Overall coverage decreased (-0.01%) to 38.258%

Files with Coverage Reduction	New Missed Lines	%
org/grobid/core/utilities/Utilities.java	1	26.63%
org/grobid/core/GrobidModels.java	8	85.82%
org/grobid/core/engines/ReferenceSegmenterParser.java	38	49.77%
org/grobid/trainer/sax/TEISegmentationSaxParser.java	62	3.74%

Totals
Change from base Build 21969397686:	-0.01%
Covered Lines:	17266
Relevant Lines:	42518

💛 - Coveralls

lfoppiano force-pushed the feature/literature-law-flavour branch from 65cfb01 to 7272991 Compare

February 13, 2026 13:19

This was referenced Mar 4, 2026

not doing well on recognizing references in footnotes #929

Open

reference segment for reference with comments #888

Open

lfoppiano force-pushed the feature/literature-law-flavour branch from 825e438 to 03a79f3 Compare

April 2, 2026 09:51

lfoppiano and others added 22 commits

May 9, 2026 16:57


          add new flavor, first training data, model 0 and update the interface

a75d842


          annotations corrections and new data

c031e7f


          update model

6ba6263


          rename old data, and add new corrected training data

61cbbc6


          update segmentation model

f2e240c


          add more training data

0435e55


          update model

caa33ba


          feat:add more training data

8c1b5c6


          update model

1427cd4


          fix: handling special incorrect sequences #1355


          fix: TrainerRunner must know about the footnote flavor (#1385)

d37767f


          feat: add the updated 105 segmentations training documents

5d3e328


          fix: training data corrections and cleanup

c5c7991


          feat: updated segmentation model

25a92dd


          Training experiments for the literature footnote flavour (#1394)

8e99a63

 changes:
  - rename segmentation template identifiers
  - verify training data
  - merge low-resources labels
  - use the correct the batch job in wapiti
  - add regularization in wapiti

---------

Signed-off-by: Luca Foppiano <luca@foppiano.org>


          fix: add segmentation-dh-law model in the training list

f282586


          fix: adjust training data to the new pdfalto

8b0bf9f


          fix: disable label remapping for training

4770a3c


          feat: update model

6cbc533

Signed-off-by: Luca Foppiano <luca@foppiano.org>


          fix: lmdb failure due to recent breaking changes

a1d3384


          feat: add a token-based segmentation dh flavor

9344e16


          feat: update pdfalto with better line recognition

9d06f10

lfoppiano added 10 commits

May 9, 2026 17:13


          chore: add training data for the reference segmenter

c3c0dcb


          fix: training data organisation

8ea6bd7


          feat: support flavor in the reference segmenter

edefbaf


          models: add new reference segmenter wapiti model

9701c50


          feat: enhance reference segmenter with flavor support and label propa…

e464348

…gation


          feat: update template to give more context

d2773eb

Signed-off-by: Luca Foppiano <luca@foppiano.org>


          fix: make the parser ignoring the teiheader

4c65d82

Signed-off-by: Luca Foppiano <luca@foppiano.org>


          feat: add more training data for the law/literature flavor for the re…

7a60af1

…ference segmenter


          feat: update model

705d88a

Signed-off-by: Luca Foppiano <luca@foppiano.org>


          chore: spotless

fa98d42

lfoppiano force-pushed the feature/literature-law-flavour branch from 0eac3b9 to fa98d42 Compare

May 9, 2026 16:21

lfoppiano added 15 commits

May 11, 2026 22:09


          fix: re-generate raw feature training data

337ab25


          fix: reference-segmenter parser and trainer were dropping lot of refe…

02a4b3f

…rences on the floor


          models: update the reference-segmenter wapiti model

72952e8


          chore: spotless + glitch in build.gradle

afc1b63

Signed-off-by: Luca Foppiano <luca@foppiano.org>


          models: add DL models for the reference-segmenter

9209e3b


          chore: update the trainer registry

da7e7af


          chore: spotless

dd0608b


          fix: correctly switch to the right flavor model when using DeLFT

7aa4374


          feat: update configuration after fix in the model loading

c782a88


          feat: add optional grouping label for footnotes in TEI output

d393236


          feat: add new annotations for segmentation

0705b43


          feat: add new annotations for batch 8 of reference segmenter

d9fe426


          fix: wrongly generated feature file training data

57c5858


          fix: use all segmentation files for training

12487ec


          feat: updated segmentation model for the flavor

77a8a22

github-code-quality Bot found potential problems

View reviewed changes

grobid-trainer/scripts/sync_lb_positions.py

    
              import argparse

              import difflib

              import os

lfoppiano and others added 3 commits

June 26, 2026 12:12


          Merge remote-tracking branch 'origin/master' into feature/literature-…

…law-flavour

# Conflicts:
#	grobid-core/src/main/java/org/grobid/core/GrobidModels.java
#	grobid-core/src/main/java/org/grobid/core/engines/FullTextParser.java
#	grobid-core/src/main/java/org/grobid/core/engines/ReferenceSegmenterParser.java
#	grobid-home/config/grobid.yaml
#	grobid-trainer/src/main/java/org/grobid/trainer/SegmentationTrainer.java


          chore: codespell

b5460cf

Signed-off-by: Luca Foppiano <luca@foppiano.org>


          Merge branch 'master' into feature/literature-law-flavour

d246509

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet