375 job management

docs/guides/generating-mosaics.md

@@ -0,0 +1,28 @@
+# Generating mosaics
+
+Once layers are grouped into layersets, and each layerset is trimmed in the [multimask](./trimming.md) interface, the final step is to collapse the content into a single mosaic per layerset, like finally gluing down all the individual clippings in a collage. You'll find these within within a map's **Mosaic** → **Derivatives** (see [New Iberia, 1885](https://oldinsurancemaps.net/map/sanborn03375_001#mosaic)).
+
+Currently, OIM supports the creation of two different derivatives per layerset:
+
+- **Cloud Optimized GeoTIFFs (COG)**
+    - This is the primary output, and when present, is used within OIM in certain contexts
+- **Static XYZ Tileset**
+    - This is a rendering of the latest COG mosaic to actual PNG tiles in a {z}/{y}/{x}.png folder structure.
+
+!!! note
+
+    You may be wondering what happens to Key Maps and other categories of maps, i.e. other layersets. Though these are handled on the backend exactly like the Main Content layerset, they are only shown in the derivatives list if they have more than one layer.
+
+## Keeping derivatives up-to-date
+
+The creation of a single geotiff (or static XYZ tileset) is effectively a snapshot of the work that has been performed on a map's layers and masks up to that point. Timestamps are displayed showing when masks were last updated, and when derivatives were last generated.
+
+![up-to-date cog derivative with timestamps](../images/mosaic-cog-fresh.png)
+
+However, this means that if the ground control points are subsequently updated for a layer, or the masks are adjusted, the derivative artifact will be out-of-date. In this case, the interface will show timestamps of out-of-date artifacts in <span style="color:red">red</span> and a button will be available allowing you to queue this artifact to be rebuilt.
+
+![out-of-date cog derivative with red timestamp](../images/mosaic-cog-stale.png)
+
+Because the process to create a single geotiff (or xyz tileset) from dozens or hundreds of layers can take several hours, we use a queueing system to ensure the server is never overloaded. You can see all mosaicking processes (queued, running, or completed) in [Jobs](https://oldinsurancemaps.net/jobs).
+
+In the current setup, only two jobs are allowed to run at any time, and queued mosaic operations are processed with a first in, first out policy.

docs/guides/services.md

@@ -0,0 +1,48 @@
+
+# Downloads & web services
+
+Historical map content can be accessed in a number of different ways and used in many different applications. Throughout the platform you will find places to download or link to:
+
+1. Individual layers that have been georeferenced
+    - Find these within a map's **Georeferenced** section: https://oldinsurancemaps.net/map/sanborn03375_001#georeferenced
+2. Unified mosaics made from multiple layers
+    - Find these within a map's **Mosaic** → **Derivatives** section: https://oldinsurancemaps.net/map/sanborn03375_001#mosaic
+
+!!! warning
+
+    When downloading or otherwise accessing maps, please keep in mind that **OldInsuranceMaps.net provides no guarantee as to the accuracy of georeferenced content.**
+
+!!! important
+
+    Over time, someone may come along and improve the georeferencing on a layer, or update multimask boundaries and regenerated a mosaic. When this happens the original file is deleted and a new one is created. Because many of the endpoints described below contain the actual filename, **tile service links in 3rd-party integrations may need to be updated after the source layer or mosaic has been updated**.
+
+## Downloadable geotiffs (COGs)
+
+OIM relies heavily on [cloud optimized geotiffs](https://cogeo.org/), or "COGs", and you can download these files at any time and easily use them directly in software like QGIS or ArcGIS Desktop. All COGs use JPEG compression in order to keep file sizes manageable.
+
+## Tile services
+
+OIM provides service endpoints that can be used in many different applications. The particular service endpoint you use will largely depend on where you are trying to use it, as not all applications support all types of endpoints.
+
+Generally speaking, XYZ tiles are the most widely supported, TileJSON is a superset spec that wraps an XYZ tile endpoint with some extra information, and WMS is an older standard still in use in many applications.
+
+OIM uses TiTiler to dynamically produce tiled services from the same source COG files that you can download as described above.
+
+### XYZ tiles
+
+XYZ tiles are a very widely supported standard supported by most geospatial software and mapping applications. Here is more info for how to use these URLs in desktop software like [QGIS](https://docs.qgis.org/3.22/en/docs/user_manual/managing_data_source/opening_data.html#using-xyz-tile-services) and [ArcGIS](https://esribelux.com/2021/04/16/xyz-tile-layers-in-arcgis-platform/), as well as web mapping applications like [Leaflet](https://leafletjs.com/reference.html#tilelayer), [OpenLayers](https://openlayers.org/en/latest/examples/xyz.html), [Mapbox/Maplibre GL JS](https://maplibre.org/maplibre-gl-js-docs/example/map-tiles/), [ArcGIS Online](https://doc.arcgis.com/en/arcgis-online/create-maps/add-layers-from-url.htm#ESRI_SECTION1_13A68145A4F94A338AB2002AD5F9844D), and [OpenHistoricalMap (iD Editor)](https://learnosm.org/en/beginner/id-editor/#configuring-the-background-layer).
+
+!!! note
+
+    Some mosaics will also have an **Static XYZ Tileset** available. This is a static, pre-rendered tileset that you can download, unpack, and host on your own infrastructure. It is different from the XYZ tiles endpoints in that it does not rely on any dynamic tiling service, like TiTiler.
+
+### TileJSON
+
+[TileJSON](https://github.com/mapbox/tilejson-spec) is a useful spec that wraps an XYZ endpoint with a few other pieces of information.
+
+Currently, this is the most reliable OIM-provided endpoint to use because while the underlying XYZ url which may change (as described above), the TileJSON url itself will not change. However, TileJSON is not as widely supported as XYZ tiles.
+
+
+## IIIF Georeference Extension
+
+For each layer and mosaic you can also access a collection of the georeferencing information formatted as the [IIIF Georeference Extension](https://iiif.io/api/extension/georef/). This annotation will includes the IIIF URLs from the source resources in the Library of Congress and when you open it in Allmaps Viewer, the tiles will be coming directly from LOC and being warped in the browser according to control points and masks created in OIM. Learn more in this slideshow: [creating IIIF georeference annotations](https://docs.google.com/presentation/d/18FRpq1Lcz3T7WincZNRvTjdL87ULy1bU68BCxr425dk/edit?usp=sharing).

docs/guides/trimming.md

@@ -2,4 +2,8 @@
 
 Overlapping edges of layers cause messy mosaics, so a mask must be added to each layer to remove its margins. These masks should all be contiguous (sharing adjacent borders and vertices) so it is best to handle them at the map-level, not by masking each layer individually. This is the idea behind a **MultiMask**, which is a unified collection of masks for all layers within a map.
 
-![The multitrim interface.](../images/multitrim.gif)
+![The multitrim interface.](../images/multitrim.gif)
+
+## Masking across multiple volumes
+
+In large cities it is common to have multiple volumes for a single atlas edition, each volume only covering a subset of the entire city. In the OIM data model, these volumes are treated as separate entities, so while the pages within each one can be carefully masked to form a mosaic, there is no affordance for combining those mosaics across all volumes within an edition. This is something we hope to support in the future.

ohmg/api/filters.py

@@ -17,6 +17,11 @@ class FilterSessionSchema(FilterSchema):
     type: Optional[str]
 
 
+class FilterJobSchema(FilterSchema):
+    operation: Optional[str] = Field(q="operation")
+    stage: Optional[str] = Field(q="stage")
+
+
 class FilterDocumentSchema(FilterSchema):
     map: str = Field(q="map_id")
     prepared: Optional[bool] = Field(q="prepared")

ohmg/api/routes.py

@@ -21,6 +21,7 @@
     Region,
 )
 from ohmg.georeference.models import (
+    Job,
     SessionBase,
     SessionLock,
 )
@@ -29,6 +30,7 @@
 from .filters import (
     FilterAllDocumentsSchema,
     FilterDocumentSchema,
+    FilterJobSchema,
     FilterRegionSchema,
     FilterSessionSchema,
 )
@@ -39,6 +41,7 @@
 )
 from .schemas import (
     DocumentSchema,
+    JobSchema,
     LayerSchema,
     LayerSetSchema,
     MapFullSchema,
@@ -76,6 +79,14 @@ def authenticate(self, request, key):
 )
 
 
+@beta2.get("jobs/", response=List[JobSchema], url_name="job_list")
+@paginate
+def list_jobs(request, filters: FilterJobSchema = Query(...)):
+    queryset = Job.objects.all().order_by("-date_queued")
+    queryset = filters.filter(queryset)
+    return queryset
+
+
 @beta2.get("sessions/", response=List[SessionSchema], url_name="session_list")
 @paginate(SessionPagination)
 def list_sessions(request, filters: FilterSessionSchema = Query(...), date_range: str = ""):

ohmg/api/schemas.py

@@ -29,6 +29,13 @@
 logger = logging.getLogger(__name__)
 
 
+def _datefield_to_timestamp(obj, field: str) -> float | None:
+    try:
+        return getattr(obj, field).timestamp()
+    except (AttributeError, ValueError):
+        return None
+
+
 class UserSchema(Schema):
     username: str
     profile_url: str
@@ -476,6 +483,7 @@ def resolve_urls(obj):
 
 
 class LayerSetSchema(Schema):
+    pk: int
     id: str
     name: str
     map_id: str
@@ -487,6 +495,9 @@ class LayerSetSchema(Schema):
     multimask_extent: Optional[tuple]
     mosaic_cog_url: Optional[str]
     xyz_tiles_url: Optional[str]
+    multimask_date: Optional[float]
+    latest_cog_job: Optional["JobSchema"]
+    latest_xyz_job: Optional["JobSchema"]
 
     @staticmethod
     def resolve_id(obj):
@@ -504,6 +515,18 @@ def resolve_layers_masked_ct(obj):
     def resolve_name(obj):
         return str(obj.category)
 
+    @staticmethod
+    def resolve_multimask_date(obj):
+        return _datefield_to_timestamp(obj, "multimask_date")
+
+    @staticmethod
+    def resolve_latest_cog_job(obj):
+        return obj.get_latest_cog_job()
+
+    @staticmethod
+    def resolve_latest_xyz_job(obj):
+        return obj.get_latest_xyz_job()
+
 
 class LayerSetDisplaySchema(Schema):
     map_id: str
@@ -861,6 +884,49 @@ def resolve_status(obj):
                 return "split"
 
 
+class JobTargetSchema(Schema):
+    name: str
+    url: str
+
+    @staticmethod
+    def resolve_name(obj):
+        return str(obj)
+
+    @staticmethod
+    def resolve_url(obj):
+        return f"/map/{obj.map.pk}"
+
+
+class JobSchema(Schema):
+    id: int
+    operation: str
+    stage: str
+    message: Optional[str]
+    target: JobTargetSchema
+    date_created: Optional[float]
+    date_queued: Optional[float]
+    date_started: Optional[float]
+    date_ended: Optional[float]
+    run_duration: Optional[int]
+
+    @staticmethod
+    def resolve_date_created(obj):
+        return _datefield_to_timestamp(obj, "date_created")
+
+    @staticmethod
+    def resolve_date_queued(obj):
+        return _datefield_to_timestamp(obj, "date_queued")
+
+    @staticmethod
+    def resolve_date_started(obj):
+        return _datefield_to_timestamp(obj, "date_started")
+
+    @staticmethod
+    def resolve_date_ended(obj):
+        return _datefield_to_timestamp(obj, "date_ended")
+
+
 DocumentFullSchema.update_forward_refs()
 RegionFullSchema.update_forward_refs()
 MapListSchema2.update_forward_refs()
+LayerSetSchema.update_forward_refs()

ohmg/conf/http.py

@@ -38,7 +38,7 @@ def user_info_from_request(request):
         user_info["api_keys"] = user.api_keys
         user_info["is_authenticated"] = True
         user_info["is_staff"] = user.is_staff
-        user_info["perms"] = list(user.get_all_permissions())
+        user_info["perms"] = sorted(list(user.get_all_permissions()))
     else:
         user_info = {
             "is_authenticated": False,
@@ -90,7 +90,7 @@ def __init__(self, message="object not found"):
 
 class JsonResponseUnauthorized(JsonResponse):
     def __init__(self, message="unauthorized"):
-        self.status_code = HTTPStatus.NOT_FOUND
+        self.status_code = HTTPStatus.UNAUTHORIZED
         super().__init__(
             {
                 "success": False,
@@ -101,7 +101,7 @@ def __init__(self, message="unauthorized"):
 
 class JsonResponseBadRequest(JsonResponse):
     def __init__(self, message="bad request"):
-        self.status_code = HTTPStatus.NOT_FOUND
+        self.status_code = HTTPStatus.BAD_REQUEST
         super().__init__(
             {
                 "success": False,

ohmg/conf/management/commands/configure-services.py

@@ -220,7 +220,7 @@ def generate_celery_mosaic_service(self, state_path: Path):
     --statedb={str(state_path.resolve())}/celery_mosaic_worker.state \\
     --loglevel=INFO \\
     --logfile={log_dir}/celery_mosaic.log \\
-    --concurrency=1 -n worker2@%h
+    --concurrency={settings.MAX_CONCURRENT_MOSAIC_JOBS} -n worker2@%h
 Restart=always
 
 [Install]

ohmg/conf/settings.py

@@ -303,6 +303,8 @@
 # this will be removed once Django is upgraded
 SWAP_COORDINATE_ORDER = ast.literal_eval(os.getenv("SWAP_COORDINATE_ORDER", "False"))
 
+MAX_CONCURRENT_MOSAIC_JOBS = os.getenv("MAX_CONCURRENT_MOSAIC_JOBS", 1)
+
 # CONFIGURE CELERY
 CELERY_BROKER_URL = os.getenv("BROKER_URL")
 CELERY_RESULT_BACKEND = "rpc://"
@@ -327,6 +329,7 @@
     "ohmg.georeference.tasks.delete_stale_sessions": {"queue": "background"},
     "ohmg.georeference.tasks.delete_preview_vrts": {"queue": "background"},
     "ohmg.georeference.tasks.cleanup_existing_tileset": {"queue": "background"},
+    "ohmg.georeference.tasks.run_queued_mosaic_jobs": {"queue": "background"},
     "ohmg.georeference.tasks.create_mosaic_cog": {"queue": "mosaic"},
     "ohmg.georeference.tasks.create_mosaic_tileset": {"queue": "mosaic"},
 }
@@ -336,7 +339,11 @@
     "remove_stale_sessions": {
         "task": "ohmg.georeference.tasks.delete_stale_sessions",
         "schedule": 60.0,
-    }
+    },
+    "run_queued_mosaic_jobs": {
+        "task": "ohmg.georeference.tasks.run_queued_mosaic_jobs",
+        "schedule": 15.0,
+    },
 }
 
 # note: this is app_label.ModelClass,

ohmg/core/admin.py

@@ -61,12 +61,18 @@ class LayerAdmin(admin.ModelAdmin):
 
 
 class LayerSetAdmin(admin.ModelAdmin):
+    list_display = (
+        "map",
+        "category",
+        "multimask_date",
+    )
     raw_id_fields = ("map",)
     readonly_fields = (
         "layer_display_list",
         "extent",
         "multimask_extent",
         "xyz_tiles_url",
+        "multimask_date",
     )
     search_fields = ("map__title",)
     list_filter = ("category",)

ohmg/core/migrations/0014_alter_permissions_options_and_more.py

@@ -0,0 +1,32 @@
+# Generated by Django 4.2.27 on 2026-06-23 09:40
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0013_permissions'),
+    ]
+
+    operations = [
+        migrations.AlterModelOptions(
+            name='permissions',
+            options={'default_permissions': (), 'managed': False, 'permissions': (('use_helmert', 'Can use Helmert transformation'), ('queue_mosaic_cog', 'Can queue creation of mosaic COG'), ('queue_mosaic_xyz', 'Can queue creation of mosaic XYZ tileset'))},
+        ),
+        migrations.AddField(
+            model_name='layerset',
+            name='mosaic_geotiff_date',
+            field=models.DateTimeField(blank=True, null=True),
+        ),
+        migrations.AddField(
+            model_name='layerset',
+            name='multimask_date',
+            field=models.DateTimeField(blank=True, null=True),
+        ),
+        migrations.AddField(
+            model_name='layerset',
+            name='xyz_tiles_date',
+            field=models.DateTimeField(blank=True, null=True),
+        ),
+    ]

ohmg/core/migrations/0015_remove_layerset_mosaic_geotiff_date_and_more.py

@@ -0,0 +1,21 @@
+# Generated by Django 4.2.27 on 2026-06-23 18:07
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0014_alter_permissions_options_and_more'),
+    ]
+
+    operations = [
+        migrations.RemoveField(
+            model_name='layerset',
+            name='mosaic_geotiff_date',
+        ),
+        migrations.RemoveField(
+            model_name='layerset',
+            name='xyz_tiles_date',
+        ),
+    ]