diff --git a/ROADMAP.md b/ROADMAP.md index cb25018a4..3c653a5bb 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -92,8 +92,6 @@ And alongside it: ([#853](https://github.com/opengeos/RustGIS/issues/853)). _Coordinate first_ (it changes `FieldType`). - Editable standalone tables (they are read-only today). -- Export into an existing GeoPackage as an added or replaced table, leaving its other tables - ([#852](https://github.com/opengeos/RustGIS/issues/852); export refuses an existing file today). - Polygon editing: shared edges, where moving one boundary moves its neighbour's too. Coincident polygon vertices can already move together with Shared Vertices (`edit.moveSharedVertex`); different vertex densities and linked insertion/deletion remain @@ -149,6 +147,9 @@ Open questions rather than plans; discuss before building. ## Done +- Export into an existing GeoPackage on desktop: a layer or table added as one more table, or + replacing one of its tables, leaving the others as they were (`mode` on `share.exportLayer` and + `share.exportTable`). - Polygon editing: Cut Polygon, Reshape (polygons and lines) and Trace along an edge while drawing, on the map and through `edit.cutPolygon`, `edit.reshape` and `edit.trace`. - Geometry conversions: Polygon To Line, Split Line At Vertices, Feature Envelope To Polygon diff --git a/apps/rustgis-python/python/rustgis/__init__.py b/apps/rustgis-python/python/rustgis/__init__.py index 6b502925c..62e4f4774 100644 --- a/apps/rustgis-python/python/rustgis/__init__.py +++ b/apps/rustgis-python/python/rustgis/__init__.py @@ -268,7 +268,14 @@ def to_geopandas(self, selected: bool = False) -> Any: collection = self.to_geojson(selected) return gpd.GeoDataFrame.from_features(collection["features"], crs="EPSG:4326") - def to_file(self, path: PathLike, crs: Optional[str] = None) -> Any: + def to_file( + self, + path: PathLike, + crs: Optional[str] = None, + *, + mode: Optional[str] = None, + table_name: Optional[str] = None, + ) -> Any: """Writes the layer's features to a file (``share.exportLayer``). Args: @@ -276,10 +283,19 @@ def to_file(self, path: PathLike, crs: Optional[str] = None) -> Any: ``.gpkg``, ``.parquet``, ``.shp``, ``.zip``, ``.kml``, ``.kmz`` or ``.gpx``. crs: The coordinate system of a GeoPackage, Shapefile or FlatGeobuf, e.g. ``"EPSG:3857"`` (the one the layer was read in when left out). + mode: For a GeoPackage: ``"newFile"`` (the default; an existing file is refused), + ``"addTable"`` (one more table in an existing GeoPackage, or a new file when + there is none) or ``"replaceTable"`` (replacing its table of that name, which + must be there). + table_name: The GeoPackage table (the layer's name when left out). """ params: Dict[str, Any] = {"layer": self.id, "path": os.fspath(path)} if crs is not None: params["crs"] = crs + if mode is not None: + params["mode"] = mode + if table_name is not None: + params["tableName"] = table_name return self.session.run("share.exportLayer", params) def __eq__(self, other: object) -> bool: diff --git a/crates/data/src/export.rs b/crates/data/src/export.rs index 691d96daf..167e50d77 100644 --- a/crates/data/src/export.rs +++ b/crates/data/src/export.rs @@ -6,7 +6,7 @@ use std::path::Path; -use crate::io::{CsvWriter, GeoJsonWriter, PROJECTED_FORMATS, check_projected, project_each, write_atomic_with}; +use crate::io::{CsvWriter, GeoJsonWriter, GpkgTable, GpkgWritten, PROJECTED_FORMATS, check_projected, project_each, write_atomic_with}; use crate::raster::Crs; use crate::source::FeatureSource; use crate::{DataError, Dataset, Feature, GeometryType, Result}; @@ -43,8 +43,32 @@ pub fn write_source_in(src: &dyn FeatureSource, layer: &Dataset, path: &Path, cr write_chunks_in(src, layer, path, crs, CHUNK_BYTES) } +/// As [`write_source_in`], a GeoPackage's table placed as `gpkg` says (see +/// [`crate::io::write_path_to`]): also returns the table written. +pub fn write_source_to( + src: &dyn FeatureSource, + layer: &Dataset, + path: &Path, + crs: Crs, + gpkg: &GpkgTable, +) -> Result<(Exported, Option)> { + write_chunks_to(src, layer, path, crs, gpkg, CHUNK_BYTES) +} + /// [`write_source_in`], `bytes` of features a chunk. fn write_chunks_in(src: &dyn FeatureSource, layer: &Dataset, path: &Path, crs: Crs, bytes: usize) -> Result { + write_chunks_to(src, layer, path, crs, &GpkgTable::default(), bytes).map(|(e, _)| e) +} + +/// [`write_source_to`], `bytes` of features a chunk. +fn write_chunks_to( + src: &dyn FeatureSource, + layer: &Dataset, + path: &Path, + crs: Crs, + gpkg: &GpkgTable, + bytes: usize, +) -> Result<(Exported, Option)> { let ext = path.extension().and_then(|e| e.to_str()).unwrap_or("").to_ascii_lowercase(); if !streams(&ext) { return Err(DataError::Unsupported(format!(".{ext} written a chunk at a time"))); @@ -83,14 +107,18 @@ fn write_chunks_in(src: &dyn FeatureSource, layer: &Dataset, path: &Path, crs: C skipped = batch.skipped; Ok(()) }; + let mut table = None; let left_out = match ext.as_str() { #[cfg(not(target_arch = "wasm32"))] "gpkg" => { - crate::source::gpkg::write_features_in(&layer.name, geometry_type, &layer.fields, path, crs, &mut feed)?; + table = Some(crate::source::gpkg::write_features_in(&layer.name, geometry_type, &layer.fields, path, crs, gpkg, &mut feed)?); 0 } #[cfg(target_arch = "wasm32")] - "gpkg" => return Err(DataError::Unsupported("GeoPackage in the browser".into())), + "gpkg" => { + let _ = (gpkg, &mut table); + return Err(DataError::Unsupported("GeoPackage in the browser".into())); + } "geojson" | "json" => write_atomic_with(path, |out| { let mut w = GeoJsonWriter::new(out, &layer.name, &layer.fields)?; feed(&mut |features| w.write(features))?; @@ -111,7 +139,7 @@ fn write_chunks_in(src: &dyn FeatureSource, layer: &Dataset, path: &Path, crs: C Ok::<_, DataError>(0) })?, }; - Ok(Exported { written: total.saturating_sub(left_out), left_out, skipped }) + Ok((Exported { written: total.saturating_sub(left_out), left_out, skipped }, table)) } /// The kind of a source's first geometry, read up to it; `None` for a source without any (read diff --git a/crates/data/src/export/tests.rs b/crates/data/src/export/tests.rs index 91220bc08..24d067315 100644 --- a/crates/data/src/export/tests.rs +++ b/crates/data/src/export/tests.rs @@ -278,3 +278,51 @@ fn a_panicking_writer_leaves_nothing_behind() { assert!(r.is_err()); assert_eq!(std::fs::read_dir(&d).unwrap().count(), 0); } + +/// Streamed into an existing GeoPackage, an export that starts again (a multipoint, then a +/// repeated id) rolls back what it wrote and writes the table once, as a whole export would; one +/// whose source fails leaves the file's bytes as they were. +#[test] +fn a_streamed_export_into_an_existing_geopackage_starts_again_inside_its_transaction() { + let d = folder("redo-existing"); + let county = d.join("county.gpkg"); + let mut parcels = Dataset::new("parcels", GeometryType::Point, vec![]); + parcels.push_next(Some(Geometry::Point(Coord::new(5.0, 5.0))), vec![]); + crate::io::write_path(&parcels, &county).unwrap(); + let fields = vec![Field::new("n", FieldType::Integer)]; + let mut both = Dataset::new("both", GeometryType::Point, fields.clone()); + for i in 0..20 { + let c = Coord::new(i as f64, 1.0); + let g = if i == 15 { Geometry::MultiPoint(vec![c, Coord::new(i as f64, 2.0)]) } else { Geometry::Point(c) }; + both.push_next(Some(g), vec![Value::Int(i)]); + } + let first = both.features.first().map(|f| f.id).unwrap(); + if let Some(f) = both.features.get_mut(18) { + f.id = first; + } + let layer = Dataset::new("both", GeometryType::Point, fields); + let add = GpkgTable { mode: crate::io::GpkgMode::AddTable, name: None }; + let src = Runs::new(&both, 4); + let (e, table) = write_chunks_to(&src, &layer, &county, Crs::LonLat, &add, 1).unwrap(); + assert_eq!(src.passes.load(std::sync::atomic::Ordering::Relaxed), 3, "written three times"); + assert_eq!((e.written, table), (20, Some(GpkgWritten { table: "both".into(), replaced: false, dependents: vec![] }))); + let whole = d.join("whole.gpkg"); + crate::io::write_path(&both, &whole).unwrap(); + let read = + |p: &Path, t: &str| crate::source::open_path(p, Some(t)).unwrap().read(&Request { extent: None, limit: 100 }).unwrap().features; + assert_eq!(read(&county, "both"), read(&whole, "both")); + assert_eq!(read(&county, "parcels").len(), 1); + let c = rusqlite::Connection::open(&county).unwrap(); + let kind: String = + c.query_row("SELECT geometry_type_name FROM gpkg_geometry_columns WHERE table_name = 'both'", [], |r| r.get(0)).unwrap(); + assert_eq!(kind, "MULTIPOINT"); + drop(c); + // A source failing on the second pass: nothing of the table is left. + let before = std::fs::read(&county).unwrap(); + let mut layer2 = layer.clone(); + layer2.name = "again".into(); + let mut failing = Runs::new(&both, 4); + (failing.fail_after, failing.fail_pass) = (Some(2), Some(1)); + assert!(write_chunks_to(&failing, &layer2, &county, Crs::LonLat, &add, 1).is_err()); + assert_eq!(std::fs::read(&county).unwrap(), before); +} diff --git a/crates/data/src/io.rs b/crates/data/src/io.rs index 6ad79a569..0069de9a2 100644 --- a/crates/data/src/io.rs +++ b/crates/data/src/io.rs @@ -494,9 +494,44 @@ pub fn write_path(ds: &Dataset, path: &Path) -> Result { write_path_in(ds, path, Crs::LonLat) } +/// How a GeoPackage export treats a GeoPackage already at its path. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub enum GpkgMode { + /// Writes a new file; an existing one is refused, as it may hold other tables. + #[default] + NewFile, + /// Adds the table to the GeoPackage (a new file when there is none); refuses a table of its + /// name. + AddTable, + /// Replaces the GeoPackage's feature or attribute table of its name, which must be there. + ReplaceTable, +} + +/// Where a GeoPackage export puts its table: how, and under what name (default: the layer's). +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct GpkgTable { + pub mode: GpkgMode, + pub name: Option, +} + +/// The table a GeoPackage export wrote: its name in the file (a reserved name is prefixed), +/// whether it replaced one, and the views and triggers on other tables that name a replaced one. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GpkgWritten { + pub table: String, + pub replaced: bool, + pub dependents: Vec, +} + /// As [`write_path`], in the coordinate system `crs` (a GeoPackage, Shapefile or FlatGeobuf; see /// [`write_bytes_in`]). pub fn write_path_in(ds: &Dataset, path: &Path, crs: Crs) -> Result { + write_path_to(ds, path, crs, &GpkgTable::default()).map(|(left_out, _)| left_out) +} + +/// As [`write_path_in`], a GeoPackage's table placed as `gpkg` says (other formats ignore it): +/// also returns the table written. +pub fn write_path_to(ds: &Dataset, path: &Path, crs: Crs, gpkg: &GpkgTable) -> Result<(usize, Option)> { let ext = path.extension().and_then(|e| e.to_str()).unwrap_or("").to_ascii_lowercase(); let projected; let ds = if crs == Crs::LonLat || !PROJECTED_FORMATS.contains(&ext.as_str()) { @@ -511,11 +546,14 @@ pub fn write_path_in(ds: &Dataset, path: &Path, crs: Crs) -> Result { #[cfg(not(target_arch = "wasm32"))] { crate::progress::begin(ds.features.len()); - crate::source::gpkg::write_in(ds, path, crs)?; - Ok(0) + let table = crate::source::gpkg::write_table_in(ds, path, crs, gpkg)?; + Ok((0, Some(table))) } #[cfg(target_arch = "wasm32")] - Err(DataError::Unsupported("GeoPackage in the browser".into())) + { + let _ = gpkg; + Err(DataError::Unsupported("GeoPackage in the browser".into())) + } } "shp" => { crate::progress::begin(ds.features.len()); @@ -592,7 +630,7 @@ pub fn write_path_in(ds: &Dataset, path: &Path, crs: Crs) -> Result { for stale in ["qix", "sbn", "sbx"] { let _ = std::fs::remove_file(path.with_extension(stale)); } - Ok(left_out) + Ok((left_out, None)) } _ if crs != Crs::LonLat && !PROJECTED_FORMATS.contains(&ext.as_str()) => { Err(DataError::Unsupported(format!(".{ext} in {} (it holds longitude/latitude only)", crs.label()))) @@ -600,7 +638,7 @@ pub fn write_path_in(ds: &Dataset, path: &Path, crs: Crs) -> Result { _ => { let (bytes, left_out) = projected_bytes(ds, &ext, crs)?; write_atomic(path, &bytes)?; - Ok(left_out) + Ok((left_out, None)) } } } diff --git a/crates/data/src/source/gpkg.rs b/crates/data/src/source/gpkg.rs index 966152d2f..2d4eb7c26 100644 --- a/crates/data/src/source/gpkg.rs +++ b/crates/data/src/source/gpkg.rs @@ -13,8 +13,11 @@ use rustgis_geom::Extent; use std::path::{Path, PathBuf}; use std::sync::Mutex; +mod add; pub mod edit; +pub use crate::io::{GpkgMode, GpkgTable, GpkgWritten}; + /// Rows fetched by id per query. const CHUNK: usize = 500; @@ -533,14 +536,38 @@ pub fn write(ds: &crate::Dataset, path: &Path) -> Result<()> { /// Writes a dataset whose geometry is in `crs`'s coordinates as a new GeoPackage in that system. /// Existing destinations are refused, as with [`write`]. pub fn write_in(ds: &crate::Dataset, path: &Path, crs: Crs) -> Result<()> { + write_table_in(ds, path, crs, &GpkgTable::default()).map(|_| ()) +} + +/// As [`write_in`], the table placed as `target` says: in a new file, or added to (or replacing +/// a table of) the GeoPackage at `path` when it exists, its other tables left as they were. +pub fn write_table_in(ds: &crate::Dataset, path: &Path, crs: Crs, target: &GpkgTable) -> Result { + let name = target.name.as_deref().unwrap_or(&ds.name); + if let Some(replace) = into_existing(path, target)? { + return add::write_features(path, name, ds.geometry_type, &ds.fields, crs, replace, &mut |put| put(&ds.features)); + } check_new_destination(path)?; let tmp = temp_database_beside(path)?; let conn = Connection::open(&tmp).map_err(err)?; - build(&conn, ds, crs)?; - install(conn, tmp, path) + let table = build(&conn, ds, name, crs)?; + install(conn, tmp, path)?; + Ok(GpkgWritten { table, replaced: false, dependents: Vec::new() }) +} + +/// Whether an export goes into the existing GeoPackage at `path` (anything there, which opening +/// it as one then checks) and, if so, whether it replaces a table: `None` for a new file. A +/// table is added to a new file when there is none; one can't be replaced in a file not there. +fn into_existing(path: &Path, target: &GpkgTable) -> Result> { + let exists = std::fs::symlink_metadata(path).is_ok(); + match target.mode { + GpkgMode::NewFile => Ok(None), + GpkgMode::AddTable => Ok(exists.then_some(false)), + GpkgMode::ReplaceTable if exists => Ok(Some(true)), + GpkgMode::ReplaceTable => Err(DataError::Format(format!("{}: no GeoPackage there to replace a table in", path.display()))), + } } -/// As [`write_in`], for a layer `name` of `geometry_type` with `fields`: the features `feed` +/// As [`write_table_in`], for a layer `name` of `geometry_type` with `fields`: the features `feed` /// hands to the function it is given, a run at a time, already in `crs`'s coordinates, so a /// whole layer needn't be held. A point layer's table type and whether ids are kept are chosen /// from the features so far; should a later one contradict that choice, the export starts @@ -551,49 +578,74 @@ pub(crate) fn write_features_in( fields: &[Field], path: &Path, crs: Crs, + target: &GpkgTable, feed: &mut dyn FnMut(&mut dyn FnMut(&[crate::Feature]) -> Result<()>) -> Result<()>, -) -> Result<()> { +) -> Result { + let name = target.name.as_deref().unwrap_or(name); + if let Some(replace) = into_existing(path, target)? { + return add::write_features(path, name, geometry_type, fields, crs, replace, feed); + } check_new_destination(path)?; let (mut multi_points, mut keep_ids) = (None, true); // Each choice changes at most once. for _ in 0..3 { let tmp = temp_database_beside(path)?; let conn = Connection::open(&tmp).map_err(err)?; - let (mut layout, mut extent, mut redo) = (None, Extent::EMPTY, None); - let fed = feed(&mut |features| { - if layout.is_none() { - let multi = - multi_points.unwrap_or_else(|| features.iter().any(|f| matches!(f.geometry, Some(crate::Geometry::MultiPoint(_))))); - layout = Some(create(&conn, name, geometry_type, fields, crs, multi)?); - } - let Some(l) = layout.as_ref() else { return Err(DataError::Format("GeoPackage: no table to write".into())) }; - for g in features.iter().filter_map(|f| f.geometry.as_ref()) { - extent = extent.union(&g.extent()); - } - redo = insert(&conn, l, features, keep_ids)?; - match redo { - // Stops the traversal; the export starts again below. - Some(_) => Err(DataError::Format("GeoPackage: starting again".into())), - None => Ok(()), - } - }); - match (fed, redo) { - (_, Some(Redo::MultiPoints)) => multi_points = Some(true), - (_, Some(Redo::Renumber)) => keep_ids = false, - (Err(e), None) => return Err(e), - (Ok(()), None) => { - let layout = match layout { - Some(l) => l, - None => create(&conn, name, geometry_type, fields, crs, multi_points.unwrap_or(false))?, - }; + let mut make = |multi| create(&conn, name, geometry_type, fields, crs, multi); + match fill(&conn, multi_points, keep_ids, &mut make, feed)? { + Err(Redo::MultiPoints) => multi_points = Some(true), + Err(Redo::Renumber) => keep_ids = false, + Ok((layout, extent)) => { finish(&conn, &layout, extent)?; - return install(conn, tmp, path); + install(conn, tmp, path)?; + return Ok(GpkgWritten { table: layout.table, replaced: false, dependents: Vec::new() }); } } } Err(DataError::Format("GeoPackage: the export didn't settle on a table".into())) } +/// Feeds the features into the table `make` creates on `conn`, made once the first run shows +/// whether a point layer holds multipoints (unless `multi_points` already says): the table and +/// the extent of what went in, or what the export must start again for. With `keep_ids` each +/// feature keeps its id. +fn fill( + conn: &Connection, + multi_points: Option, + keep_ids: bool, + make: &mut dyn FnMut(bool) -> Result, + feed: &mut dyn FnMut(&mut dyn FnMut(&[crate::Feature]) -> Result<()>) -> Result<()>, +) -> Result> { + let (mut layout, mut extent, mut redo) = (None, Extent::EMPTY, None); + let fed = feed(&mut |features| { + if layout.is_none() { + let multi = multi_points.unwrap_or_else(|| features.iter().any(|f| matches!(f.geometry, Some(crate::Geometry::MultiPoint(_))))); + layout = Some(make(multi)?); + } + let Some(l) = layout.as_ref() else { return Err(DataError::Format("GeoPackage: no table to write".into())) }; + for g in features.iter().filter_map(|f| f.geometry.as_ref()) { + extent = extent.union(&g.extent()); + } + redo = insert(conn, l, features, keep_ids)?; + match redo { + // Stops the traversal; the export starts again. + Some(_) => Err(DataError::Format("GeoPackage: starting again".into())), + None => Ok(()), + } + }); + match (fed, redo) { + (_, Some(r)) => Ok(Err(r)), + (Err(e), None) => Err(e), + (Ok(()), None) => { + let layout = match layout { + Some(l) => l, + None => make(multi_points.unwrap_or(false))?, + }; + Ok(Ok((layout, extent))) + } + } +} + /// A new temporary file beside `path`, removed unless [`install`]ed. fn temp_database_beside(path: &Path) -> Result { let parent = path.parent().filter(|p| !p.as_os_str().is_empty()).unwrap_or_else(|| Path::new(".")); @@ -653,10 +705,11 @@ fn destination_error(path: &Path, error: std::io::Error) -> DataError { } } -/// Builds a whole layer's GeoPackage on `conn`, its table chosen from all its features. -fn build(conn: &Connection, ds: &crate::Dataset, crs: Crs) -> Result<()> { +/// Builds a whole layer's GeoPackage on `conn`, its table `name` chosen from all its features; +/// returns the table's name as written. +fn build(conn: &Connection, ds: &crate::Dataset, name: &str, crs: Crs) -> Result { let multi_points = ds.features.iter().any(|f| matches!(f.geometry, Some(crate::Geometry::MultiPoint(_)))); - let layout = create(conn, &ds.name, ds.geometry_type, &ds.fields, crs, multi_points)?; + let layout = create(conn, name, ds.geometry_type, &ds.fields, crs, multi_points)?; // Feature ids are kept when they are unique and fit; otherwise SQLite numbers the rows. let mut seen = std::collections::HashSet::new(); let keep_ids = ds.features.iter().all(|f| i64::try_from(f.id).is_ok() && seen.insert(f.id)); @@ -665,7 +718,8 @@ fn build(conn: &Connection, ds: &crate::Dataset, crs: Crs) -> Result<()> { if insert(conn, &layout, &ds.features, keep_ids)?.is_some() { return Err(DataError::Format("GeoPackage: a feature doesn't fit the table chosen for the layer".into())); } - finish(conn, &layout, ds.extent()) + finish(conn, &layout, ds.extent())?; + Ok(layout.table) } /// A feature table being written: its names, and what its rows hold. @@ -685,6 +739,25 @@ struct Layout { rtree_q: String, } +/// The GeoPackage's own tables, as a new file has them. +const METADATA: &str = "CREATE TABLE gpkg_spatial_ref_sys (srs_name TEXT NOT NULL, srs_id INTEGER PRIMARY KEY, organization TEXT NOT NULL, + organization_coordsys_id INTEGER NOT NULL, definition TEXT NOT NULL, description TEXT); + CREATE TABLE gpkg_contents (table_name TEXT NOT NULL PRIMARY KEY, data_type TEXT NOT NULL, identifier TEXT UNIQUE, + description TEXT DEFAULT '', last_change DATETIME NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), + min_x DOUBLE, min_y DOUBLE, max_x DOUBLE, max_y DOUBLE, srs_id INTEGER, + CONSTRAINT fk_gc_r_srs_id FOREIGN KEY (srs_id) REFERENCES gpkg_spatial_ref_sys(srs_id));"; + +/// The tables a GeoPackage holding a feature table with a spatial index needs beyond +/// [`METADATA`], made where missing (an existing GeoPackage may have only attribute tables). +const FEATURE_METADATA: &str = "CREATE TABLE IF NOT EXISTS gpkg_geometry_columns (table_name TEXT NOT NULL, column_name TEXT NOT NULL, + geometry_type_name TEXT NOT NULL, srs_id INTEGER NOT NULL, z TINYINT NOT NULL, m TINYINT NOT NULL, + CONSTRAINT pk_geom_cols PRIMARY KEY (table_name, column_name), + CONSTRAINT uk_gc_table_name UNIQUE (table_name), + CONSTRAINT fk_gc_tn FOREIGN KEY (table_name) REFERENCES gpkg_contents(table_name), + CONSTRAINT fk_gc_srs FOREIGN KEY (srs_id) REFERENCES gpkg_spatial_ref_sys (srs_id)); + CREATE TABLE IF NOT EXISTS gpkg_extensions (table_name TEXT, column_name TEXT, extension_name TEXT NOT NULL, + definition TEXT NOT NULL, scope TEXT NOT NULL, CONSTRAINT ge_tce UNIQUE (table_name, column_name, extension_name));"; + /// Creates the GeoPackage's tables and the empty feature table (and its spatial index) for a /// layer, in a transaction [`finish`] commits. A point layer holding any multipoint is a /// MULTIPOINT table, whose single points [`insert`] writes as one-point multipoints. @@ -694,7 +767,40 @@ fn create(conn: &Connection, name: &str, geometry_type: GeometryType, fields: &[ 0 => 100_000, c => i32::from(c), }; - // Not a reserved name, and never empty. + let layout = plan(name, geometry_type, fields, multi_points, srs_id); + conn.execute_batch(&format!( + "PRAGMA journal_mode = OFF; PRAGMA synchronous = OFF; + PRAGMA application_id = 1196444487; PRAGMA user_version = 10400; + {METADATA} {FEATURE_METADATA} BEGIN;" + )) + .map_err(err)?; + create_table(conn, &layout)?; + conn.execute( + "INSERT INTO gpkg_spatial_ref_sys VALUES + ('Undefined cartesian SRS', -1, 'NONE', -1, 'undefined', 'undefined cartesian coordinate reference system'), + ('Undefined geographic SRS', 0, 'NONE', 0, 'undefined', 'undefined geographic coordinate reference system'), + ('WGS 84 geodetic', 4326, 'EPSG', 4326, ?1, 'longitude/latitude coordinates in decimal degrees on the WGS 84 spheroid')", + [WGS84_WKT], + ) + .map_err(err)?; + if srs_id != 4326 { + // GeoPackage readers take an EPSG system by its code; a custom one has only its WKT + // (`undefined` when it has none). + let org = if crs.epsg() > 0 { "EPSG" } else { "NONE" }; + let wkt = crate::raster::wkt::to_wkt(crs); + conn.execute( + "INSERT INTO gpkg_spatial_ref_sys VALUES (?1, ?2, ?3, ?2, ?4, '')", + rusqlite::params![crs.label(), srs_id, org, wkt.as_deref().unwrap_or("undefined")], + ) + .map_err(err)?; + } + Ok(layout) +} + +/// The names a layer's feature table is written with, in spatial reference system `srs_id`: the +/// table named for the layer (not a reserved name, never empty), its id and geometry columns +/// clear of its fields, and its fields' columns unique without regard to case. +fn plan(name: &str, geometry_type: GeometryType, fields: &[Field], multi_points: bool, srs_id: i32) -> Layout { let mut table: String = name.chars().filter(|c| !c.is_control()).collect::().trim().to_string(); if table.is_empty() { table = "layer".into(); @@ -728,65 +834,8 @@ fn create(conn: &Connection, name: &str, geometry_type: GeometryType, fields: &[ GeometryType::None => "", }; let (t, g, i) = (quote(&table), quote(&geom), quote(&fid)); - let mut sql = String::from( - "PRAGMA journal_mode = OFF; PRAGMA synchronous = OFF; - PRAGMA application_id = 1196444487; PRAGMA user_version = 10400; - CREATE TABLE gpkg_spatial_ref_sys (srs_name TEXT NOT NULL, srs_id INTEGER PRIMARY KEY, organization TEXT NOT NULL, - organization_coordsys_id INTEGER NOT NULL, definition TEXT NOT NULL, description TEXT); - CREATE TABLE gpkg_contents (table_name TEXT NOT NULL PRIMARY KEY, data_type TEXT NOT NULL, identifier TEXT UNIQUE, - description TEXT DEFAULT '', last_change DATETIME NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), - min_x DOUBLE, min_y DOUBLE, max_x DOUBLE, max_y DOUBLE, srs_id INTEGER, - CONSTRAINT fk_gc_r_srs_id FOREIGN KEY (srs_id) REFERENCES gpkg_spatial_ref_sys(srs_id)); - CREATE TABLE gpkg_geometry_columns (table_name TEXT NOT NULL, column_name TEXT NOT NULL, geometry_type_name TEXT NOT NULL, - srs_id INTEGER NOT NULL, z TINYINT NOT NULL, m TINYINT NOT NULL, - CONSTRAINT pk_geom_cols PRIMARY KEY (table_name, column_name), - CONSTRAINT uk_gc_table_name UNIQUE (table_name), - CONSTRAINT fk_gc_tn FOREIGN KEY (table_name) REFERENCES gpkg_contents(table_name), - CONSTRAINT fk_gc_srs FOREIGN KEY (srs_id) REFERENCES gpkg_spatial_ref_sys (srs_id)); - CREATE TABLE gpkg_extensions (table_name TEXT, column_name TEXT, extension_name TEXT NOT NULL, definition TEXT NOT NULL, - scope TEXT NOT NULL, CONSTRAINT ge_tce UNIQUE (table_name, column_name, extension_name)); - BEGIN;", - ); - let mut cols = vec![format!("{i} INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL")]; - if has_geom { - cols.push(format!("{g} {geom_type}")); - } - for (f, col) in fields.iter().zip(&columns) { - let ty = match f.ty { - FieldType::Integer => "INTEGER", - FieldType::Double => "REAL", - FieldType::Boolean => "BOOLEAN", - FieldType::Text => "TEXT", - }; - cols.push(format!("{} {ty}", quote(col))); - } - sql.push_str(&format!("CREATE TABLE {t} ({});", cols.join(", "))); let rt = quote(&format!("rtree_{table}_{geom}")); - if has_geom { - sql.push_str(&format!("CREATE VIRTUAL TABLE {rt} USING rtree(id, minx, maxx, miny, maxy);")); - } - conn.execute_batch(&sql).map_err(err)?; - conn.execute( - "INSERT INTO gpkg_spatial_ref_sys VALUES - ('Undefined cartesian SRS', -1, 'NONE', -1, 'undefined', 'undefined cartesian coordinate reference system'), - ('Undefined geographic SRS', 0, 'NONE', 0, 'undefined', 'undefined geographic coordinate reference system'), - ('WGS 84 geodetic', 4326, 'EPSG', 4326, ?1, 'longitude/latitude coordinates in decimal degrees on the WGS 84 spheroid')", - [WGS84_WKT], - ) - .map_err(err)?; - if srs_id != 4326 { - // GeoPackage readers take an EPSG system by its code; a custom one has only its WKT - // (`undefined` when it has none). - let org = if crs.epsg() > 0 { "EPSG" } else { "NONE" }; - let wkt = crate::raster::wkt::to_wkt(crs); - conn.execute( - "INSERT INTO gpkg_spatial_ref_sys VALUES (?1, ?2, ?3, ?2, ?4, '')", - rusqlite::params![crs.label(), srs_id, org, wkt.as_deref().unwrap_or("undefined")], - ) - .map_err(err)?; - } - - Ok(Layout { + Layout { table, geom, columns, @@ -799,7 +848,30 @@ fn create(conn: &Connection, name: &str, geometry_type: GeometryType, fields: &[ geom_q: g, fid_q: i, rtree_q: rt, - }) + } +} + +/// Creates the empty feature table [`plan`] named, and its spatial index. +fn create_table(conn: &Connection, layout: &Layout) -> Result<()> { + let Layout { columns, fields, has_geom, geom_type, table_q: t, geom_q: g, fid_q: i, rtree_q: rt, .. } = layout; + let mut cols = vec![format!("{i} INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL")]; + if *has_geom { + cols.push(format!("{g} {geom_type}")); + } + for (f, col) in fields.iter().zip(columns) { + let ty = match f.ty { + FieldType::Integer => "INTEGER", + FieldType::Double => "REAL", + FieldType::Boolean => "BOOLEAN", + FieldType::Text => "TEXT", + }; + cols.push(format!("{} {ty}", quote(col))); + } + let mut sql = format!("CREATE TABLE {t} ({});", cols.join(", ")); + if *has_geom { + sql.push_str(&format!("CREATE VIRTUAL TABLE {rt} USING rtree(id, minx, maxx, miny, maxy);")); + } + conn.execute_batch(&sql).map_err(err) } /// What a streamed export found it must start again for, having chosen from the features so far. @@ -864,22 +936,54 @@ fn insert(conn: &Connection, layout: &Layout, features: &[crate::Feature], keep_ Ok(None) } -/// Records the table in the GeoPackage's contents with its `extent`, adds the triggers that keep -/// its spatial index in step with edits, and commits. +/// [`register`]s the table and commits. fn finish(conn: &Connection, layout: &Layout, extent: Extent) -> Result<()> { + register(conn, layout, extent)?; + conn.execute_batch("COMMIT;").map_err(err) +} + +/// Records the table in the GeoPackage's contents with its `extent` (identified by its table +/// name, numbered if another table already has that identifier) and adds the triggers that keep its +/// spatial index in step with edits. +fn register(conn: &Connection, layout: &Layout, extent: Extent) -> Result<()> { let Layout { table, geom, has_geom, geom_type, srs_id, table_q: t, geom_q: g, fid_q: i, rtree_q: rt, .. } = layout; let (has_geom, srs_id) = (*has_geom, *srs_id); let e = extent; let bounds: [Option; 4] = if e.is_empty() { [None; 4] } else { [Some(e.min_x), Some(e.min_y), Some(e.max_x), Some(e.max_y)] }; let srs: Option = has_geom.then_some(i64::from(srs_id)); + // The table's name, numbered when another table of an existing GeoPackage has it as its + // (unique) identifier. + let mut identifier = table.clone(); + for i in 2.. { + let taken: bool = conn + .query_row("SELECT EXISTS (SELECT 1 FROM gpkg_contents WHERE identifier = ?1)", [&identifier], |r| r.get(0)) + .map_err(err)?; + if !taken { + break; + } + identifier = format!("{table}_{i}"); + } conn.execute( - "INSERT INTO gpkg_contents (table_name, data_type, identifier, min_x, min_y, max_x, max_y, srs_id) VALUES (?1, ?2, ?1, ?3, ?4, ?5, ?6, ?7)", - rusqlite::params![table, if has_geom { "features" } else { "attributes" }, bounds[0], bounds[1], bounds[2], bounds[3], srs], + "INSERT INTO gpkg_contents (table_name, data_type, identifier, last_change, min_x, min_y, max_x, max_y, srs_id) + VALUES (?1, ?2, ?3, strftime('%Y-%m-%dT%H:%M:%fZ', 'now'), ?4, ?5, ?6, ?7, ?8)", + rusqlite::params![ + table, + if has_geom { "features" } else { "attributes" }, + identifier, + bounds[0], + bounds[1], + bounds[2], + bounds[3], + srs + ], ) .map_err(err)?; if has_geom { - conn.execute("INSERT INTO gpkg_geometry_columns VALUES (?1, ?2, ?3, ?4, 0, 0)", rusqlite::params![table, geom, geom_type, srs_id]) - .map_err(err)?; + conn.execute( + "INSERT INTO gpkg_geometry_columns (table_name, column_name, geometry_type_name, srs_id, z, m) VALUES (?1, ?2, ?3, ?4, 0, 0)", + rusqlite::params![table, geom, geom_type, srs_id], + ) + .map_err(err)?; // The GeoPackage 1.4 triggers that keep the index in step with edits. let trig = |s: &str| quote(&format!("rtree_{table}_{geom}_{s}")); conn.execute_batch(&format!( @@ -911,12 +1015,12 @@ fn finish(conn: &Connection, layout: &Layout, extent: Extent) -> Result<()> { )) .map_err(err)?; conn.execute( - "INSERT INTO gpkg_extensions VALUES (?1, ?2, 'gpkg_rtree_index', 'http://www.geopackage.org/spec120/#extension_rtree', 'write-only')", + "INSERT INTO gpkg_extensions (table_name, column_name, extension_name, definition, scope) + VALUES (?1, ?2, 'gpkg_rtree_index', 'http://www.geopackage.org/spec120/#extension_rtree', 'write-only')", rusqlite::params![table, geom], ) .map_err(err)?; } - conn.execute_batch("COMMIT;").map_err(err)?; Ok(()) } diff --git a/crates/data/src/source/gpkg/add.rs b/crates/data/src/source/gpkg/add.rs new file mode 100644 index 000000000..1611fcc95 --- /dev/null +++ b/crates/data/src/source/gpkg/add.rs @@ -0,0 +1,292 @@ +//! Exports into an existing GeoPackage: one table added, or one of its feature or attribute +//! tables replaced, in a single transaction that leaves the file's other tables as they were. + +use super::{FEATURE_METADATA, GpkgWritten, Redo, WGS84_WKT, create_table, err, fill, plan, quote, register}; +use crate::raster::Crs; +use crate::{DataError, Field, GeometryType, Result}; +use rusqlite::config::DbConfig; +use rusqlite::{Connection, OpenFlags, OptionalExtension, TransactionBehavior}; +use std::cell::Cell; +use std::path::Path; +use std::time::Duration; + +/// Writes the features `feed` hands over as table `name` of the GeoPackage at `path` (see +/// [`super::write_features_in`]). A table of that name is refused, or, with `replace`, required: +/// a feature or attribute table, dropped first; nothing else in the file is touched. Every +/// change is one transaction, so a failed export leaves the file as it was. +pub(super) fn write_features( + path: &Path, + name: &str, + geometry_type: GeometryType, + fields: &[Field], + crs: Crs, + replace: bool, + feed: &mut dyn FnMut(&mut dyn FnMut(&[crate::Feature]) -> Result<()>) -> Result<()>, +) -> Result { + let file = path.file_name().unwrap_or_default().to_string_lossy().into_owned(); + // Never made here: the file must exist, as checked before. + let mut conn = Connection::open_with_flags(path, OpenFlags::SQLITE_OPEN_READ_WRITE | OpenFlags::SQLITE_OPEN_NO_MUTEX).map_err(err)?; + // The file may be hostile: what it defines (triggers, views) may not call functions, nor write + // to SQLite's own records. + conn.set_db_config(DbConfig::SQLITE_DBCONFIG_DEFENSIVE, true).map_err(err)?; + conn.set_db_config(DbConfig::SQLITE_DBCONFIG_TRUSTED_SCHEMA, false).map_err(err)?; + // Nor may its triggers run when the export writes its GeoPackage records: one could change + // other tables. The new table's own triggers are made after its rows are in. + conn.set_db_config(DbConfig::SQLITE_DBCONFIG_ENABLE_TRIGGER, false).map_err(err)?; + // Nor may its foreign keys turn the export's deletes (of a replaced table's GeoPackage + // records) into deletes elsewhere; a table referring to the replaced one is refused instead. + conn.execute_batch("PRAGMA foreign_keys = OFF").map_err(err)?; + conn.busy_timeout(Duration::from_secs(5)).map_err(err)?; + let (mut multi_points, mut keep_ids) = (None, true); + // Each choice changes at most once; an export starting again rolls back what it wrote. + for _ in 0..3 { + let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate).map_err(err)?; + check_geopackage(&tx, &file)?; + // An attribute table has no spatial reference system. + let srs_id = if geometry_type == GeometryType::None { 0 } else { srs_id(&tx, crs)? }; + // The views and triggers naming a replaced table, when one is. + let replaced = Cell::new(None); + let mut make = |multi| { + let layout = plan(name, geometry_type, fields, multi, srs_id); + replaced.set(make_room(&tx, &file, &layout.table, replace)?); + if layout.has_geom { + tx.execute_batch(FEATURE_METADATA).map_err(err)?; + let index = format!("rtree_{}_{}", layout.table, layout.geom); + if let Some(kind) = object(&tx, &index)? { + return Err(DataError::Format(format!( + "{file} already has {} named {index}, which the table's spatial index needs; choose another table name", + with_article(&kind) + ))); + } + } + create_table(&tx, &layout)?; + Ok(layout) + }; + match fill(&tx, multi_points, keep_ids, &mut make, feed)? { + Err(Redo::MultiPoints) => multi_points = Some(true), + Err(Redo::Renumber) => keep_ids = false, + Ok((layout, extent)) => { + register(&tx, &layout, extent)?; + // A cancel that came while it was written rolls it all back; the last look, as + // committing writes the table, and a Cancel after it finds it written. + crate::progress::check()?; + tx.commit().map_err(err)?; + let dependents = replaced.take(); + return Ok(GpkgWritten { table: layout.table, replaced: dependents.is_some(), dependents: dependents.unwrap_or_default() }); + } + } + } + Err(DataError::Format("GeoPackage: the export didn't settle on a table".into())) +} + +/// Refuses a database without a GeoPackage's contents and spatial reference systems, or marked +/// as another application's (an unmarked one, as older programs write, is taken). +fn check_geopackage(conn: &Connection, file: &str) -> Result<()> { + // "GPKG", and GeoPackage 1.0's and 1.1's "GP10" and "GP11". + let id: i64 = conn.query_row("PRAGMA application_id", [], |r| r.get(0)).map_err(err)?; + if !matches!(id, 0 | 0x4750_4B47 | 0x4750_3130 | 0x4750_3131) { + return Err(DataError::Format(format!("{file}: not a GeoPackage (its application id is {id:#010x})"))); + } + for table in ["gpkg_contents", "gpkg_spatial_ref_sys"] { + if object(conn, table)?.as_deref() != Some("table") { + return Err(DataError::Format(format!("{file}: not a GeoPackage (it has no {table} table)"))); + } + } + Ok(()) +} + +/// What kind of SQLite object (`table`, `view`, `index`, `trigger`) is named `name`, without +/// regard to case as SQLite names are. +fn object(conn: &Connection, name: &str) -> Result> { + conn.query_row("SELECT type FROM sqlite_master WHERE lower(name) = lower(?1) LIMIT 1", [name], |r| r.get(0)).optional().map_err(err) +} + +/// The GeoPackage's id for `crs`: an entry it has for the EPSG code, or with the same definition +/// for a system without one; else one added, numbered by its EPSG code where that is free and +/// otherwise from 100000 up, as a new file's custom system is. +fn srs_id(conn: &Connection, crs: Crs) -> Result { + let epsg = crs.epsg(); + let wkt = if epsg == 4326 { Some(WGS84_WKT.to_string()) } else { crate::raster::wkt::to_wkt(crs) }; + let found: Option = if epsg > 0 { + conn.query_row( + "SELECT srs_id FROM gpkg_spatial_ref_sys WHERE upper(organization) = 'EPSG' AND organization_coordsys_id = ?1 ORDER BY srs_id LIMIT 1", + [i64::from(epsg)], + |r| r.get(0), + ) + .optional() + .map_err(err)? + } else { + // A system without a WKT form is found by its name. + let (sql, key) = match &wkt { + Some(wkt) => ("SELECT srs_id FROM gpkg_spatial_ref_sys WHERE definition = ?1 ORDER BY srs_id LIMIT 1", wkt.clone()), + None => ( + "SELECT srs_id FROM gpkg_spatial_ref_sys WHERE srs_name = ?1 AND definition = 'undefined' AND srs_id >= 100000 ORDER BY srs_id LIMIT 1", + crs.label(), + ), + }; + conn.query_row(sql, [key], |r| r.get(0)).optional().map_err(err)? + }; + let id = match found { + Some(id) => id, + None => { + let preferred = if epsg > 0 { i64::from(epsg) } else { 100_000 }; + let taken: bool = conn + .query_row("SELECT EXISTS (SELECT 1 FROM gpkg_spatial_ref_sys WHERE srs_id = ?1)", [preferred], |r| r.get(0)) + .map_err(err)?; + let id = if taken { + conn.query_row("SELECT max(100000, max(srs_id) + 1) FROM gpkg_spatial_ref_sys", [], |r| r.get(0)).map_err(err)? + } else { + preferred + }; + let (org, code) = if epsg > 0 { ("EPSG", i64::from(epsg)) } else { ("NONE", id) }; + conn.execute( + "INSERT INTO gpkg_spatial_ref_sys (srs_name, srs_id, organization, organization_coordsys_id, definition, description) + VALUES (?1, ?2, ?3, ?4, ?5, '')", + rusqlite::params![crs.label(), id, org, code, wkt.as_deref().unwrap_or("undefined")], + ) + .map_err(err)?; + id + } + }; + i32::try_from(id).map_err(|_| DataError::Format(format!("GeoPackage: spatial reference system id {id} is out of range"))) +} + +/// Clears the way for table `table`: nothing of that name, or (with `replace`, which needs one) a +/// feature or attribute table of it, dropped with its spatial index and its GeoPackage records. +/// For a table replaced, returns the views and other tables' triggers whose SQL names it (they +/// read the new table, and break if a column they use is gone); anything else of the name is +/// refused. +fn make_room(conn: &Connection, file: &str, table: &str, replace: bool) -> Result>> { + let registered: Option<(String, String)> = conn + .query_row("SELECT table_name, data_type FROM gpkg_contents WHERE lower(table_name) = lower(?1)", [table], |r| { + Ok((r.get(0)?, r.get(1)?)) + }) + .optional() + .map_err(err)?; + let kind = object(conn, table)?; + match (kind.as_deref(), registered) { + // Replacing a table not there is most likely a misspelt name: the stale table would stay. + (None, None) if replace => { + Err(DataError::Format(format!("{file} has no table {table} to replace; check the name, or add the layer as a new table"))) + } + (None, None) => Ok(None), + (None, Some((old, _))) => { + Err(DataError::Format(format!("{file} lists a table {old} in its contents that it doesn't have; choose another table name"))) + } + (Some("table"), Some((old, data_type))) if matches!(data_type.as_str(), "features" | "attributes") => { + if !replace { + return Err(DataError::Format(format!( + "{file} already has a table {old}: choose another table name, or replace that table" + ))); + } + let children = referencing(conn, &old)?; + if !children.is_empty() { + return Err(DataError::Format(format!( + "{file}: {} refer to {old} by foreign key, whose rows a replaced table would delete or break; choose another table name", + children.join(", ") + ))); + } + if related(conn, &old)? { + return Err(DataError::Format(format!( + "{file}: {old} is in a related-tables relation, whose links a replaced table would break; choose another table name" + ))); + } + let dependents = dependents(conn, &old)?; + drop_table(conn, &old)?; + Ok(Some(dependents)) + } + (_, Some((old, data_type))) => Err(DataError::Format(format!( + "{file}: {old} is a {data_type} table, which an export doesn't replace; choose another table name" + ))), + (Some(kind), None) => Err(DataError::Format(format!( + "{file} already has {} named {table}, which an export doesn't replace; choose another table name", + with_article(kind) + ))), + } +} + +/// The other tables with a foreign key to `table`. Dropping it would delete or empty their rows +/// (SQLite enforces foreign keys here), or leave them pointing at rows that aren't the same. +fn referencing(conn: &Connection, table: &str) -> Result> { + let mut st = conn + .prepare( + "SELECT DISTINCT m.name FROM sqlite_master m, pragma_foreign_key_list(m.name) f + WHERE m.type = 'table' AND lower(m.name) != lower(?1) AND lower(f.\"table\") = lower(?1) ORDER BY m.name", + ) + .map_err(err)?; + let names = st.query_map([table], |r| r.get::<_, String>(0)).map_err(err)?; + names.collect::>>().map_err(err) +} + +/// Whether the GeoPackage's related-tables extension names `table` as a base, related or mapping +/// table: its rows are linked by id, which a new table's rows don't keep. +fn related(conn: &Connection, table: &str) -> Result { + if object(conn, "gpkgext_relations")?.as_deref() != Some("table") { + return Ok(false); + } + conn.query_row( + "SELECT EXISTS (SELECT 1 FROM gpkgext_relations WHERE lower(?1) IN + (lower(base_table_name), lower(related_table_name), lower(mapping_table_name)))", + [table], + |r| r.get(0), + ) + .map_err(err) +} + +/// The views, and triggers on other tables, whose SQL mentions `table` (by a case-blind search, +/// so a longer name containing it counts too), by name. +fn dependents(conn: &Connection, table: &str) -> Result> { + let mut st = conn + .prepare( + "SELECT name FROM sqlite_master WHERE type IN ('view', 'trigger') AND lower(tbl_name) != lower(?1) + AND instr(lower(sql), lower(?1)) > 0 ORDER BY name", + ) + .map_err(err)?; + let names = st.query_map([table], |r| r.get::<_, String>(0)).map_err(err)?; + names.collect::>>().map_err(err) +} + +/// An SQLite object's kind with its article: "an index", "a view". +fn with_article(kind: &str) -> String { + if kind.starts_with(['a', 'e', 'i', 'o', 'u']) { format!("an {kind}") } else { format!("a {kind}") } +} + +/// Drops feature or attribute table `table`, its spatial indexes, and its rows in the GeoPackage +/// tables that describe it (the optional ones where the file has them). A table named as its +/// spatial index that isn't an R-tree is left, and the new table's index then refused. +fn drop_table(conn: &Connection, table: &str) -> Result<()> { + let has = |t: &str| object(conn, t).map(|k| k.as_deref() == Some("table")); + let mut sql = String::new(); + if has("gpkg_geometry_columns")? { + let mut st = conn.prepare("SELECT column_name FROM gpkg_geometry_columns WHERE lower(table_name) = lower(?1)").map_err(err)?; + let columns = st.query_map([table], |r| r.get::<_, String>(0)).map_err(err)?.collect::>>().map_err(err)?; + for c in columns { + let index = format!("rtree_{table}_{c}"); + let rtree: bool = conn + .query_row( + // Its declaration without spaces, line breaks and quotes of any kind, so `USING "rtree" (` + // counts and `USING rtree_i32(` doesn't. + "SELECT EXISTS (SELECT 1 FROM sqlite_master WHERE type = 'table' AND lower(name) = lower(?1) + AND replace(replace(replace(replace(replace(replace(replace(replace(lower(sql), ' ', ''), char(9), ''), char(10), ''), char(13), ''), '\"', ''), '`', ''), '[', ''), ']', '') + LIKE 'createvirtualtable%usingrtree(%')", + [&index], + |r| r.get(0), + ) + .map_err(err)?; + if rtree { + sql.push_str(&format!("DROP TABLE {};", quote(&index))); + } + } + } + sql.push_str(&format!("DROP TABLE {};", quote(table))); + conn.execute_batch(&sql).map_err(err)?; + // `gpkg_ogr_contents` is the feature count GDAL keeps, which would otherwise outlive the table. + for records in + ["gpkg_geometry_columns", "gpkg_extensions", "gpkg_data_columns", "gpkg_metadata_reference", "gpkg_ogr_contents", "gpkg_contents"] + { + if has(records)? { + conn.execute(&format!("DELETE FROM {records} WHERE lower(table_name) = lower(?1)"), [table]).map_err(err)?; + } + } + Ok(()) +} diff --git a/crates/data/src/source/tests.rs b/crates/data/src/source/tests.rs index 62ef213f4..531800ed6 100644 --- a/crates/data/src/source/tests.rs +++ b/crates/data/src/source/tests.rs @@ -2334,3 +2334,225 @@ fn flatgeobuf_binary_columns_are_said_and_left_out() { let src = open_path(&clean, None).unwrap(); assert!(src.info().notes.is_empty(), "{:?}", src.info().notes); } + +/// The tables of a GeoPackage, by name. +fn gpkg_tables(path: &Path) -> Vec { + let c = rusqlite::Connection::open(path).unwrap(); + let mut st = c.prepare("SELECT table_name FROM gpkg_contents ORDER BY table_name").unwrap(); + st.query_map([], |r| r.get(0)).unwrap().collect::>().unwrap() +} + +/// An export into an existing GeoPackage adds a table beside the others, refuses one whose name +/// is taken, and replaces a feature or attribute table (its spatial index too) when asked; the +/// other tables, an attribute table among them, are as they were. +#[test] +fn geopackage_export_adds_and_replaces_tables_and_keeps_the_others() { + use crate::io::{GpkgMode, GpkgTable, GpkgWritten, write_path_to}; + let workspace = tempfile::tempdir().unwrap(); + let path = workspace.path().join("county.gpkg"); + let mut parcels = Dataset::new("parcels", GeometryType::Point, vec![Field::new("apn", FieldType::Text)]); + parcels.push(Some(Geometry::Point(Coord::new(1.0, 2.0))), vec![Value::Text("001".into())]); + crate::io::write_path(&parcels, &path).unwrap(); + // An attribute table, and the feature counts GDAL keeps. + rusqlite::Connection::open(&path) + .unwrap() + .execute_batch( + "CREATE TABLE owners (name TEXT); INSERT INTO owners VALUES ('Shelby'); + INSERT INTO gpkg_contents (table_name, data_type, identifier) VALUES ('owners', 'attributes', 'owners'); + CREATE TABLE gpkg_ogr_contents (table_name TEXT NOT NULL PRIMARY KEY, feature_count INTEGER DEFAULT NULL); + INSERT INTO gpkg_ogr_contents VALUES ('parcels', 1); + CREATE TRIGGER wipe AFTER INSERT ON gpkg_contents BEGIN DELETE FROM owners; END;", + ) + .unwrap(); + let mut permits = Dataset::new("permits", GeometryType::Point, vec![Field::new("n", FieldType::Integer)]); + for i in 0..3 { + permits.push(Some(Geometry::Point(Coord::new(10.0 + i as f64, 20.0))), vec![Value::Int(i)]); + } + let to = |mode, name: Option<&str>| GpkgTable { mode, name: name.map(String::from) }; + let written = write_path_to(&permits, &path, Crs::LonLat, &to(GpkgMode::AddTable, None)).unwrap(); + assert_eq!(written, (0, Some(GpkgWritten { table: "permits".into(), replaced: false, dependents: vec![] }))); + rusqlite::Connection::open(&path).unwrap().execute_batch("INSERT INTO gpkg_ogr_contents VALUES ('permits', 3)").unwrap(); + assert_eq!(gpkg_tables(&path), ["owners", "parcels", "permits"]); + // The file's own trigger on its contents didn't run. + let owners = + |p: &Path| rusqlite::Connection::open(p).unwrap().query_row("SELECT count(*) FROM owners", [], |r| r.get::<_, i64>(0)).unwrap(); + assert_eq!(owners(&path), 1); + let read = |t: &str, extent| open_path(&path, Some(t)).unwrap().read(&Request { extent, limit: 100 }).unwrap().features; + assert_eq!(read("permits", None).len(), 3); + // Its spatial index answers an extent query. + assert_eq!(read("permits", Some(Extent::new(10.5, 19.0, 11.5, 21.0))).len(), 1); + assert_eq!(read("parcels", None)[0].attrs, [Value::Text("001".into())]); + + // A name taken (whatever its case) is refused unless replacing, and the file is unchanged. + let before = std::fs::read(&path).unwrap(); + let mut before_index = before.clone(); + for name in [None, Some("PERMITS"), Some("owners_idx")] { + if name == Some("owners_idx") { + assert_eq!(std::fs::read(&path).unwrap(), before, "the refusals changed nothing"); + rusqlite::Connection::open(&path).unwrap().execute_batch("CREATE INDEX owners_idx ON owners (name)").unwrap(); + before_index = std::fs::read(&path).unwrap(); + } + let e = write_path_to(&permits, &path, Crs::LonLat, &to(GpkgMode::AddTable, name)).unwrap_err().to_string(); + assert!(e.contains("already has"), "{name:?}: {e}"); + } + assert_eq!(std::fs::read(&path).unwrap(), before_index, "the index refusal changed nothing"); + assert_ne!(before, before_index, "the index was added"); + + // Replacing: fewer features, in another system, and its old spatial index gone. A view on the + // table is named, as it reads the new one. + rusqlite::Connection::open(&path).unwrap().execute_batch("CREATE VIEW recent_permits AS SELECT * FROM permits").unwrap(); + let mut fewer = permits.clone(); + fewer.features.truncate(1); + let replaced = write_path_to(&fewer, &path, Crs::WebMercator, &to(GpkgMode::ReplaceTable, None)).unwrap(); + assert_eq!(replaced.1, Some(GpkgWritten { table: "permits".into(), replaced: true, dependents: vec!["recent_permits".into()] })); + assert_eq!(gpkg_tables(&path), ["owners", "parcels", "permits"]); + let s = open_path(&path, Some("permits")).unwrap(); + assert_eq!(s.info().count, Some(1)); + assert_eq!(s.info().coordinate_system(), Some(Crs::WebMercator)); + let c = rusqlite::Connection::open(&path).unwrap(); + let count = |sql: &str| c.query_row(sql, [], |r| r.get::<_, i64>(0)).unwrap(); + assert_eq!(count("SELECT count(*) FROM rtree_permits_geom"), 1); + assert_eq!(count("SELECT count(*) FROM gpkg_geometry_columns WHERE table_name = 'permits'"), 1); + assert_eq!(count("SELECT count(*) FROM gpkg_extensions WHERE table_name = 'permits'"), 1); + assert_eq!(count("SELECT count(*) FROM gpkg_spatial_ref_sys WHERE srs_id = 3857"), 1); + // The old table's cached count is gone with it; the others' are kept. + assert_eq!(count("SELECT count(*) FROM gpkg_ogr_contents WHERE table_name = 'permits'"), 0); + assert_eq!(count("SELECT count(*) FROM gpkg_ogr_contents WHERE table_name = 'parcels'"), 1); + assert_eq!(c.query_row("SELECT name FROM owners", [], |r| r.get::<_, String>(0)).unwrap(), "Shelby"); + drop(c); + // A second table in that system reuses its entry. + let more = write_path_to(&fewer, &path, Crs::WebMercator, &to(GpkgMode::AddTable, Some("permits_2027"))).unwrap(); + assert_eq!(more.1, Some(GpkgWritten { table: "permits_2027".into(), replaced: false, dependents: vec![] })); + // Replacing a table not there (a misspelt name) is refused rather than adding one. + let e = write_path_to(&fewer, &path, Crs::LonLat, &to(GpkgMode::ReplaceTable, Some("permit"))).unwrap_err().to_string(); + assert!(e.contains("no table permit to replace"), "{e}"); + let c = rusqlite::Connection::open(&path).unwrap(); + assert_eq!(c.query_row("SELECT count(*) FROM gpkg_spatial_ref_sys WHERE srs_id = 3857", [], |r| r.get::<_, i64>(0)).unwrap(), 1); + // A table without geometry replaces an attribute table, adding no spatial reference system. + let systems = || { + let c = rusqlite::Connection::open(&path).unwrap(); + c.query_row("SELECT count(*) FROM gpkg_spatial_ref_sys", [], |r| r.get::<_, i64>(0)).unwrap() + }; + let before_systems = systems(); + let mut owners = Dataset::new("owners", GeometryType::None, vec![Field::new("name", FieldType::Text)]); + owners.push(None, vec![Value::Text("Memphis".into())]); + owners.push(None, vec![Value::Text("Germantown".into())]); + drop(c); + write_path_to(&owners, &path, Crs::from_epsg(32616).unwrap(), &to(GpkgMode::ReplaceTable, None)).unwrap(); + assert_eq!(systems(), before_systems); + let c = rusqlite::Connection::open(&path).unwrap(); + assert_eq!(c.query_row("SELECT count(*) FROM owners", [], |r| r.get::<_, i64>(0)).unwrap(), 2); + drop(c); + assert_eq!(read("parcels", None).len(), 1, "untouched throughout"); +} + +/// An export into an existing GeoPackage never replaces what isn't a feature or attribute table +/// (a tile table, a view, an unregistered table) or a file that isn't there, and refuses a file +/// that isn't a GeoPackage; adding a table where there is no file makes a new one. +#[test] +fn geopackage_export_into_a_file_refuses_what_it_cannot_replace() { + use crate::io::{GpkgMode, GpkgTable, write_path_to}; + let workspace = tempfile::tempdir().unwrap(); + let path = workspace.path().join("county.gpkg"); + let replace = |name: &str| GpkgTable { mode: GpkgMode::ReplaceTable, name: Some(name.into()) }; + let mut ds = Dataset::new("permits", GeometryType::Point, vec![]); + ds.push(Some(Geometry::Point(Coord::new(1.0, 2.0))), vec![]); + // No file: nothing to replace in, but a table can be added to a new GeoPackage. + assert!(write_path_to(&ds, &path, Crs::LonLat, &replace("permits")).unwrap_err().to_string().contains("no GeoPackage there")); + assert!(!path.exists()); + let add = GpkgTable { mode: GpkgMode::AddTable, name: None }; + assert!(write_path_to(&ds, &path, Crs::LonLat, &add).unwrap().1.is_some_and(|t| !t.replaced)); + rusqlite::Connection::open(&path) + .unwrap() + .execute_batch( + "CREATE TABLE ortho (id INTEGER PRIMARY KEY, tile_data BLOB); + INSERT INTO gpkg_contents (table_name, data_type, identifier) VALUES ('ortho', 'tiles', 'ortho'); + CREATE VIEW recent AS SELECT * FROM permits; CREATE TABLE scratch (x);", + ) + .unwrap(); + let before = std::fs::read(&path).unwrap(); + for name in ["ortho", "recent", "scratch"] { + let e = write_path_to(&ds, &path, Crs::LonLat, &replace(name)).unwrap_err().to_string(); + assert!(e.contains("doesn't replace"), "{name}: {e}"); + assert_eq!(std::fs::read(&path).unwrap(), before, "{name}"); + } + // A related-tables relation's table isn't replaced; another application's database isn't + // written to. + rusqlite::Connection::open(&path) + .unwrap() + .execute_batch( + "CREATE TABLE gpkgext_relations (id INTEGER PRIMARY KEY, base_table_name TEXT, base_primary_column TEXT, + related_table_name TEXT, related_primary_column TEXT, relation_name TEXT, mapping_table_name TEXT); + INSERT INTO gpkgext_relations VALUES (1, 'permits', 'fid', 'owners', 'id', 'attributes', 'permits_owners');", + ) + .unwrap(); + let e = write_path_to(&ds, &path, Crs::LonLat, &replace("permits")).unwrap_err().to_string(); + assert!(e.contains("related-tables relation"), "{e}"); + // Nor is a table others refer to by foreign key: their rows would be deleted with it. + rusqlite::Connection::open(&path) + .unwrap() + .execute_batch( + "DELETE FROM gpkgext_relations; + CREATE TABLE inspections (id INTEGER PRIMARY KEY, permit INTEGER REFERENCES permits (fid) ON DELETE CASCADE); + INSERT INTO inspections VALUES (1, 1);", + ) + .unwrap(); + let e = write_path_to(&ds, &path, Crs::LonLat, &replace("permits")).unwrap_err().to_string(); + assert!(e.contains("inspections refer to permits"), "{e}"); + let kept: i64 = rusqlite::Connection::open(&path).unwrap().query_row("SELECT count(*) FROM inspections", [], |r| r.get(0)).unwrap(); + assert_eq!(kept, 1); + // A table referring to the GeoPackage's own records keeps its rows when a replace deletes them. + rusqlite::Connection::open(&path) + .unwrap() + .execute_batch( + "DROP TABLE inspections; + CREATE TABLE notes (layer TEXT REFERENCES gpkg_contents (table_name) ON DELETE CASCADE); + INSERT INTO notes VALUES ('permits');", + ) + .unwrap(); + write_path_to(&ds, &path, Crs::LonLat, &replace("permits")).unwrap(); + let notes: i64 = rusqlite::Connection::open(&path).unwrap().query_row("SELECT count(*) FROM notes", [], |r| r.get(0)).unwrap(); + assert_eq!(notes, 1); + // An ordinary table named as the spatial index is kept, and the replace refused. + let squatted = workspace.path().join("squatted.gpkg"); + write_path_to(&ds, &squatted, Crs::LonLat, &GpkgTable { mode: GpkgMode::AddTable, name: Some("sites".into()) }).unwrap(); + rusqlite::Connection::open(&squatted) + .unwrap() + .execute_batch( + "DROP TABLE rtree_sites_geom; CREATE TABLE rtree_sites_geom (note TEXT); INSERT INTO rtree_sites_geom VALUES ('keep');", + ) + .unwrap(); + let before = std::fs::read(&squatted).unwrap(); + let e = write_path_to(&ds, &squatted, Crs::LonLat, &replace("sites")).unwrap_err().to_string(); + assert!(e.contains("rtree_sites_geom"), "{e}"); + assert_eq!(std::fs::read(&squatted).unwrap(), before, "the failed replace changed nothing"); + let kept: String = + rusqlite::Connection::open(&squatted).unwrap().query_row("SELECT note FROM rtree_sites_geom", [], |r| r.get(0)).unwrap(); + assert_eq!(kept, "keep"); + // A spatial index declared over lines, with the module quoted, is still the index; an + // `rtree_i32` table of that name isn't. + for (module, replaced) in [("\n \"rtree\" (", true), ("[rtree](", true), ("rtree_i32(", false)] { + let file = workspace.path().join(format!("index-{}.gpkg", module.len())); + write_path_to(&ds, &file, Crs::LonLat, &GpkgTable { mode: GpkgMode::AddTable, name: Some("sites".into()) }).unwrap(); + rusqlite::Connection::open(&file) + .unwrap() + .execute_batch(&format!( + "DROP TABLE rtree_sites_geom; CREATE VIRTUAL TABLE rtree_sites_geom USING {module}id, minx, maxx, miny, maxy);" + )) + .unwrap(); + assert_eq!(write_path_to(&ds, &file, Crs::LonLat, &replace("sites")).is_ok(), replaced, "{module:?}"); + } + let other = workspace.path().join("other.gpkg"); + std::fs::copy(&path, &other).unwrap(); + rusqlite::Connection::open(&other).unwrap().execute_batch("PRAGMA application_id = 1297105496").unwrap(); + let add = GpkgTable { mode: GpkgMode::AddTable, name: Some("more".into()) }; + assert!(write_path_to(&ds, &other, Crs::LonLat, &add).unwrap_err().to_string().contains("application id")); + let plain = workspace.path().join("plain.gpkg"); + rusqlite::Connection::open(&plain).unwrap().execute_batch("CREATE TABLE t (x)").unwrap(); + let e = write_path_to(&ds, &plain, Crs::LonLat, &replace("permits")).unwrap_err().to_string(); + assert!(e.contains("not a GeoPackage"), "{e}"); + let text = workspace.path().join("text.gpkg"); + std::fs::write(&text, b"not a database").unwrap(); + assert!(write_path_to(&ds, &text, Crs::LonLat, &replace("permits")).is_err()); + assert_eq!(std::fs::read(&text).unwrap(), b"not a database"); +} diff --git a/crates/engine/src/cmd/exports.rs b/crates/engine/src/cmd/exports.rs index c584bef24..8ed6f7110 100644 --- a/crates/engine/src/cmd/exports.rs +++ b/crates/engine/src/cmd/exports.rs @@ -14,8 +14,14 @@ use crate::{CmdResult, Session}; pub struct Wrote { pub(crate) answer: Value, pub(crate) notes: Vec, + /// What it does to the session once written, after its notes, and adds to its answer (a + /// GeoPackage table replaced: the layers reading it read it again). + pub(crate) then: Option, } +/// What an export does to the session once its file is written, and adds to its answer. +pub(crate) type Then = Box; + /// The writing of an export, its inputs read: it needs no session. pub(crate) type Work = Box Result + Send>; @@ -57,10 +63,14 @@ fn written_under(name: &str, control: &std::sync::Arc Value { - for note in wrote.notes { + let Wrote { mut answer, notes, then } = wrote; + for note in notes { s.notify(note, false); } - wrote.answer + if let Some(then) = then { + then(s, &mut answer); + } + answer } #[cfg(not(target_arch = "wasm32"))] @@ -115,7 +125,7 @@ mod tests { // The work places its file, then Cancel comes before the run returns. let work: Work = Box::new(move || { late.cancel(); - Ok(Wrote { answer: serde_json::json!({"path": "x"}), notes: vec!["Exported x".into()] }) + Ok(Wrote { answer: serde_json::json!({"path": "x"}), notes: vec!["Exported x".into()], then: None }) }); let wrote = written_under("Export Raster", &control, work).unwrap(); assert_eq!(wrote.notes, ["Exported x"]); diff --git a/crates/engine/src/cmd/gpkg.rs b/crates/engine/src/cmd/gpkg.rs index 2cec20711..2d61af3cd 100644 --- a/crates/engine/src/cmd/gpkg.rs +++ b/crates/engine/src/cmd/gpkg.rs @@ -134,6 +134,52 @@ fn add_layer(s: &mut Session, p: &Value) -> CmdResult { } } +/// Makes every layer read in place from `table` of the GeoPackage at `path` (or from that file +/// with no table named) read it again, in every map: its rows were written, or (with `reopen`) +/// the table was made anew, so its fields and system are read again too. Returns how many read +/// it again, and the layers that couldn't reopen it with why (each also keeps its error). +pub(crate) fn reset_sources(s: &mut Session, path: &std::path::Path, table: &str, reopen: bool) -> (usize, Vec) { + let (mut n, mut failed) = (0, Vec::new()); + let canonical = std::fs::canonicalize(path).ok(); + for map in &mut s.project.maps { + for l in map.layers.iter_mut() { + if let Some(source) = &mut l.source + && source.table.as_deref().is_none_or(|t| t.eq_ignore_ascii_case(table)) + && source.path.as_deref().is_some_and(|p| match (canonical.as_ref(), std::fs::canonicalize(p).ok()) { + (Some(a), Some(b)) => *a == b, + _ => std::path::Path::new(p) == path, + }) + { + if !reopen { + source.reset(); + n += 1; + } else { + match source.reopen() { + Ok(()) => n += 1, + Err(e) => failed.push(format!("{} ({e})", l.name)), + } + } + } + } + } + (n, failed) +} + +/// The layers loaded for editing from `table` of the GeoPackage at `path`, by name, in every map: +/// after the table is replaced they still hold its old rows (with their unsaved edits). +pub(crate) fn editing(s: &Session, path: &std::path::Path, table: &str) -> Vec { + let canonical = std::fs::canonicalize(path).ok(); + let same = |p: &str| match (canonical.as_ref(), std::fs::canonicalize(p).ok()) { + (Some(a), Some(b)) => *a == b, + _ => std::path::Path::new(p) == path, + }; + let layers = s.project.maps.iter().flat_map(|m| m.layers.iter()); + layers + .filter(|l| l.gpkg.as_ref().is_some_and(|g| g.table.eq_ignore_ascii_case(table) && same(&g.path))) + .map(|l| l.name.clone()) + .collect() +} + fn save_edits(s: &mut Session, p: &Value) -> CmdResult { #[cfg(target_arch = "wasm32")] { @@ -244,20 +290,7 @@ fn save_edits(s: &mut Session, p: &Value) -> CmdResult { } s.saved_over.retain(rustgis_map::source::Versions::held_elsewhere); s.saved_over.push(edit.versions); - let canonical = std::fs::canonicalize(path).ok(); - for map in &mut s.project.maps { - for l in map.layers.iter_mut() { - if let Some(source) = &mut l.source - && source.table.as_deref().is_none_or(|t| t.eq_ignore_ascii_case(table)) - && source.path.as_deref().is_some_and(|p| match (canonical.as_ref(), std::fs::canonicalize(p).ok()) { - (Some(a), Some(b)) => *a == b, - _ => p == edit.path, - }) - { - source.reset(); - } - } - } + reset_sources(s, path, table, false); s.notify( format!( "{name}: saved to {table} in {}: {} new, {} changed, {} deleted", diff --git a/crates/engine/src/cmd/raster.rs b/crates/engine/src/cmd/raster.rs index 6ccee98bc..9223e2fae 100644 --- a/crates/engine/src/cmd/raster.rs +++ b/crates/engine/src/cmd/raster.rs @@ -959,7 +959,7 @@ fn write_geotiff(r: &Raster, bi: usize, cog: bool, path: &str) -> Result Vec { C::new("share.exportLayer", "Export Features", "Publish › Output", |s, p| super::exports::write_here(s, p, "share.exportLayer")) .view() .when(need_layer) - .params(r#"{"layer"?, "path": "*.geojson|*.csv|*.fgb|*.gpkg|*.parquet|*.shp|*.zip|*.kml|*.kmz|*.gpx", "crs"?: "EPSG:… (or WKT) for a GeoPackage, Shapefile or FlatGeobuf (default: the system the layer was read in, else EPSG:4326)"}"#), + .params(r#"{"layer"?, "path": "*.geojson|*.csv|*.fgb|*.gpkg|*.parquet|*.shp|*.zip|*.kml|*.kmz|*.gpx", "crs"?: "EPSG:… (or WKT) for a GeoPackage, Shapefile or FlatGeobuf (default: the system the layer was read in, else EPSG:4326)", "mode"?: "newFile (default: refuses an existing file) | addTable | replaceTable, for a GeoPackage", "tableName"?: "the GeoPackage table (default: the layer's name)"}"#), C::new("share.exportTable", "Export Table", "Publish › Output", |s, p| { let path = ps(p, "path").ok_or("path is required")?; + let ext = Path::new(path).extension().and_then(|e| e.to_str()).unwrap_or("").to_ascii_lowercase(); + let gpkg = gpkg_table(p, &ext)?; let t = s.project.tables.get(s.table_index(p)?).ok_or("no such table")?; // Tables may retain geometry and its import losses, just as feature layers do. let notes = rustgis_data::export_notes([(&t.dataset, t.dataset.extent())], path, Crs::LonLat); - let ext = Path::new(path).extension().and_then(|e| e.to_str()).unwrap_or("").to_ascii_lowercase(); let mut dbf_notes = Vec::new(); if matches!(ext.as_str(), "shp" | "zip") { dbf_notes.extend(rustgis_data::io::shapefile_names_note(&t.dataset.fields, path)); dbf_notes.extend(rustgis_data::io::shapefile_values_notes(&t.dataset, path)); } - let left_out = rustgis_data::io::write_path(&t.dataset, Path::new(path)).map_err(|e| e.to_string())?; + let (left_out, table) = + rustgis_data::io::write_path_to(&t.dataset, Path::new(path), Crs::LonLat, &gpkg).map_err(|e| e.to_string())?; if left_out > 0 { s.notify(format!("{path}: {left_out} features without drawable geometry were left out"), false); } for note in notes.into_iter().chain(dbf_notes) { s.notify(note, false); } - Ok(json!({"path": path})) + let mut out = json!({"path": path}); + report_table(s, path, &gpkg, table, &mut out); + Ok(out) }) .params( - r#"{"table": name|id (a standalone table), "path": "*.geojson|*.csv|*.fgb|*.gpkg|*.parquet|*.shp|*.zip|*.kml|*.kmz|*.gpx"}"#, + r#"{"table": name|id (a standalone table), "path": "*.geojson|*.csv|*.fgb|*.gpkg|*.parquet|*.shp|*.zip|*.kml|*.kmz|*.gpx", "mode"?: "newFile (default: refuses an existing file) | addTable | replaceTable, for a GeoPackage", "tableName"?: "the GeoPackage table (default: the table's name)"}"#, ) .view(), ] } +/// A GeoPackage export's table, from `mode` and `tableName`, which only a GeoPackage (extension +/// `ext`) takes; the front end checks a request with it before asking for a file. +pub fn gpkg_table(p: &Value, ext: &str) -> Result { + // Not a string is refused, not taken for the default. + let text = |key: &str| match p.get(key) { + None | Some(Value::Null) => Ok(None), + Some(v) => v.as_str().map(|s| Some(s.trim())).ok_or_else(|| format!("{key} must be a string")), + }; + let mode = match text("mode")? { + None | Some("newFile") => GpkgMode::NewFile, + Some("addTable") => GpkgMode::AddTable, + Some("replaceTable") => GpkgMode::ReplaceTable, + Some(other) => return Err(format!("unknown mode '{other}' (newFile, addTable or replaceTable)")), + }; + let name = text("tableName")?.filter(|n| !n.is_empty()).map(String::from); + if ext != "gpkg" && (mode != GpkgMode::NewFile || name.is_some()) { + return Err("mode and tableName are for a GeoPackage (.gpkg) export".into()); + } + Ok(GpkgTable { mode, name }) +} + +/// Puts the table a GeoPackage export wrote, and whether it replaced one, in its result `out`; +/// an export into an existing GeoPackage also says so, what went with a replaced table, and +/// which views and triggers elsewhere name it. Layers read in place from a replaced table read +/// it again. +fn report_table(s: &mut Session, path: &str, gpkg: &GpkgTable, table: Option, out: &mut Value) { + let Some(table) = table else { return }; + if gpkg.mode != GpkgMode::NewFile { + let (reread, failed) = + if table.replaced { super::gpkg::reset_sources(s, Path::new(path), &table.table, true) } else { (0, Vec::new()) }; + let note = if table.replaced { + format!( + "{path}: replaced its table {}, dropping the old table's indexes, triggers, column descriptions, metadata links and extension records", + table.table + ) + } else { + format!("{path}: added the table {}", table.table) + }; + s.notify(note, false); + if reread > 0 { + let layers = if reread == 1 { "layer reads" } else { "layers read" }; + s.notify(format!("{path}: {reread} {layers} {} in place, now the new table", table.table), false); + } + let held = if table.replaced { super::gpkg::editing(s, Path::new(path), &table.table) } else { Vec::new() }; + if !held.is_empty() { + let n = held.len(); + let (layers, show) = if n == 1 { ("layer", "shows") } else { ("layers", "show") }; + let note = format!( + "{path}: {n} {layers} loaded for editing from {} still {show} the old table's rows; Discard Edits loads the new one: {}", + table.table, + held.join(", ") + ); + s.notify(note, false); + } + if !failed.is_empty() { + let layers = if failed.len() == 1 { "layer" } else { "layers" }; + let note = format!( + "{path}: {} {layers} reading {} in place couldn't open the new table: {}", + failed.len(), + table.table, + failed.join(", ") + ); + s.notify(note, true); + } + if !table.dependents.is_empty() { + let n = table.dependents.len(); + s.notify( + format!( + "{path}: {n} view{} or trigger{} naming {} may no longer work if a column it uses is gone: {}", + if n == 1 { "" } else { "s" }, + if n == 1 { "" } else { "s" }, + table.table, + table.dependents.join(", ") + ), + false, + ); + } + } + out["table"] = json!(table.table); + out["replaced"] = json!(table.replaced); +} + /// Renders the active map at the current view (or a requested size) to PNG bytes. pub fn map_png(s: &mut Session, width: Option, height: Option, dpi: Option) -> Result, String> { let (vw, vh) = s.view_size; @@ -110,28 +197,37 @@ pub(crate) fn layer_work(s: &Session, p: &Value) -> Result { // export's thread. let l = s.map().layer(id).ok_or("no layer")?.clone(); let asked = ps(p, "crs").map(str::trim).filter(|c| !c.is_empty()).map(Crs::parse).transpose()?; - Ok(Box::new(move || write_layer(&l, &path, asked))) + // Checked now, so a bad table option is refused before the export starts. + let ext = Path::new(&path).extension().and_then(|e| e.to_str()).unwrap_or("").to_ascii_lowercase(); + let gpkg = gpkg_table(p, &ext)?; + Ok(Box::new(move || write_layer(&l, &path, asked, gpkg))) } -/// Writes layer `l` to `path`, in `asked` or the system the format defaults to. -fn write_layer(l: &rustgis_map::Layer, path: &str, asked: Option) -> Result { +/// Writes layer `l` to `path`, in `asked` or the system the format defaults to; a GeoPackage's +/// table goes where `gpkg` says. +fn write_layer(l: &rustgis_map::Layer, path: &str, asked: Option, gpkg: GpkgTable) -> Result { let ext = Path::new(&path).extension().and_then(|e| e.to_str()).unwrap_or("").to_ascii_lowercase(); + // A layer read in place from the GeoPackage it goes into is read whole first: SQLite can't + // write a file while the same export is reading it. + let from_target = gpkg.mode != GpkgMode::NewFile + && l.source.as_ref().and_then(|src| src.path.as_deref()).is_some_and(|src| same_file(Path::new(src), Path::new(&path))); // A streamed layer exports its whole source, not the features read for the view: a chunk at // a time where the source and format allow, so its size isn't limited by memory. - let chunks = if l.is_streamed() && rustgis_data::export::streams(&ext) { l.chunk_export()? } else { None }; + let chunks = if l.is_streamed() && rustgis_data::export::streams(&ext) && !from_target { l.chunk_export()? } else { None }; let (whole, unread) = if l.is_streamed() && chunks.is_none() { l.load_all_counted().map(|(ds, n)| (Some(ds), n))? } else { (None, 0) }; let ds = chunks.as_ref().map(|(_, layer)| layer).or(whole.as_ref()).unwrap_or(&l.dataset); // GeoJSON, KML and GPX are longitude/latitude; the others keep the layer's own system unless // told another. let crs = asked.unwrap_or_else(|| rustgis_data::io::default_export_crs(ds, &ext)); - let (written, left_out, skipped) = match &chunks { + let (written, left_out, skipped, table) = match &chunks { Some((src, layer)) => { - let e = rustgis_data::export::write_source_in(src.as_ref(), layer, Path::new(&path), crs).map_err(|e| e.to_string())?; - (e.written, e.left_out, e.skipped) + let (e, table) = + rustgis_data::export::write_source_to(src.as_ref(), layer, Path::new(&path), crs, &gpkg).map_err(|e| e.to_string())?; + (e.written, e.left_out, e.skipped, table) } None => { - let left_out = rustgis_data::io::write_path_in(ds, Path::new(&path), crs).map_err(|e| e.to_string())?; - (ds.features.len().saturating_sub(left_out), left_out, unread) + let (left_out, table) = rustgis_data::io::write_path_to(ds, Path::new(&path), crs, &gpkg).map_err(|e| e.to_string())?; + (ds.features.len().saturating_sub(left_out), left_out, unread, table) } }; // A streamed layer's extent is its whole source's. @@ -158,5 +254,16 @@ fn write_layer(l: &rustgis_map::Layer, path: &str, asked: Option) -> Result said.push(format!("{path}: no .prj written, as {} has no WKT form", crs.label())); } said.push(format!("Exported {path}")); - Ok(Wrote { answer: json!({"path": path, "features": written, "leftOut": left_out, "crs": crs.label()}), notes: said }) + let answer = json!({"path": path, "features": written, "leftOut": left_out, "crs": crs.label()}); + // The table written, said and put in the answer once the export lands, where the session + // (layers reading a replaced table) is at hand. + let path = path.to_string(); + let then: Option = table + .map(|t| Box::new(move |s: &mut Session, out: &mut Value| report_table(s, &path, &gpkg, Some(t), out)) as super::exports::Then); + Ok(Wrote { answer, notes: said, then }) +} + +/// Whether `a` and `b` are the same existing file. +fn same_file(a: &Path, b: &Path) -> bool { + matches!((std::fs::canonicalize(a), std::fs::canonicalize(b)), (Ok(a), Ok(b)) if a == b) } diff --git a/crates/engine/src/tests.rs b/crates/engine/src/tests.rs index 9202fb7e2..2e5cd3f71 100644 --- a/crates/engine/src/tests.rs +++ b/crates/engine/src/tests.rs @@ -13619,6 +13619,96 @@ fn exporting_features_refuses_to_replace_an_existing_geopackage() { std::fs::remove_dir_all(dir).unwrap(); } +/// Export Features adds a layer to an existing GeoPackage as one more table, or replaces one of +/// its tables, saying which; a layer read in place from that GeoPackage can go back into it (its +/// filtered copy as another table). `mode` and `tableName` are for GeoPackages only. +#[test] +fn exporting_features_adds_or_replaces_a_table_in_an_existing_geopackage() { + let dir = std::env::temp_dir().join(format!("rustgis-engine-gpkg-add-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("county.gpkg"); + let mut s = world(); + s.run("share.exportLayer", &json!({"layer": "Cities", "path": path})).unwrap(); + let mark = s.notice_mark(); + let out = s.run("share.exportLayer", &json!({"layer": "Countries", "path": path, "mode": "addTable"})).unwrap(); + assert_eq!((out["table"].as_str(), out["replaced"].as_bool()), (Some("Countries"), Some(false))); + let texts: Vec = s.notices_since(mark).iter().map(|n| n.text.clone()).collect(); + assert!(texts.iter().any(|t| t.ends_with("added the table Countries")), "{texts:?}"); + let tables = rustgis_data::source::layers(&path).unwrap(); + assert_eq!(tables, ["Cities", "Countries"]); + // Taken: refused, unless replacing. + let e = s.run("share.exportLayer", &json!({"layer": "Cities", "path": path, "mode": "addTable"})).unwrap_err(); + assert!(e.to_string().contains("already has a table Cities"), "{e}"); + let cities = s.map().layer_by_name("Cities").map(|l| l.dataset.features.len()).unwrap(); + let mark = s.notice_mark(); + let out = + s.run("share.exportLayer", &json!({"layer": "Countries", "path": path, "mode": "replaceTable", "tableName": "Cities"})).unwrap(); + assert_eq!(out["replaced"].as_bool(), Some(true)); + assert!(s.notices_since(mark).iter().any(|n| n.text.contains("replaced its table Cities, dropping"))); + let countries = s.map().layer_by_name("Countries").map(|l| l.dataset.features.len()).unwrap(); + let count = |t: &str| rustgis_data::source::open_path(&path, Some(t)).unwrap().info().count; + assert_eq!(count("Cities"), Some(countries as u64)); + assert_ne!(cities, countries); + // Read in place from the GeoPackage, then exported back into it. + s.run("layer.addData", &json!({"path": path.display().to_string(), "layer": "Countries", "stream": true, "name": "InPlace"})).unwrap(); + s.run("share.exportLayer", &json!({"layer": "InPlace", "path": path, "mode": "addTable", "tableName": "copy"})).unwrap(); + assert_eq!(count("copy"), Some(countries as u64)); + // Replacing the table a layer reads in place makes it read the new table, and says so. + let kind = + |s: &Session| s.map().layer_by_name("InPlace").and_then(|l| l.source.as_ref()).and_then(|src| src.info()).map(|i| i.geometry_type); + s.map_mut().layers.iter_mut().for_each(|l| { + l.sync_source(&rustgis_geom::Extent::new(-20.0, -20.0, 20.0, 20.0), true); + }); + assert_eq!(kind(&s), Some(rustgis_data::GeometryType::Polygon)); + let mark = s.notice_mark(); + s.run("share.exportLayer", &json!({"layer": "Cities", "path": path, "mode": "replaceTable", "tableName": "Countries"})).unwrap(); + assert_eq!(kind(&s), Some(rustgis_data::GeometryType::Point), "the new table's"); + let read = s.map().layer_by_name("InPlace").map(|l| l.load_all().map(|ds| ds.features.len())); + assert_eq!(read, Some(Ok(cities)), "its features are the new table's"); + assert!(s.notices_since(mark).iter().any(|n| n.text.ends_with("1 layer reads Countries in place, now the new table"))); + // A layer loaded for editing from a replaced table keeps its rows and edits, and is named. + s.run("gpkg.addLayer", &json!({"path": path, "table": "Cities", "name": "Editing"})).unwrap(); + let mark = s.notice_mark(); + s.run("share.exportLayer", &json!({"layer": "Countries", "path": path, "mode": "replaceTable", "tableName": "Cities"})).unwrap(); + assert!( + s.notices_since(mark) + .iter() + .any(|n| n.text.contains("1 layer loaded for editing from Cities still shows") && n.text.ends_with(": Editing")), + "{:?}", + s.notices_since(mark) + ); + // A standalone table goes in as an attribute table. + s.run("layer.addDataText", &json!({"fileName": "owners.csv", "text": "name,n\nShelby,1\n"})).unwrap(); + let table = s.project.tables.last().map(|t| t.id).unwrap(); + let out = s.run("share.exportTable", &json!({"table": table, "path": path, "mode": "addTable"})).unwrap(); + assert_eq!(out["table"].as_str(), Some("owners")); + let c = rusqlite::Connection::open(&path).unwrap(); + assert_eq!( + c.query_row("SELECT data_type FROM gpkg_contents WHERE table_name = 'owners'", [], |r| r.get::<_, String>(0)).unwrap(), + "attributes" + ); + drop(c); + // Other formats take neither; a mode not known is refused. + let geojson = dir.join("x.geojson"); + for p in [json!({"mode": "addTable"}), json!({"tableName": "x"})] { + let mut p = p; + p["path"] = json!(geojson); + assert!(s.run("share.exportLayer", &p).unwrap_err().to_string().contains("GeoPackage"), "{p}"); + } + assert!(s.run("share.exportLayer", &json!({"path": path, "mode": "append"})).is_err()); + // Not strings: refused, not taken for the default. + for p in [json!({"mode": 1}), json!({"mode": "addTable", "tableName": ["x"]})] { + let mut p = p; + p["path"] = json!(path); + assert!(s.run("share.exportLayer", &p).unwrap_err().to_string().contains("must be a string"), "{p}"); + } + assert!(!geojson.exists()); + // The layer read in place holds the file open, which Windows won't remove. + drop(s); + std::fs::remove_dir_all(dir).unwrap(); +} + /// A layer whose Z values were dropped on import says so when added, read whole or in place, and /// again when exported or converted; a 2D layer's export says nothing of it. #[test] diff --git a/crates/ui-egui/src/dialogs.rs b/crates/ui-egui/src/dialogs.rs index d7dae83cb..1cbe7ec14 100644 --- a/crates/ui-egui/src/dialogs.rs +++ b/crates/ui-egui/src/dialogs.rs @@ -41,6 +41,11 @@ pub struct Dialogs { /// Add From Database: load the PostgreSQL table whole, to be edited and saved back. pub edit: bool, pub mode: &'static str, + /// Export Features' GeoPackage destination: one of [`GPKG_MODES`]' codes (empty: a new file). + pub gpkg_mode: &'static str, + /// The layer Export Features was opened on (its id and instance), exported whatever is + /// selected while the dialog stays open. + pub export_layer: Option<(u32, u64)>, pub flag: bool, pub error: Option, pub info: Option, @@ -167,7 +172,13 @@ impl Dialogs { "renameLayer" => self.text = active.map(|l| l.name.clone()).unwrap_or_default(), // A file name can't hold a folder separator, which a project name may. "saveProject" => self.text = format!("{}.rgis", s.project.name.replace(['/', '\\'], "-")), - "exportLayer" => self.mode = "geojson", + "exportLayer" => { + self.mode = "geojson"; + self.gpkg_mode = "newFile"; + self.export_layer = active.map(|l| (l.id, l.instance_id())); + // The GeoPackage table's name. + self.text = active.map(|l| l.name.clone()).unwrap_or_default(); + } // The active layer when it is from PostGIS, else the first that is. "pgSave" | "pgDiscard" => { let pg = |l: &&rustgis_engine::map::Layer| l.postgis.is_some(); @@ -583,6 +594,10 @@ pub const EXPORT_FORMATS: &[(&str, &str)] = &[ ("GPX", "gpx"), ]; +/// Where Export Features puts a GeoPackage's table: (`share.exportLayer` mode, label shown). +pub const GPKG_MODES: &[(&str, &str)] = + &[("newFile", "A new GeoPackage"), ("addTable", "A new table in a GeoPackage"), ("replaceTable", "A table replaced in a GeoPackage")]; + /// The formats Export Features offers: all of [`EXPORT_FORMATS`] on the desktop. The web app /// leaves out GeoPackage, written with SQLite, and FlatGeobuf, whose writer needs a temporary /// file; the browser has neither. @@ -611,12 +626,46 @@ fn export_layer(app: &mut RustGisApp, ui: &mut egui::Ui) -> bool { .response .labelled_by(input_label.id); ui.end_row(); + // A GeoPackage can take the layer as one more table (desktop only: the browser offers none). + if app.dialogs.mode == "gpkg" && !web { + let into_label = ui.label(tl!("Into")); + let shown = GPKG_MODES.iter().find(|(m, _)| *m == app.dialogs.gpkg_mode).map_or(GPKG_MODES[0].1, |(_, l)| *l); + egui::ComboBox::from_id_salt("export_gpkg_mode") + .selected_text(tl!(shown)) + .width(220.0) + .show_ui(ui, |ui| { + for (mode, label) in GPKG_MODES { + ui.selectable_value(&mut app.dialogs.gpkg_mode, *mode, tl!(*label)); + } + }) + .response + .labelled_by(into_label.id); + ui.end_row(); + if !matches!(app.dialogs.gpkg_mode, "" | "newFile") { + let table_label = ui.label(tl!("Table")); + ui.add(egui::TextEdit::singleline(&mut app.dialogs.text).desired_width(220.0)).labelled_by(table_label.id); + ui.end_row(); + } + } }); if app.dialogs.mode == "shp" && web { ui.label(egui::RichText::new(tl!("The Shapefile's files download as one zip.")).small().weak()); } + let into_existing = app.dialogs.mode == "gpkg" && !web && !matches!(app.dialogs.gpkg_mode, "" | "newFile"); + if into_existing { + ui.label(egui::RichText::new(tl!("Pick the GeoPackage next. Its other tables are kept.")).small().weak()); + } buttons(app, ui, if web { "Download" } else { "Export…" }, |app| { - app.run("ui.exportLayer", json!({"format": app.dialogs.mode})).map(|_| ()) + let mut p = json!({"format": app.dialogs.mode}); + if let Some((layer, instance)) = app.dialogs.export_layer { + p["layer"] = json!(layer); + p["instance"] = json!(instance); + } + if into_existing { + p["mode"] = json!(app.dialogs.gpkg_mode); + p["tableName"] = json!(app.dialogs.text); + } + app.run("ui.exportLayer", p).map(|_| ()) }) } diff --git a/crates/ui-egui/src/lib.rs b/crates/ui-egui/src/lib.rs index 234ec4432..219d19cac 100644 --- a/crates/ui-egui/src/lib.rs +++ b/crates/ui-egui/src/lib.rs @@ -997,6 +997,20 @@ pub struct RustGisApp { } impl RustGisApp { + /// The layer to export: `layer` (an id or name, as commands take it; the dialog's is checked + /// against its `instance`, so a reused id isn't taken for it), else the active layer. + fn export_target(&self, p: &Value) -> Result { + if p.get("layer").is_none_or(Value::is_null) { + return self.session.active_layer.ok_or_else(|| "select a layer in the Layers pane first".to_string()); + } + let id = self.session.layer_id(p)?; + let l = self.session.map().layer(id).ok_or("the layer Export Features was opened on is gone")?; + match p.get("instance").and_then(Value::as_u64) { + Some(instance) if instance != l.instance_id() => Err("the layer Export Features was opened on is gone".into()), + _ => Ok(id), + } + } + /// The Ctrl (⌘) shortcuts the keymap takes, as (Alt, Shift, key), the clipboard's left out: /// in a browser, what the web app keeps from the browser's own shortcuts (⌘D bookmarking /// the page, ⌘Y opening its history, ⌘= zooming it), as design tools in the browser do. @@ -3374,14 +3388,12 @@ impl RustGisApp { self.dialogs.open("exportLayer", &self.session)?; } "ui.exportLayer" if self.services.download.is_some() => { - let l = self - .session - .active_layer - .and_then(|id| self.session.map().layer(id)) - .ok_or("select a layer in the Layers pane first")?; + let id = self.export_target(p)?; + let l = self.session.map().layer(id).ok_or("select a layer in the Layers pane first")?; // Any format the dialog offers here (`dialogs::export_formats`): not GeoPackage or // FlatGeobuf, which the data crate refuses in the browser. let ext = p.get("format").and_then(Value::as_str).unwrap_or("geojson").to_ascii_lowercase(); + rustgis_engine::cmd::share::gpkg_table(p, &ext)?; // A streamed layer exports its whole source, not the features read for the view. let whole = if l.is_streamed() { Some(l.load_all()?) } else { None }; // In the layer's own coordinate system, as share.exportLayer writes a file. @@ -3456,22 +3468,40 @@ impl RustGisApp { .find(|(_, e)| *e == ext) .map(|(l, _)| *l) .ok_or_else(|| format!("unknown format '{ext}'"))?; - let name = self - .session - .active_layer - .and_then(|id| self.session.map().layer(id)) - .map(|l| format!("{}.{ext}", l.name)) - .unwrap_or_default(); + // Into a GeoPackage that exists: picked to open, as it is kept, not replaced. + // Checked as the command checks them, before a file is asked for; then passed on. + let gpkg = rustgis_engine::cmd::share::gpkg_table(p, &ext)?; + let id = self.export_target(p)?; + let with_table = |mut args: Value| { + for key in ["mode", "tableName"] { + if let Some(v) = p.get(key) { + args[key] = v.clone(); + } + } + args + }; + if gpkg.mode != rustgis_engine::data::io::GpkgMode::NewFile { + if let Some(path) = dialogs::pick_open(&["gpkg"], label) { + let args = with_table(json!({"layer": id, "path": path})); + #[cfg(not(target_arch = "wasm32"))] + return exportjob::start(self, "share.exportLayer", args); + #[cfg(target_arch = "wasm32")] + return self.run("share.exportLayer", args); + } + return Ok(Value::Null); + } + let name = self.session.map().layer(id).map(|l| format!("{}.{ext}", l.name)).unwrap_or_default(); if let Some(path) = dialogs::pick_save(&name, &[ext.as_str()], label) { // The format chosen decides: its extension is added unless it was typed (a dot // in the name, as in "U.S. Cities", isn't taken for one). let typed = std::path::Path::new(&path).extension().is_some_and(|e| e.eq_ignore_ascii_case(&ext)); let path = if typed { path } else { format!("{path}.{ext}") }; // Written on a thread of its own, with progress and Cancel in the status bar. + let args = with_table(json!({"layer": id, "path": path})); #[cfg(not(target_arch = "wasm32"))] - return exportjob::start(self, "share.exportLayer", json!({"path": path})); + return exportjob::start(self, "share.exportLayer", args); #[cfg(target_arch = "wasm32")] - return self.run("share.exportLayer", json!({"path": path})); + return self.run("share.exportLayer", args); } } // Add Zarr Variable, connected to a store (a link, a script). @@ -4627,6 +4657,12 @@ mod download_service_tests { app }; let mut app = with(true); + // The browser's export refuses a GeoPackage mode for another format, as the desktop's does. + assert!(app.run("ui.exportLayer", json!({"format": "geojson", "mode": "replaceTable"})).is_err()); + assert_eq!(changed(&app), 0); + // As the command reads them: no table name given is none. + app.run("ui.exportLayer", json!({"format": "geojson", "mode": " newFile ", "tableName": ""})).unwrap(); + assert_eq!(changed(&app), 0, "GeoJSON keeps the names"); app.run("ui.exportLayer", json!({"format": "shp"})).unwrap(); assert_eq!(changed(&app), 1); assert!(app.session.notices.iter().any(|n| n.text == "Sites.zip: 1 text value was cut to at most 254 bytes, in 'notes'")); @@ -4654,6 +4690,44 @@ mod download_service_tests { assert!(!app.session.notices.iter().any(|n| n.text.contains("254 bytes") || n.text.contains("left empty"))); } + /// Export Features into an existing GeoPackage: the file is picked to open, not to save over, + /// and the layer goes in as the table named in the dialog, beside the file's others. + #[test] + fn export_features_adds_a_table_to_a_picked_geopackage() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("county.gpkg"); + let mut app = RustGisApp::with_sample(); + app.run("share.exportLayer", json!({"layer": "Cities", "path": path})).unwrap(); + app.session.active_layer = app.session.map().layer_by_name("Countries").map(|l| l.id); + app.run("ui.dialog", json!({"name": "exportLayer"})).unwrap(); + let (id, instance) = app.dialogs.export_layer.unwrap(); + // Another layer selected while the dialog is open: the dialog's is the one exported. + app.session.active_layer = app.session.map().layer_by_name("Cities").map(|l| l.id); + crate::dialogs::PICKED.with(|p| *p.borrow_mut() = Some(path.display().to_string())); + let p = json!({"format": "gpkg", "mode": "addTable", "tableName": "borders", "layer": id, "instance": instance}); + app.run("ui.exportLayer", p).unwrap(); + // Written on a thread of its own: waits for it to land. + let end = std::time::Instant::now() + std::time::Duration::from_secs(30); + while crate::exportjob::land(&mut app) { + assert!(std::time::Instant::now() < end, "the export never landed"); + std::thread::sleep(std::time::Duration::from_millis(10)); + } + assert_eq!(rustgis_engine::data::source::layers(&path).unwrap(), ["Cities", "borders"]); + let countries = app.session.map().layer_by_name("Countries").map(|l| l.dataset.features.len() as u64); + assert_eq!(rustgis_engine::data::source::open_path(&path, Some("borders")).unwrap().info().count, countries); + // An unknown mode is refused before a file is asked for (the pick isn't taken). + crate::dialogs::PICKED.with(|p| *p.borrow_mut() = Some(path.display().to_string())); + assert!(app.run("ui.exportLayer", json!({"format": "gpkg", "mode": "append"})).unwrap_err().contains("unknown mode")); + assert!(crate::dialogs::PICKED.with(|p| p.borrow_mut().take()).is_some()); + // A layer by name is that layer; one not there is refused, not taken for the active one. + let e = app.run("ui.exportLayer", json!({"format": "gpkg", "mode": "addTable", "layer": "Nowhere"})).unwrap_err(); + assert!(e.contains("Nowhere"), "{e}"); + // Cancelling the pick writes nothing. + let before = std::fs::read(&path).unwrap(); + app.run("ui.exportLayer", json!({"format": "gpkg", "mode": "replaceTable", "tableName": "Cities"})).unwrap(); + assert_eq!(std::fs::read(&path).unwrap(), before); + } + #[test] fn direct_file_result_reports_delivery_instead_of_pending() { let pending = json!({"pending": true, "download": 7, "file": "cities.pmtiles"}); diff --git a/docs/python.md b/docs/python.md index 7d8e96725..bc43cf8ee 100644 --- a/docs/python.md +++ b/docs/python.md @@ -271,8 +271,14 @@ s["Cities"].to_geojson() # a FeatureCollection dict big.to_file("big-cities.gpkg", crs="EPSG:3857") # GeoJSON, CSV, FlatGeobuf, GeoParquet, … ``` -GeoPackage exports through `Layer.to_file` or `share.exportLayer` require a new filename. -An existing `.gpkg` is refused to preserve its tables; export does not append or replace tables. +A GeoPackage export through `Layer.to_file` or `share.exportLayer` writes a new file and refuses +an existing `.gpkg`, unless told to add a table to it or replace one of its tables; its other +tables are kept: + +```python +roads.to_file("big-cities.gpkg", mode="addTable", table_name="roads_2026") +big.to_file("big-cities.gpkg", mode="replaceTable") # its table named for the layer, "Big cities" +``` `add_geojson` also takes any object with `__geo_interface__` (a Shapely geometry, a GeoSeries). `add_data`, `add_geojson` and `add_geopandas` return a `Layer` for a single layer, a list for diff --git a/docs/user-guide/data-formats.md b/docs/user-guide/data-formats.md index 5cdb7f0b6..20282d7f3 100644 --- a/docs/user-guide/data-formats.md +++ b/docs/user-guide/data-formats.md @@ -512,8 +512,8 @@ were rounded, and Export Raster, which writes the rounded values, says so again. | Format | How to write it | Notes | | --- | --- | --- | -| GeoJSON (`.geojson`, `.json`), CSV | Data › Export (`share.exportLayer`) | The Export Features dialog picks the format; the chosen extension is appended when needed. In the web app it downloads any vector format below except GeoPackage and FlatGeobuf (a Shapefile as one zip). A streamed layer exports its whole source: on the desktop, to GeoJSON, CSV, FlatGeobuf, GeoPackage or GeoParquet a chunk at a time with no limit on its size (a GeoParquet source's row groups are read whole, so on the desktop each holds at most 16,777,216 rows), otherwise loaded whole (see [Streamed layers](data.md)); remote sources on the web must be downloaded first. | -| GeoPackage | Data › Export to `.gpkg` (desktop) | A new file with one feature table (an attributes table for a layer without geometry) in the layer's coordinate system (see below), with an R-tree and GeoPackage 1.4 triggers to maintain it. An existing destination is refused; adding or replacing tables in an existing GeoPackage is not supported. Unique feature ids that fit signed 64-bit integers are kept; otherwise SQLite assigns new ids. | +| GeoJSON (`.geojson`, `.json`), CSV | Data › Export (`share.exportLayer`) | The Export Features dialog picks the format; the chosen extension is appended when needed. In the web app it downloads any vector format below except GeoPackage and FlatGeobuf (a Shapefile as one zip). A streamed layer exports its whole source: on the desktop, to GeoJSON, CSV, FlatGeobuf, GeoPackage or GeoParquet a chunk at a time with no limit on its size (a GeoParquet source's row groups are read whole, so on the desktop each holds at most 16,777,216 rows; a layer read in place from the GeoPackage it goes into is read whole first), otherwise loaded whole (see [Streamed layers](data.md)); remote sources on the web must be downloaded first. | +| GeoPackage | Data › Export to `.gpkg` (desktop) | A new file with one feature table (an attributes table for a layer without geometry) in the layer's coordinate system (see below), with an R-tree and GeoPackage 1.4 triggers to maintain it. An existing destination is refused unless the export adds a table to it or replaces one of its feature or attribute tables (`mode`, see [Exporting](sharing.md)), leaving its other tables as they were. Unique feature ids that fit signed 64-bit integers are kept; otherwise SQLite assigns new ids. | | GeoParquet (`.parquet`, `.geoparquet`) | Data › Export | GeoParquet 1.1: WKB geometry in WGS 84, Zstd level 9 compression, row groups of up to 65,536 features and a `bbox` covering column with byte-stream-split encoding. The covering keeps spatial range reads available; compression favours smaller files over export speed. A standalone table without geometry writes plain Parquet. | | Shapefile | Data › Export to `.shp` (or `.zip` for the files zipped) | Writes `.shp`, `.shx`, `.dbf`, `.prj` (the coordinate system, see below) and `.cpg` (UTF-8). Field names are cut to 10 bytes (made unique ignoring case, and reported), text to at most 254 bytes, and numbers that do not fit their dBase field are left empty. Export notices count shortened text, non-null values left empty (including incompatible types), and numbers rounded to the field's precision, and name the affected source fields; features without geometry, and lines of a single point, are left out (and reported). | | FlatGeobuf | Data › Export to `.fgb` | In the layer's coordinate system (see below), with a spatial index; features without geometry are left out (and reported). Desktop only: the writer needs a temporary file, which the browser doesn't have. | diff --git a/docs/user-guide/sharing.md b/docs/user-guide/sharing.md index c9f928f97..efdffa363 100644 --- a/docs/user-guide/sharing.md +++ b/docs/user-guide/sharing.md @@ -26,9 +26,32 @@ as before. One export runs at a time. A cancelled export writes nothing, and a f replace stays as it was. Scripts, models, the AI Assistant and the command line still wait for the file. Export Map and layout exports still run before the window goes on. -GeoPackage exports require a new filename. Exporting to an existing `.gpkg` is refused to -protect its tables and data, even if the save dialog offers to replace the file. Export does -not append a table to an existing GeoPackage or replace one of its tables. +A GeoPackage export writes a new file by default, and refuses an existing `.gpkg` to protect its +tables, even if the save dialog offers to replace the file. To put the layer into an existing +GeoPackage instead, choose GeoPackage in **Export Features**, then under **Into** either +**A new table in a GeoPackage** or **A table replaced in a GeoPackage**, name the table, and +pick the file (on the desktop). The GeoPackage's other tables stay as they were: one table is +added, or one feature or attribute table of that name is replaced (a feature table with its +spatial index; an attribute table has none), in a single transaction, so a failed export changes +nothing. A name already used is refused when +adding (with no file there, adding makes a new GeoPackage). Replacing needs a feature or +attribute table of that name, so a misspelt name changes nothing; tiles, views and other tables +are never replaced, nor is a table in a related-tables relation. A replaced table's own indexes, +triggers and column descriptions go with it, and the export names any views or triggers elsewhere +that use it. The GeoPackage's own triggers don't run while the export writes. A layer read in +place from the GeoPackage it is exported into is read whole first, rather than a chunk at a +time, as SQLite can't read and write one file in the same export. +From the command line, `share.exportLayer` and `share.exportTable` take `"mode": "addTable"` or +`"replaceTable"` and an optional `"tableName"` (default: the layer's or table's name). The name is +written as a GeoPackage allows: control characters are removed, surrounding spaces trimmed, an +empty name becomes `layer`, and one starting with `gpkg_`, `sqlite_` or `rtree_` (reserved for the +GeoPackage's own tables) gets `layer_` in front. Adding and replacing look for that name, and the +command's result gives it as `table`: + +```sh +rustgis-cli run --data permits.geojson \ + --cmd 'share.exportLayer={"layer":"permits","path":"county.gpkg","mode":"addTable"}' +``` Packages and project item files are described under [Project items](data.md#project-items).