From 48269aa1288397d036f820b5aa3cd6aa3b5f2be9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Signe=20Kirk=20Br=C3=B8db=C3=A6k?= Date: Wed, 12 Aug 2026 16:07:22 +0200 Subject: [PATCH 1/4] =?UTF-8?q?feat:=20=E2=9C=A8=20post=20on=20publishing?= =?UTF-8?q?=20fastreg?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- posts/publish-fastreg/index.qmd | 88 +++++++++++++++++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 posts/publish-fastreg/index.qmd diff --git a/posts/publish-fastreg/index.qmd b/posts/publish-fastreg/index.qmd new file mode 100644 index 0000000..9ff9add --- /dev/null +++ b/posts/publish-fastreg/index.qmd @@ -0,0 +1,88 @@ +--- +title: "Published our R package fastreg" +description: | + Continuing and formalising the prior work by Luke, we've built an R package + to standardise and parallelise the conversion of large Danish register SAS + files to Parquet. And finally, we've published it to CRAN! +author: + - Signe Kirk Brødbæk +date: "2026-08-12" +categories: + - packaging + - publishing + - programming +--- + +Earlier this year, we published our second R package to +[CRAN](https://CRAN.R-project.org/package=fastreg). The package, +[fastreg](https://dp-next.github.io/fastreg/), continues Luke's earlier work, +previously known as "DataPrep", to convert large Danish register SAS files to +the [Apache Parquet](https://parquet.apache.org/) format. + +The package's purpose, as described on the package +[website](https://dp-next.github.io/fastreg/), is to simplify the process of +converting the large Danish registers into the more modern +[Parquet](https://parquet.apache.org/) format as well as to simplify reading +these Parquet files in R. By converting data from SAS to the more modern and +efficient Parquet format, the package reduces storage costs and aims to improve +performance in data analysis workflows. + +::: callout-note +Luke gave a presentation on fastreg at Steno Diabetes Center Aarhus and Aarhus +University in June 2026. The slides for that presentation can be found +[here](https://slides.lwjohnst.com/s/2026-06-23/#/title-slide). + +Luke also gave a similar presentation on [using Parquet data format in Denmark +Statistics](/posts/parquet-dst-2025/index.qmd) last year. +::: + +## Who is it for and why use it? + +The main reason for building the fastreg package was to provide a tool for data +managers and researchers who work with Danish register data to convert the +increasingly larger SAS files provided by Statistics Denmark (DST) to Parquet. + +This conversion has multiple benefits: + +1. Parquet is smaller on disk. Parquet's efficient compression significantly + reduces disk space, especially for large datasets. This is especially + beneficial since DST charges for storage space. +2. Parquet is faster to read. The columnar layout speeds up analytical queries + that only need a subset of columns. In addition, fastreg converts the + registers into year-based Hive-partitioned datasets and provides + [functionality](#functionality) to more easily read a register in R. +3. Parquet works well with modern tools like DuckDB and Arrow in R + ([DuckDB](https://r.duckdb.org/) and + [Arrow](https://arrow.apache.org/docs/r/)) and Python ( + [DuckDB](https://duckdb.org/docs/current/clients/python/overview) and + [pyarrow](https://arrow.apache.org/docs/python/parquet.html)). + +One drawback of this conversion that we've become aware of is that researchers +working in, e.g., Stata can't load the Parquet files on the Statistics Denmark +servers. + +## Functionality + +fastreg provides functionality to: + +- convert a single SAS register file to Parquet +- convert multiple SAS registers in parallel with a targets pipeline +- read converted registers + +You can find an overview of the functions in the [reference +documentation](https://dp-next.github.io/fastreg/reference/index.html) and a +guide on how to [Get +started](https://dp-next.github.io/fastreg/articles/fastreg.html) on the fastreg +website. + +## Next steps + +The next step is to train the data managers at Steno Diabetes Center Aarhus how +to convert their register SAS files using the targets template---e.g., in their +[Register Project +Database](https://steno-aarhus.github.io/registers-project-database/) and the +[DARTER](https://steno-aarhus.github.io/darter-project/) project. + +After the conversion, we've discussed using the converted Parquet files with +[osdc](/posts/published-osdc/index.qmd) to create a diabetes population in the +Register Project Database. From 4ce7c4f5cd60fecb63a8105d802bf9ab03352fd6 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Wed, 12 Aug 2026 14:08:06 +0000 Subject: [PATCH 2/4] =?UTF-8?q?chore:=20=E2=9C=8F=EF=B8=8F=20automatic=20p?= =?UTF-8?q?re-commit=20hook=20fixes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- learn/books/philosophy-software-design.qmd | 8 +++----- learn/shorts/intro-gh-actions.qmd | 16 +++++++--------- learn/shorts/python-packaging-basics.qmd | 14 ++++++-------- 3 files changed, 16 insertions(+), 22 deletions(-) diff --git a/learn/books/philosophy-software-design.qmd b/learn/books/philosophy-software-design.qmd index 92e3dda..60ea9a9 100644 --- a/learn/books/philosophy-software-design.qmd +++ b/learn/books/philosophy-software-design.qmd @@ -308,13 +308,11 @@ A module can also have different "depths", with a spectrum between "deep" and functionality. - Shallow: These have less extensive functionality along with a less than simple - interface. Their cost-to-benefit ratio is higher. - ::: callout-caution - A shallow module is, in general, a red flag. More (user-exposed) functions and + interface. Their cost-to-benefit ratio is higher. ::: callout-caution A + shallow module is, in general, a red flag. More (user-exposed) functions and classes in a module can increase cognitive load on the user or developer. Modules should provide the functionality necessary for the needs, but still be -as simple as possible. - ::: + as simple as possible. ::: A deep module hides information. Information leaking is when a design decision is reflected in multiple modules. This creates a dependency between the modules. diff --git a/learn/shorts/intro-gh-actions.qmd b/learn/shorts/intro-gh-actions.qmd index bfaa89e..c81fbe7 100644 --- a/learn/shorts/intro-gh-actions.qmd +++ b/learn/shorts/intro-gh-actions.qmd @@ -198,17 +198,15 @@ parts: - The `run` section, which is the actual code to run. If you use the `run` section, you don't need to use the `uses` section. The default code that you can use in the `run` section is - [Bash](https://en.wikipedia.org/wiki/Bash_(Unix_shell)) code. - ::: callout-tip + [Bash](https://en.wikipedia.org/wiki/Bash_(Unix_shell)) code. ::: callout-tip This `GITHUB_TOKEN` is a special token that GitHub provides to workflows that allows them to interact with the GitHub API. It is automatically created and - available to all workflows, but you need to provide it to the workflow in order - to use it. You give it to the workflow by using `${{ secrets.GITHUB_TOKEN }}` in - the `env` section. Here, the `${{ }}` is a way to tell GitHub to "inject" the - value of the `secrets` object, which also contains the `GITHUB_TOKEN` into the - workflow. The `.` is what tells GitHub to look inside the `secrets` object for -the `GITHUB_TOKEN` value. - ::: + available to all workflows, but you need to provide it to the workflow in + order to use it. You give it to the workflow by using + `${{ secrets.GITHUB_TOKEN }}` in the `env` section. Here, the `${{ }}` is a + way to tell GitHub to "inject" the value of the `secrets` object, which also + contains the `GITHUB_TOKEN` into the workflow. The `.` is what tells GitHub to + look inside the `secrets` object for the `GITHUB_TOKEN` value. ::: An example of a job (called `checkout-repository`) with one step (to checkout a repository by cloning it), could look like: diff --git a/learn/shorts/python-packaging-basics.qmd b/learn/shorts/python-packaging-basics.qmd index 39ef319..6ed078a 100644 --- a/learn/shorts/python-packaging-basics.qmd +++ b/learn/shorts/python-packaging-basics.qmd @@ -88,14 +88,12 @@ the: is empty, all code is included. - `pyproject.toml` file in the parent folder. This file contains the necessary metadata for Python to properly install the Python package on your computer, - metadata like version number, title, description, and dependencies. - ::: callout-tip - The file `pyproject.toml` is the current way of declaring package metadata (see - [PEP 621](https://peps.python.org/pep-0621/)). Before you needed to use the - [`setuptools`](https://setuptools.pypa.io/en/latest/index.html) package and use - a file called `setup.py` instead (or a similar name). So you may see this file -used in other, older packages. - ::: + metadata like version number, title, description, and dependencies. ::: + callout-tip The file `pyproject.toml` is the current way of declaring package + metadata (see [PEP 621](https://peps.python.org/pep-0621/)). Before you needed + to use the [`setuptools`](https://setuptools.pypa.io/en/latest/index.html) + package and use a file called `setup.py` instead (or a similar name). So you + may see this file used in other, older packages. ::: You might notice the duplicate `packagename/` folder names being used, one for the parent folder and the other as a sub-folder. While technically only the From 9ba5beaba5f1ae2f3fe765a9e8da0f7ba9527b30 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Signe=20Kirk=20Br=C3=B8db=C3=A6k?= <40836345+signekb@users.noreply.github.com> Date: Mon, 17 Aug 2026 11:43:05 +0200 Subject: [PATCH 3/4] refactor: :pencil2: apply suggestions from review Co-authored-by: Luke W. Johnston --- posts/publish-fastreg/index.qmd | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/posts/publish-fastreg/index.qmd b/posts/publish-fastreg/index.qmd index 9ff9add..2ef3386 100644 --- a/posts/publish-fastreg/index.qmd +++ b/posts/publish-fastreg/index.qmd @@ -1,7 +1,7 @@ --- title: "Published our R package fastreg" description: | - Continuing and formalising the prior work by Luke, we've built an R package + Continuing and formalising prior work on DST, we've built an R package to standardise and parallelise the conversion of large Danish register SAS files to Parquet. And finally, we've published it to CRAN! author: @@ -16,7 +16,7 @@ categories: Earlier this year, we published our second R package to [CRAN](https://CRAN.R-project.org/package=fastreg). The package, [fastreg](https://dp-next.github.io/fastreg/), continues Luke's earlier work, -previously known as "DataPrep", to convert large Danish register SAS files to +previously known as "dstDataPrep", to convert large Danish register SAS files to the [Apache Parquet](https://parquet.apache.org/) format. The package's purpose, as described on the package @@ -48,7 +48,7 @@ This conversion has multiple benefits: reduces disk space, especially for large datasets. This is especially beneficial since DST charges for storage space. 2. Parquet is faster to read. The columnar layout speeds up analytical queries - that only need a subset of columns. In addition, fastreg converts the + that only need a subset of columns. With fastreg, it also converts the registers into year-based Hive-partitioned datasets and provides [functionality](#functionality) to more easily read a register in R. 3. Parquet works well with modern tools like DuckDB and Arrow in R @@ -57,7 +57,7 @@ This conversion has multiple benefits: [DuckDB](https://duckdb.org/docs/current/clients/python/overview) and [pyarrow](https://arrow.apache.org/docs/python/parquet.html)). -One drawback of this conversion that we've become aware of is that researchers +One drawback of this conversion we've become aware of is that researchers working in, e.g., Stata can't load the Parquet files on the Statistics Denmark servers. @@ -78,7 +78,7 @@ website. ## Next steps The next step is to train the data managers at Steno Diabetes Center Aarhus how -to convert their register SAS files using the targets template---e.g., in their +to convert their register SAS files using the [targets](https://books.ropensci.org/targets/) template---e.g., in their [Register Project Database](https://steno-aarhus.github.io/registers-project-database/) and the [DARTER](https://steno-aarhus.github.io/darter-project/) project. From 5c4969d36db309b0806a177920b1af860babab09 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 17 Aug 2026 09:43:12 +0000 Subject: [PATCH 4/4] =?UTF-8?q?chore:=20=E2=9C=8F=EF=B8=8F=20automatic=20p?= =?UTF-8?q?re-commit=20hook=20fixes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- posts/publish-fastreg/index.qmd | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/posts/publish-fastreg/index.qmd b/posts/publish-fastreg/index.qmd index 2ef3386..7746f14 100644 --- a/posts/publish-fastreg/index.qmd +++ b/posts/publish-fastreg/index.qmd @@ -78,7 +78,8 @@ website. ## Next steps The next step is to train the data managers at Steno Diabetes Center Aarhus how -to convert their register SAS files using the [targets](https://books.ropensci.org/targets/) template---e.g., in their +to convert their register SAS files using the +[targets](https://books.ropensci.org/targets/) template---e.g., in their [Register Project Database](https://steno-aarhus.github.io/registers-project-database/) and the [DARTER](https://steno-aarhus.github.io/darter-project/) project.