Update programmatic workflows for the Project data model

docs/howto/manage/program/api.md

@@ -2,6 +2,10 @@
 
 This guide is about using NOMAD's REST APIs directly, e.g. via Python's `requests` library.
 
+The GUI organizes files and entries in projects. The API represents each project
+as an upload resource, so endpoint paths, request fields, and response fields retain
+terms such as `uploads` and `upload_id`.
+
 To access the processed data with our client library `nomad-lab` follow
 [How to access processed data](archive_query.md). You can also watch our
 [video tutorial on the API](../../../tutorial/access_api.md).
@@ -127,7 +131,7 @@ This will give you something like this:
 }
 ```
 
-The `entry_id` is a unique identifier for, well, entries. You can use it to access other
+The `entry_id` is a unique identifier for entries. You can use it to access other
 entry data. For example, you want to access the entry's archive. More precisely, you want
 to gather the formula and energies from the main workflow result. The following requests
 the archive based on the `entry_id` and only requires some archive sections.
@@ -269,7 +273,7 @@ with different functions in the API:
 
 - Entries
 - Uploads
-- Datasets
+- Datasets (retained backend resources for existing collections)
 - Users
 
 The API URLs typically start with the entity, followed by the kind of data. Examples
@@ -376,7 +380,7 @@ you'll probably want to do one of the following things.
 ## Download raw files
 
 You can use [queries](#queries) to download raw files, but typically you don't want to
-download file-by-file or entry-by-entry. Therefore, we allow to download a large set of
+download file by file or entry by entry. Therefore, we allow to download a large set of
 files in one big zip-file. Here, you might want to use a program like *curl* to download
 directly from the shell:
 
@@ -447,9 +451,9 @@ page size.
 ## User Groups
 
 You can create a user group and add other users to that group via the API. These groups
-can be used as [upload members](../gui/upload.md#visibility-and-access) to make it
-easier to give viewing or editing rights for an upload to multiple users at once, e.g.
-a working group.
+can be added as [project collaborators](../gui/upload.md#visibility-and-access) to make it
+easier to give viewing or editing rights for a project to multiple users at once, e.g.
+a working group. In the API, project collaborators are represented as upload members.
 
 To create a group, send an authenticated POST request to `/groups` including the
 `group_name` and a list of user IDs in the `members` field:

docs/howto/manage/program/download.md

@@ -24,9 +24,10 @@ Refer to our guide for [auth with `curl`](./auth.md#with-curl).
 ### Raw files vs processed data
 
 We are covering two types of resources: *raw files* and *processed data*.
-The former is organized into uploads and sub directory. The organization depends
+The former is organized into backend uploads and subdirectories. Each upload
+resource corresponds to a project in the GUI. The organization depends
 on how the author was providing the files.
-The later is organized by entries. Each NOMAD entry has corresponding structured data.
+The latter is organized by entries. Each NOMAD entry has corresponding structured data.
 
 Endpoints that target raw files typically contain `raw`, e.g. `uploads/<id>/raw`
 or `entries/raw/query`. Endpoints that target processed data contain `archive`
@@ -35,73 +36,80 @@ or `entries/raw/query`. Endpoints that target processed data contain `archive`
 
 ### Entry vs upload
 
-API endpoints for data download either target *entries* or *uploads*. For both types
+API endpoints for data download either target *entries* or backend *uploads*. For both types
 of entities, endpoints for raw files and processed data (as well as searchable metadata)
 exist. API endpoint paths start with the entity, e.g. `uploads/<id>/raw` or `entries/<id>/raw`.
 
-## Download a whole upload
+## Download a whole project
 
-Let's assume you want to download an entire upload. In this example the upload ID is
+The API represents a project as an upload resource. To download an entire project,
+use its upload ID—the value shown as **Project ID** under **SETTINGS** in the GUI.
+In this example, the upload ID is
 `wW45wJKiREOYTY0ARuknkA`.
 
 ```sh
 curl -X GET "{{ nomad_url() }}/v1/uploads/wW45wJKiREOYTY0ARuknkA/raw" -o download.zip
 ```
 
-This will create a `download.zip` file in the current folder. The zip file will contain
-the raw file directory of the upload.
+This creates a `download.zip` file in the current folder containing the project's
+raw-file directory.
 
-The used `uploads/<id>/raw` endpoint is only available for published uploads. For those,
+The `uploads/<id>/raw` endpoint is only available for published projects. For those,
 all raw files have already been
-packed into a zip file and this endpoint simply lets you download it. This is the simplest
+packed into a ZIP file and this endpoint simply lets you download it. This is the simplest
 and most reliable download implementation.
 
 Alternatively, you can download specific files or sub-directories. This method is available
-for all uploads. Including un-published uploads.
+for all projects, including unpublished projects.
 
 ```sh
 curl -X GET "{{ nomad_url() }}/v1/uploads/wW45wJKiREOYTY0ARuknkA/raw/?compress=true" -o download.zip
 ```
 
 This endpoint looks very similar, but is implemented very differently. Note that we
 put an empty path `/` to the end of the URL, plus a query parameter `compress=true`.
-The path can be replaced with any directory or file path in the upload; `/` would denote the
-whole upload. The query parameter says that we want to download the whole directory
-as a zip file, instead of an individual file. This traverses through all files and
-creates a zip file on the fly.
+The path can be replaced with any directory or file path in the project; `/` denotes the
+whole project. The query parameter says that we want to download the whole directory
+as a ZIP file instead of an individual file. This traverses all files and
+creates a ZIP file on the fly.
 
-## Download a whole dataset
+## Download raw files matching a query
 
-Now let's assume that you want to download all raw files that are associated with
-all the entries of an entire dataset. In this example the dataset DOI is
-`10.17172/NOMAD/2023.11.17-2`.
+To download raw files associated with multiple entries, submit a search query to
+the `entries/raw/query` endpoint. This example selects entries whose material
+contains both titanium and oxygen:
 
 ```sh
 curl -X POST "{{ nomad_url() }}/v1/entries/raw/query" \
 -H 'Content-Type: application/json' \
 -d '{
     "query": {
-        "datasets.doi": "10.17172/NOMAD/2023.11.17-2"
+        "results.material.elements": {
+            "all": ["Ti", "O"]
+        }
     }
 }' \
 -o download.zip
 ```
 
-This time, we use the `entries/raw/query` endpoint that is based on entries and not on uploads. Here, we
-select entries with a query. In the example, we query for the dataset DOI, but you
-can replace this with any NOMAD search query (look out for the `<>` symbol on the
-[search interface]({{ nomad_url() }}/../gui/search/entries)). The zip file will contain all raw files from all the
-directories that have the mainfile of one of the entries that match the queries.
+The ZIP file contains the raw files from directories containing the mainfiles of
+the matching entries. Each project's files are stored under a directory named
+with its upload ID. The ZIP file also contains `manifest.json`, which maps each
+entry to its upload ID, entry ID, and mainfile. You can replace the example with
+any [How-to guides > ... > Queries](api.md#queries).
 
-This might not necessarily download all uploaded files. Alternatively, you can use a query to
-get all upload ids and then use the method from the previous section:
+This does not necessarily download every file in each matching project. To
+download complete projects instead, aggregate the upload IDs for the matching
+entries and use the method from the previous section:
 
 ```sh
 curl -X POST "{{ nomad_url() }}/v1/entries/query" \
 -H 'Content-Type: application/json' \
 -d '{
     "query": {
-        "datasets.doi": "10.17172/NOMAD/2023.11.17-2"
+        "results.material.elements": {
+            "all": ["Ti", "O"]
+        }
     },
     "pagination": {
         "page_size": 0
@@ -116,24 +124,40 @@ curl -X POST "{{ nomad_url() }}/v1/entries/query" \
 }'
 ```
 
-The last command will print JSON data that contains all the upload ids. It uses
+The last command prints JSON data containing all the upload IDs. It uses
 the `entries/query` endpoint that allows you to query NOMAD's search.
 It does not return any results (`page_size: 0`),
-but performs an aggregation over all search results and collects the upload ids
+but performs an aggregation over all search results and collects the upload IDs
 from all entries.
 
-## Download some processed data for a whole dataset
+Existing dataset links and DOIs are still resolved by NOMAD. To download the
+entries from one of these retained backend collections, use the
+`entries/raw/query` endpoint with a query for its DOI:
+
+```json
+{
+    "query": {
+        "datasets.doi": "10.17172/NOMAD/2023.11.17-2"
+    }
+}
+```
+
+You can instead identify an existing dataset by using `datasets.dataset_id`.
+
+## Download processed data matching a query
 
 Similar to raw files, you can also download processed data. This is also an
-entry based operation based on a query. This time we also specify a `required`
+entry-based operation based on a query. This time we also specify a `required`
 to explain which parts of the processed data, we are interested in:
 
 ```sh
 curl -X POST "{{ nomad_url() }}/v1/entries/archive/download/query" \
 -H 'Content-Type: application/json' \
 -d '{
     "query": {
-        "datasets.doi": "10.17172/NOMAD/2023.11.17-2"
+        "results.material.elements": {
+            "all": ["Ti", "O"]
+        }
     },
     "required": {
         "metadata": {
@@ -154,9 +178,11 @@ curl -X POST "{{ nomad_url() }}/v1/entries/archive/download/query" \
 -o download.zip
 ```
 
-Here we use the `entries/archive/download/query` endpoint. The result is a zip file
-with one json file per entry. There are no directories and the files are named
-`<entry-id>.json`. To associate the json files with entries, you should require
-information that tells you more about the entries, e.g. `required.metadata.mainfile`.
+Here we use the `entries/archive/download/query` endpoint. The resulting ZIP
+file contains one JSON file per entry, stored as
+`<upload-id>/<entry-id>.json`. It also contains `manifest.json`, which records
+the upload ID, entry ID, parser name, and JSON path for each result. Request
+additional metadata such as `required.metadata.mainfile` when it will help you
+interpret the downloaded processed data.
 
-See also the [How to access processed data](./archive_query.md) how-to guide.
+See also [How-to guides > ... > Access processed data](./archive_query.md).

docs/howto/manage/program/graph_api.md

@@ -129,7 +129,9 @@ There are a few existing data resources (called documents) stored in MongoDB (se
 
 1. `uploads`: The metadata of an upload, including, `upload_id`, `upload_name`, `main_author`, etc.
 2. `entries`: The metadata of an entry, including, `entry_id`, `entry_create_time`, `mainfile`, etc.
-3. `datasets`: The metadata of a dataset, including, `dataset_id`, `dataset_name`, `user_id`, etc.
+3. `datasets`: A retained backend resource for existing dataset collections,
+   including `dataset_id`, `dataset_name`, `user_id`, etc. The GUI can resolve
+   and display these datasets, but projects are the current user-facing unit.
 4. `groups`: The metadata of a user group, including, `owner`, `members`, etc.
 
 One can apply the same logic to fetch data from these structures.