The EO AI Ready library allows to generate a training set which you can use with Prithvi-EO-2.0 geospatial foundation model and other foundation models supported by TerraTorch framework.
Given a ground truth dataset, the library generates an AI-ready dataset for the model by creating the necessary EO input files from Sentinel data.
To install the library, download the code and run the following command in the root folder of the repository:
pip install -e .In the following, substitute <GD_DATA> with the path of the folder where your ground truth data is stored and <TRAIN_DATA> with the path of the folder where you want to store the training dataset.
In all the following examples, the arguments --max_train_size, --max_validate_size and --max_test_size are used to limit the number of training samples to be generated. You can remove these arguments to generate training samples for the entire set of ground truth data.
The following command will generate a training dataset where the input for each ground truth data sample is a multi-band file which:
- contains B02, B03 Sentinel 2 bands
- matches the spatial coverage and resolution of the ground truth sample
- is as close as possible (depending on data availability) to the time instant of the ground truth data sample
python -m eoaiready generate --data_root <GD_DATA> --train_data_root <TRAIN_DATA> --bands B02 B03 --max_train_size 3 --max_validate_size 3 --max_test_size 3Create a training set with two Sentinel 2 bands and a different distribution of train, validate and test sets
By default, 65% of available ground truth samples are used to create the train set, 25% to create the validation set and 10% to create the test set. These values can be changed used respectively the following parameters:
- --train_percentage
- --validate_percentage
- --test_percentage
The following command will generate a training dataset where the input for each ground truth data sample is a multi-band file which:
- contains B02, B03 Sentinel 2 bands
- matches the spatial coverage and resolution of the ground truth sample
- is as close as possible (depending on data availability) to the time instant of the ground truth data sample
python -m eoaiready generate --data_root <GD_DATA> --train_data_root <TRAIN_DATA> --bands B02 B03 --train_percentage 0.6 --validate_percentage 0.2 --test_percentage 0.2The following command will generate a training dataset where the input for each ground truth data sample is composed of two multi-band files which:
- contain B02, B03 Sentinel 2 bands
- match the spatial coverage and resolution of the ground truth sample
- are selected from the same month of the time instant of the ground truth data sample, tentatively (depending on data availability) one in the first half and one in the second half of the month
python -m eoaiready generate --data_root <GD_DATA> --train_data_root <TRAIN_DATA> --bands B02 B03 --time_instants_per_target 2 --target_date_time_resolution monthly --max_train_size 3 --max_validate_size 3 --max_test_size 3Note
The --target_date_time_resolution monthly has been added for clarity, since the monthly value is the default when --time_instants_per_target > 1
The following command will generate a training dataset where the input for each ground truth data sample is composed of 4 multi-band files which:
- contain B02, B03 Sentinel 2 bands
- match the spatial coverage and resolution of the ground truth sample
- are selected from the same year of the time instant of the ground truth data sample, tentatively (depending on data availability) one every 3 months
python -m eoaiready generate --data_root <GD_DATA> --train_data_root <TRAIN_DATA> --bands B02 B03 --time_instants_per_target 4 --target_date_time_resolution yearly --max_train_size 3 --max_validate_size 3 --max_test_size 3The following table displays the mapping of the band names you can use in the generate command to the bands and platforms that will be used.
| Band Name | Mapping |
|---|---|
| B02 | Sentinel 2 L2A - B02 |
| B03 | Sentinel 2 L2A - B03 |
| B04 | Sentinel 2 L2A - B04 |
| B05 | Sentinel 2 L2A - B05 |
| B06 | Sentinel 2 L2A - B06 |
| B07 | Sentinel 2 L2A - B07 |
| B08 | Sentinel 2 L2A - B08 |
| B8A | Sentinel 2 L2A - B8A |
| B09 | Sentinel 2 L2A - B09 |
| B11 | Sentinel 2 L2A - B11 |
| B12 | Sentinel 2 L2A - B12 |
| hls-s30-B02 | Harmonized Landsat Sentinel - Sentinel B02 |
| hls-s30-B03 | Harmonized Landsat Sentinel - Sentinel B03 |
| hls-s30-B04 | Harmonized Landsat Sentinel - Sentinel B04 |
| hls-s30-B05 | Harmonized Landsat Sentinel - Sentinel B05 |
| hls-s30-B06 | Harmonized Landsat Sentinel - Sentinel B06 |
| hls-s30-B07 | Harmonized Landsat Sentinel - Sentinel B07 |
| hls-s30-B08 | Harmonized Landsat Sentinel - Sentinel B08 |
| hls-s30-B8A | Harmonized Landsat Sentinel - Sentinel B8A |
| hls-s30-B09 | Harmonized Landsat Sentinel - Sentinel B09 |
| hls-s30-B11 | Harmonized Landsat Sentinel - Sentinel B11 |
| hls-s30-B12 | Harmonized Landsat Sentinel - Sentinel B12 |
Once the training dataset has been generated, it is possible to automate the configuration file for running the training of Prithvi-EO-2.0 model using the TerraTorch framework.
In the following, substitute <TRAIN_DATA> with the path of the folder where you stored the training dataset and <TEMPLATE_FILE_PATH> with the path of a template configuration for TerraTorch. Also, you can optionally substitute <DEST_CONFIG_FILE> with the path where you want to store the generated configuration file (defaults to autogenerated.yaml).
python -m eoaiready configure --training_data <TRAIN_DATA> --template <TEMPLATE_FILE_PATH> --dest <DEST_CONFIG_FILE>The command will generate a configuration file that you can use to run the training of a model with TerraTorch.
It is also possible to generate a configuration file to use only a subset of the available bands.
python -m eoaiready configure --training_data <TRAIN_DATA> --template <TEMPLATE_FILE_PATH> --dest <DEST_CONFIG_FILE> --bands B02In this example we will generate a training dataset for a crop type classification task, using as ground truth data the Multi-Temporal Crop Classification with HLS Imagery across CONUS dataset. This dataset contains temporal Harmonized Landsat-Sentinel imagery of diverse land cover and crop type classes across the Contiguous United States for the year 2022.
The example downloads the original dataset and generates a small training dataset (16, 5, and 5 elements in train, validation and test splits) for the Prithvi-EO-2.0 model, using HLS Sentinel 2 bands. Finally, it runs a quick fine-tuning process (10 epochs) using the generated training dataset using TerraTorch.
To run the example execute the following command (substitute <WORKING_FOLDER> with the path of the folder where you want to store the generated training dataset:
python examples/multi-crop-e2e.py <WORKING_FOLDER>