[RFC0043] Cloud Controller Blobstore Type: storage-cli

[beyhan]

This is a tracking issue for RFC0043.

Delivered with:

• CF-Deployment version:
  • Azure: https://github.com/cloudfoundry/cf-deployment/releases/tag/v54.9.0
  • AliCloud: https://github.com/cloudfoundry/cf-deployment/releases/tag/v54.10.0
  • GCS: https://github.com/cloudfoundry/cf-deployment/releases/tag/v54.11.0
  • S3: https://github.com/cloudfoundry/cf-deployment/releases/tag/v55.2.0
  • webdav: https://github.com/cloudfoundry/cf-deployment/releases/tag/v58.1.0
• CLI version: n/a
• CF Docs: https://docs.cloudfoundry.org/deploying/common/cc-blobstore-config.html

Tasks

• New FI area: Storage CLI #1275
• Storage CLIs: bosh-davcli should live with others #1286
• Add davcli to Storage CLIs RFC #1288
• Create new storage-cli team concourse-infra-for-fiwg#76
• ADR: Use Bosh Storage CLIs for Blobstore Operations cloud_controller_ng#4443
• bosh-azure-storage-cli based blobstore client cloud_controller_ng#4476
• Improve error handling in Download method bosh-azure-storage-cli#21
• Extend bosh-azure-storage-cli: new blob commands + upload timeout bosh-azure-storage-cli#22
• Require azure_storage_cli_client cloud_controller_ng#4550
• Add Azure Storage CLI connection config to specs and yml.erb files capi-release#574
• Add storage-cli repo #1329
• Read storage-cli config JSON from capi (AzureRM) cloud_controller_ng#4581
• Add bosh-azure-storage-cli binary capi-release#577
• Azure storage-cli integration capi-release#580
• bosh storage cli repo consolidation storage-cli#1
• New Structure storage-cli#2
• Add ops file to use azure storage-cli capi-ci#154
• GCS integration test workflow storage-cli#11
• S3 Integration Tests Migration to Github Worfklow storage-cli#5
• Azurebs Integration Tests Migration to Github Worfklow storage-cli#6
• Alioss Integration Tests Migration to Github Worfklow storage-cli#9 (not yet running in FI AliCloud account)
• Implement missing storage-cli operations for AliCloud storage-cli#10
• One Main storage-cli#13
• Storage CLI Client cloud_controller_ng#4706
• Storage CLI Client capi-release#595
• Validate the consolidation with BOSH storage-cli#16
• GCS Extra Functionalities storage-cli#21
• Structured Logging and Debugging Option storage-cli#22
• UploadStream for Azure Client storage-cli#27
• Concurrent operations for alioss storage-cli#28
• Implement concurrent downloads and resumable chunked uploads for GCS storage-cli#31
• Use slog logger and enable s3 http request logs storage-cli#30
• (feat): make S3 transfer concurrency, part size, checksum configurable storage-cli#29
• Implement missing operations for s3 storage-cli#32
• Add Uniform Bucket-Level Access Support for GCS Buckets storage-cli#41
• Implement multipart copy for s3 storage-cli#44
• Build and publish linux arm64 variant storage-cli#51
• Add integration test for AWS ESC storage-cli#59
• SingleUploadThreshold config paramter is introduced. storage-cli#66
• Make S3 checksum settings configurable via JSON config storage-cli#81
• Add LocalClient for filesystem-based blobstore cloud_controller_ng#4980
• Modernize development environment setup cloud_controller_ng#5005
• Add fog deprecation warning cloud_controller_ng#5072
• Refactor dav: Adopt two-layer architecture to match other storage providers storage-cli#98
• Implement Copy, List, DeleteRecursive, Properties for DAV storage-cli#103
• Add Dav Signing Functionality storage-cli#105
• Add DAV integration tests  storage-cli#108
• Add WebDAV support to storage-cli client and improve test quality cloud_controller_ng#4974
• Enhance storage-cli webdav config capi-release#634

cf-deployment and docs:

• Change "use-azure-storage-blobstore.yml" ops file to use "storage-cli" cf-deployment#1309
• Update Azure Storage blobstore documentation docs-deploying-cf#214
• Change "use-alicloud-oss" ops files to use "storage-cli" cf-deployment#1314
• Cloud controller blobstore documentation for AliCloud OSS docs-deploying-cf#215
• Use storage-cli in ops file for gcs blobstore cf-deployment#1316
• Update blobstore configuration for Google Cloud Storage docs-deploying-cf#217
• Use "storage-cli" implementation in use-s3-blobstore ops file cf-deployment#1319
• Update AWS S3 blobstore configuration to Storage CLI docs-deploying-cf#218
• Use storage-cli for WebDAV blobstore cf-deployment#1356
• Remove "Fog" from Blobstore Configuration docs-deploying-cf#224
• Fix configuration for S3 compatible blobstores docs-deploying-cf#226
• Remove "Fog" leftovers cf-deployment#1364
• Remove remaining mentions of fog. docs-cloudfoundry-concepts#224

fog removal:

• Remove fog azure support from ccng coding and Gemfile cloud_controller_ng#5118
• Remove fog AzureRM legacy support from capi-release templates and specs capi-release#656
• Remove fog aliyun blobstore support cloud_controller_ng#5154
• Remove fog-aliyun support capi-release#660
• Remove use-swift-blobstore ops file and OpenStack blobstore references cf-deployment#1351
• Remove fog-local and fog-openstack blobstore support cloud_controller_ng#5197
• Remove nfs_mounter job capi-release#665
• Remove fog google blobstore support cloud_controller_ng#5261
• Remove fog-google support from storage-cli templates and specs capi-release#668
• Remove fog-aws and fog-core, complete fog removal cloud_controller_ng#5288
• Remove fog provider support  capi-release#671

[stephanme]

Planned roadmap:

• Enhance https://github.com/cloudfoundry/bosh-azure-storage-cli by missing operations and config params
• Add storage-cli provider to ccng and allow configuration for Azure
• validate in capi-ci using bbl env on Azure
• get experience with storage-cli provider in production (e.g. on Azure foundations at SAP)
• consolidate bosh storage cli implementations (Azure, AWS, GCP, AliCloud, WebDav) in a new repo (e.g. storage-cli)
  • bosh and ccng switch from single storage cli repos to the consolidated one
• implement missing operations and config params for the remaining IaaS in consolidated repo and enable them in ccng (one by one)

[stephanme]

storage-cli is meanwhile available for all IaaS providers and is used by default for CAPI when applying the external blobstore ops files of cf-deployment.

So it is time to deprecate fog and we plan to remove the blobstore_type fog starting in May 2026.

Migration tables:

blobstore_type	Comment
storage-cli	the standard blobstore provider for CAPI
fog	deprecated, will be removed
webdav	deprecated, will be replaced by a 'dav' provider for storage-cli (WIP)
local	new local file provider for development only (WIP), replaces Local fog provider

fog provider	storage-cli provider	Comment
AWS	s3	also for S3 compatible blobstores like Google and OpenStack Ceph
AzureRM	azurebs
aliyun	alioss
Google	gcs
OpenStack	n/a	use s3 provider instead, tested with OpenStack Ceph
Local (NFS)	n/a	development only, replaced by a new blobstore_type=local

[stephanme]

storage-cli is used by cf-deployment for all IaaS and webdav. fog gems were removed from cloud_controller_ng.