Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/en/administration/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,8 @@ Example:
103: 1716440760.000000000|UNLINK(1,report.txt,0,false,true):1024|(3,90)
```

Directory object updates made by the S3 Gateway use a single `SETDIRMARKER(inode,ctime,ctimensec,xattrs)` record. The operation sets the directory's atime and atimensec to zero and applies all listed extended attributes together. The `xattrs` argument is percent-escaped JSON: values are base64-encoded bytes, `null` removes an attribute, and omitted names remain unchanged. External consumers must handle this operation before processing changelogs from gateways that use atomic directory marker updates. The stored inode and xattr formats are unchanged.

## Notes and limitations {#notes}

- The changelog is not a metadata backup. Use [metadata backup](metadata_dump_load.md) for backup and restore.
Expand Down
8 changes: 8 additions & 0 deletions docs/en/guide/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,14 @@ Common application scenarios for JuiceFS S3 Gateway include:
- **Managing files in JuiceFS:** JuiceFS S3 Gateway provides a web-based file manager to manage files in JuiceFS directly from a browser.
- **Cluster replication:** In scenarios requiring cross-cluster data replication, JuiceFS S3 Gateway serves as a unified data export for clusters. This avoids cross-region metadata access and enhances data transfer performance. For details, see [Sync across regions using JuiceFS S3 Gateway](../guide/sync.md#sync-across-region).

## Conditional directory objects

A zero-byte object whose key ends in `/` is represented by a directory with an explicit object marker. Uploading `prefix/child.txt` creates the parent directory, but does not create the S3 object `prefix/`.

With the default `--head-dir=false`, a `PUT prefix/` or a zero-byte copy to `prefix/` with `If-None-Match: *` can create the marker on this existing directory. The gateway preserves the directory inode and children, and commits the marker and its managed extended attributes in one metadata transaction. Concurrent conditional creators have one winner; subsequent requests return `412 Precondition Failed` without changing the object.

When `--head-dir` is enabled, implicit directories are exposed as existing objects, so the same conditional request returns `412`. Without the conditional header, a zero-byte copy to `prefix/` creates or overwrites the directory object while preserving existing children. Directory objects must have an empty body.

## Quick start

JuiceFS S3 Gateway enables access to an existing JuiceFS volume. If you do not have one, follow the steps in this [guide](../getting-started/standalone.md) to create a JuiceFS file system.
Expand Down
2 changes: 2 additions & 0 deletions docs/zh_cn/administration/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,8 @@ VERSION: UNIX_SECONDS.NANOSECONDS|OPERATION(arguments)[:result]|(SESSION_ID,TXN_
103: 1716440760.000000000|UNLINK(1,report.txt,0,false,true):1024|(3,90)
```

S3 网关更新目录对象时使用单条 `SETDIRMARKER(inode,ctime,ctimensec,xattrs)` 记录。该操作将目录的 atime 和 atimensec 设为零,并一并更新列出的扩展属性。`xattrs` 参数为经过百分号转义的 JSON,属性值使用 base64 编码,`null` 表示删除,未列出的属性保持不变。外部消费者在处理使用原子目录标记更新的网关所产生的日志前,需要支持该操作。inode 和扩展属性的存储格式不变。

## 使用建议和限制 {#notes}

- changelog 不是元数据备份。备份和恢复应使用[元数据备份](metadata_dump_load.md)。
Expand Down
8 changes: 8 additions & 0 deletions docs/zh_cn/guide/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,14 @@ JuiceFS S3 网关的常见的使用场景有:
- **管理 JuiceFS 中的文件**:S3 网关提供了一个基于网页的文件管理器,可以在浏览器中管理 JuiceFS 中的文件;
- **集群复制**:在跨集群复制数据的场景下,作为集群的统一数据出口,避免跨区访问元数据以提升数据传输性能,详见[「使用 S3 网关进行跨区域数据同步」](../guide/sync.md#sync-across-region)

## 目录对象的条件写入

键名以 `/` 结尾的零字节对象,在 JuiceFS 中由带有显式对象标记的目录表示。上传 `prefix/child.txt` 会自动创建父目录,但不会创建 S3 对象 `prefix/`。

默认 `--head-dir=false` 时,可以通过带有 `If-None-Match: *` 的 `PUT prefix/` 或向 `prefix/` 复制零字节对象,为已有目录创建对象标记。网关保留目录 inode 和子文件,并在一次元数据事务中提交对象标记及相关扩展属性。并发条件创建只有一个请求成功,后续请求返回 `412 Precondition Failed`,且不会修改已有对象。

启用 `--head-dir` 时,隐式目录也被视为已存在的对象,因此上述条件请求返回 `412`。不带条件头时,向 `prefix/` 复制零字节对象会创建或覆盖目录对象,并保留已有子文件。目录对象的请求体必须为空。

## 快速开始

启动 S3 网关需要一个已经创建完毕的 JuiceFS 文件系统,如果尚不存在,请参考[文档](../getting-started/standalone.md)来创建。下方假定元数据引擎 URL 为 `redis://localhost:6379/1`。
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -346,7 +346,7 @@ require (
xorm.io/builder v0.3.13 // indirect
)

replace github.com/minio/minio v0.0.0-20210206053228-97fe57bba92c => github.com/juicedata/minio v0.0.0-20260515071949-69a6cfc9da65
replace github.com/minio/minio v0.0.0-20210206053228-97fe57bba92c => github.com/juicedata/minio v0.0.0-20260910033240-f2266df40f17

replace github.com/hanwen/go-fuse/v2 v2.1.1-0.20210611132105-24a1dfe6b4f8 => github.com/juicedata/go-fuse/v2 v2.1.1-0.20260819084346-22b3157c2d7f

Expand Down
2 changes: 2 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -505,6 +505,8 @@ github.com/juicedata/huaweicloud-sdk-go-obs v3.22.12-0.20230228031208-386e87b5c0
github.com/juicedata/huaweicloud-sdk-go-obs v3.22.12-0.20230228031208-386e87b5c091+incompatible/go.mod h1:Ukwa8ffRQLV6QRwpqGioPjn2Wnf7TBDA4DbennDOqHE=
github.com/juicedata/minio v0.0.0-20260515071949-69a6cfc9da65 h1:u+3ehBnL0r3uQloSRQ6QlY2vffBptFSSN6EeCJqgirE=
github.com/juicedata/minio v0.0.0-20260515071949-69a6cfc9da65/go.mod h1:1/4WHQKDOsWA1dd3ADrq9IE/jtFec9MHLy656kIXjNg=
github.com/juicedata/minio v0.0.0-20260910033240-f2266df40f17 h1:PZFzmLxyPGwk/IKgLqr0SuO0P2MI9lq2dlmH0lIX2rQ=
github.com/juicedata/minio v0.0.0-20260910033240-f2266df40f17/go.mod h1:1/4WHQKDOsWA1dd3ADrq9IE/jtFec9MHLy656kIXjNg=
github.com/juicedata/mpb/v7 v7.0.4-0.20231024073412-2b8d31be510b h1:0/6suPNZnrOlRlBaU/Bnitu8HiKkkLSzQhHbwQ9AysM=
github.com/juicedata/mpb/v7 v7.0.4-0.20231024073412-2b8d31be510b/go.mod h1:NXGsfPGx6G2JssqvEcULtDqUrxuuYs4llpv8W6ZUpzk=
github.com/juicedata/xorm v1.4.2-0.20260909084754-f6c8d2b84ec2 h1:j1Ngfz6mtEA8YUhhN3ULLz9UNprl5HLv0SKxsN5cvEY=
Expand Down
7 changes: 5 additions & 2 deletions integration/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,11 @@
all: s3test webdav ioctl

s3test:
pip install awscli==1.27.153
bash s3gateway_test.sh
@awscli_venv=$$(mktemp -d "$${TMPDIR:-/tmp}/juicefs-awscli.XXXXXX"); \
trap 'rm -rf "$$awscli_venv"' EXIT INT TERM; \
python3 -m venv "$$awscli_venv"; \
"$$awscli_venv/bin/pip" install awscli==1.44.73; \
PATH="$$awscli_venv/bin:$$PATH" bash s3gateway_test.sh

webdav:
cd /home/travis/.m2/litmus-0.13 ; for i in "basic" "copymove" "http"; do sudo ./$${i} http://127.0.0.1:9009 root 1234; done
Expand Down
Loading
Loading