cephalobox: support custom registry and images - #114
Open
nizamial09 wants to merge 3 commits into
Open
Conversation
nizamial09
force-pushed
the
fixups-cephalobox
branch
2 times, most recently
from
August 11, 2026 06:55
fb644af to
ac98fa6
Compare
for downstream cluster bootstrap, support registry and custom container images option. Deploying a cluster with custom image 1. Set `CEPHADM_IMAGE` in the .env file. ```bash CEPHADM_IMAGE=my.registry/ceph/ceph ``` Optionally, if the registry is behind auth, provide the registry details in its env variables in .env ```bash REGISTRY_URL= REGISTRY_USERNAME= REGISTRY_PASSWORD= ``` You can also set CONTAINER_IMAGE_* for each of the images incase you want to override the defaults ```bash CONTAINER_IMAGE_PROMETHEUS= CONTAINER_IMAGE_NODE_EXPORTER= CONTAINER_IMAGE_ALERTMANAGER= CONTAINER_IMAGE_GRAFANA= ``` 2. Once done all of those, start the cluster as ```bash podman pull cephalobox podman compose up --attach-dependencies cephalobox ``` 3. The image pulling happens in its own service which can be logged as ```bash podman compose logs -f ceph-image-puller ``` Disable Shared Ceph Folder 1. Set `SHARED_CEPH_REPO_DIR` to 0 to disable cephadm from mounting your local directory. 2. Start the cluster ```bash podman compose up --attach-dependencies cephalobox ``` This will pull the images locally and then mount them to the cephalobox container so that the image pulling is more faster. Signed-off-by: Nizamudeen A <nia@redhat.com>
nizamial09
force-pushed
the
fixups-cephalobox
branch
from
August 11, 2026 06:57
ac98fa6 to
fc0db29
Compare
1. Upto 5 clusters are defined in the docker-compose.yml template. This can be started by
```bash
podman compose up -d cephalobox cephalobox2 cephalobox3 cephalobox4 cephalobox5
```
2. You can view individual logs by doing
```bash
pdoamn compose logs -f cephalobox4
```
Or just all the logs by
```bash
podman compose logs -f
```
3. You can also add more clusters there by following the template defined there. Just make sure the port is mapped correctly.
For eg. inside docker-compose.yml
```yml
cephalobox6:
<<: *cephalobox-base
container_name: cephalobox6
hostname: cephalobox6
ports:
- "8448:8443"
- "3004:3000"
```
and then start as
```bash
podman compose up -d cephalobox6
```
Signed-off-by: Nizamudeen A <nia@redhat.com>
Pegonzal
reviewed
Aug 11, 2026
Pegonzal
left a comment
Pegonzal
left a comment
Member
There was a problem hiding this comment.
@nizamial09 I was trying to test the deployment but it's failing to start the mgr:
cephalobox | Aug 11 16:41:26 cephalobox ceph-mgr[2457]: asok(0x5630358b6000) register_command objecter_requests hook 0x56303581ecf0
cephalobox | Aug 11 16:41:26 cephalobox ceph-mgr[2457]: monclient: _renew_subs
cephalobox | Aug 11 16:41:26 cephalobox ceph-mgr[2457]: monclient: _send_mon_message to mon.cephalobox at v2:172.20.0.3:3300/0
cephalobox | Aug 11 16:41:26 cephalobox ceph-mgr[2457]: *** Caught signal (Aborted) **
cephalobox | in thread 7f25f3ea2fc0 thread_name:ceph-mgr
cephalobox | 12: (MgrStandby::init()+0x592) [0x56301620df72]
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 0> 2026-08-11T16:41:26.161+0000 7f25f3ea2fc0 -1 *** Caught signal (Aborted) **
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: in thread 7f25f3ea2fc0 thread_name:ceph-mgr
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]:
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: ceph version 21.3.0-1694-gaccf15ad (accf15ad0ec54a627a6954d9253d23b815465959) umbrella (dev - RelWithDebInfo)
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 1: /lib64/libc.so.6(+0x411c0) [0x7f25f42681c0]
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 2: /lib64/libc.so.6(+0x9781c) [0x7f25f42be81c]
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 3: gsignal()
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 4: abort()
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 5: /lib64/libstdc++.so.6(+0xa603a) [0x7f25f30a603a]
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 6: /lib64/libstdc++.so.6(+0xbc0ac) [0x7f25f30bc0ac]
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 7: (std::unexpected()+0) [0x7f25f30a5a39]
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 8: /lib64/libstdc++.so.6(+0xbc338) [0x7f25f30bc338]
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 9: /lib64/libstdc++.so.6(+0xb0614) [0x7f25f30b0614]
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 10: (PyModuleRegistry::probe_modules(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x6ad) [0x563016229d9d]
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 11: (PyModuleRegistry::init()+0x3af) [0x56301622a4bf]
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 12: (MgrStandby::init()+0x592) [0x56301620df72]
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 13: main()
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 14: /lib64/libc.so.6(+0x2a58e) [0x7f25f425158e]
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 15: __libc_start_main()
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: 16: _start()
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
cephalobox | Aug 11 16:41:26 cephalobox ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1-mgr-cephalobox-ytzvsv[2453]:
cephalobox | Aug 11 16:41:26 cephalobox systemd[1]: ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1@mgr.cephalobox.ytzvsv.service: Main process exited, code=exited, status=134/n/a
cephalobox | Aug 11 16:41:26 cephalobox systemd[1]: ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1@mgr.cephalobox.ytzvsv.service: Failed with result 'exit-code'.
cephalobox | [ OK ] Started libcrun container.
cephalobox | [72896.678623] bash[90]: mgr not available, waiting (15/15)...
cephalobox | Aug 11 16:41:28 cephalobox goofy_lumiere[2551]: "mgrmap": {
cephalobox | Aug 11 16:41:28 cephalobox bash[90]: mgr not available, waiting (15/15)...
cephalobox | [ OK ] Started libcrun container.
cephalobox | [72899.144536] bash[90]: Error: mgr not available after 15 tries
cephalobox | [72899.144637] bash[90]: ***************
cephalobox | [72899.144680] bash[90]: Cephadm hit an issue during cluster installation. Current cluster files will be deleted automatically.
cephalobox | [72899.144718] bash[90]: To disable this behaviour you can pass the --no-cleanup-on-failure flag. In case of any previous
cephalobox | [72899.144752] bash[90]: broken installation, users must use the following command to completely delete the broken cluster:
cephalobox | [72899.144780] bash[90]: > cephadm rm-cluster --force --zap-osds --fsid <fsid>
cephalobox | [72899.144817] bash[90]: for more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster
cephalobox | [72899.144854] bash[90]: ***************
cephalobox | [72899.144892] bash[90]: Deleting cluster with fsid: 65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1
cephalobox | Stopping Ceph mon.cephalobox for 6…e0f8-95a3-11f1-ad7d-fa4d37bbceb1...
cephalobox | Aug 11 16:41:30 cephalobox beautiful_chatterjee[2610]: "mgrmap": {
cephalobox | Aug 11 16:41:30 cephalobox bash[90]: Error: mgr not available after 15 tries
cephalobox | Aug 11 16:41:30 cephalobox bash[90]: Cephadm hit an issue during cluster installation. Current cluster files will be deleted automatically.
cephalobox | Aug 11 16:41:30 cephalobox bash[90]: > cephadm rm-cluster --force --zap-osds --fsid <fsid>
cephalobox | Aug 11 16:41:30 cephalobox bash[90]: for more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster
cephalobox | [ OK ] Stopped Ceph mon.cephalobox for 65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1.
cephalobox | [ OK ] Stopped Ceph mgr.cephalobox.ytzvsv…c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1.
cephalobox | Aug 11 16:41:31 cephalobox systemd[1]: Stopped Ceph mgr.cephalobox.ytzvsv for 65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1.
cephalobox | [ OK ] Stopped target Ceph cluster 65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1.
cephalobox | [ OK ] Removed slice Slice /system/ceph-65c6e0f8-95a3-11f1-ad7d-fa4d37bbceb1.
cephalobox | [ OK ] Stopped target All Ceph clusters and services.
cephalobox | [72900.400420] bash[90]: ERROR: mgr not available after 15 tries
cephalobox | [72900.434654] bash[40]: [FATAL ERROR] Command failed with exit code 1
cephalobox | [FAILED] Failed to start Cephalobox.
cephalobox | See 'systemctl status cephalobox.service' for details.
cephalobox | [ OK ] Reached target Multi-User System.
cephalobox | Starting Record Runlevel Change in UTMP...
cephalobox | [ OK ] Finished Record Runlevel Change in UTMP.
cephalobox | Aug 11 16:41:31 cephalobox bash[90]: ERROR: mgr not available after 15 tries
Member
Silly me forgot to pull the image again and a broken mgr one was being used. Deployment started after: |
elevate NET_ADMIN privileges Signed-off-by: Nizamudeen A <nia@redhat.com>
nizamial09
force-pushed
the
fixups-cephalobox
branch
from
August 12, 2026 06:18
b0b199a to
bc2ef03
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Starting multiple clusters
Or just all the logs by
For eg. inside docker-compose.yml
and then start as
Deploying a cluster with custom image
CEPHADM_IMAGEin the .env file.Optionally, if the registry is behind auth, provide the registry details in its env variables in .env
You can also set CONTAINER_IMAGE_* for each of the images incase you want to override the defaults
Disable Shared Ceph Folder
Set
SHARED_CEPH_REPO_DIRto 0 to disable cephadm from mounting your local directory.Start the cluster
This will pull the images locally and then mount them to the cephalobox container so that the image pulling
is more faster.