Skipping global process sync, barrier name: create_tmp_directory:post.checkpoint_1
I0716 20:30:05.510361 140313332638656 utils.py:219] Skipping global process sync, barrier name: Checkpointer:save.checkpoint_1
Traceback (most recent call last):
File "/mnt/sdb1/sumant/multi_nerf/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/mnt/sdb1/sumant/multi_nerf/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/mnt/sdb1/sumant/multi_nerf/multinerf/train.py", line 292, in <module>
app.run(main)
File "/mnt/sdb1/sumant/multi_nerf/lib/python3.9/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/mnt/sdb1/sumant/multi_nerf/lib/python3.9/site-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/mnt/sdb1/sumant/multi_nerf/multinerf/train.py", line 222, in main
checkpoints.save_checkpoint(
File "/mnt/sdb1/sumant/multi_nerf/lib/python3.9/site-packages/flax/training/checkpoints.py", line 697, in save_checkpoint
orbax_checkpointer.save(
File "/mnt/sdb1/sumant/multi_nerf/lib/python3.9/site-packages/orbax/checkpoint/checkpointer.py", line 186, in save
self._handler.finalize(tmpdir)
File "/mnt/sdb1/sumant/multi_nerf/lib/python3.9/site-packages/orbax/checkpoint/pytree_checkpoint_handler.py", line 759, in finalize
self._handler_impl.finalize(directory)
File "/mnt/sdb1/sumant/multi_nerf/lib/python3.9/site-packages/orbax/checkpoint/base_pytree_checkpoint_handler.py", line 1064, in finalize
type_handlers.merge_ocdbt_per_process_files(directory)
File "/mnt/sdb1/sumant/multi_nerf/lib/python3.9/site-packages/orbax/checkpoint/type_handlers.py", line 635, in merge_ocdbt_per_process_files
parent_tspec = _get_tensorstore_spec(os.fspath(directory), use_ocdbt=True)
File "/mnt/sdb1/sumant/multi_nerf/lib/python3.9/site-packages/orbax/checkpoint/type_handlers.py", line 745, in _get_tensorstore_spec
raise ValueError(f'Checkpoint path should be absolute. Got {directory}')
ValueError: Checkpoint path should be absolute. Got data_colmap/checkpoints/checkpoint_1.orbax-checkpoint-tmp-0
got the below error while training the custom data