Training on RE10K question

[jackswl]

Hi, we noticed that the repo's data_config.py points at SpatialVID:
video_dirs = ["data/spatialvid-hq/SpatialVID/videos"]

For your paper, you used RE10K, but does it also follow the 81 frames @ 16 fps protocol? Does that mean, many of the RE10K clips will be filtered, since most of the RE10K clips are not long enough to pass this filter? Or did you just clip 81 frames @ 16 fps from the raw youtube itself instead of the official RE10K clip?

If possible, how many RE10K training videos did you use to train Mirage? Thanks.

@MD-Student

[MD-Student]

Hey, the spatialvid-hd configuration is a legacy path in the code from early development. We collect about 40k samples from RE10K.

[jackswl]

@MD-Student, hey thanks for responding. in this case, what did you use to train your RE10K? is it 81 frames @ 16 fps? do you do multiple windows per clip? are you able to share with us on this? thank you!

[MD-Student]

@MD-Student, hey thanks for responding. in this case, what did you use to train your RE10K? is it 81 frames @ 16 fps? do you do multiple windows per clip? are you able to share with us on this? thank you!

Hey! We use 81 frames @ 16fps. All clip windows are sourced from original re10k dataset with no additional spliting.

[jackswl]

@MD-Student, however, if we do 81 frames @ 16 fps, it is not possible to reach 40K samples of RE10K isnt it? because most of the RE10K clips are not long enough to get into the 81 frames @ 16 fps. that means every source clip must be ≥5.0625 s, which is more than the median of the RE10K official data. Could you kindly advise on this? thank you so much

[jackswl]

@MD-Student from the official camera files, only ~24.7K of the 71,556 train clips are ≥5.0625 s (81 frames @ 16 fps; the median official clip is 3.84 s). So ~40K samples doesn't seem possible with one window per clip at that spec.. hoping to figure this puzzle out! haha

[MD-Student]

Thanks for checking this carefully. Your calculation seems right. My earlier comment: the “about 40K” figure was an estimate from memory because I am not accessed to the data cluster right now. For training, we used all available RE10K training clips that met the preprocessing requirements (81 frames @ 16 fps, about 5s as you calculated). Sorry for the confusion.

[jackswl]

Thank you so much! this is very helpful.