The original post: /r/datahoarder by /u/T-nash on 2024-10-10 10:53:52.
Basically I am about to start converting VHS tapes to digital using uncompressed YUV422BT.601 VCM on virtualdub capturing with Hauppauge WinTV-HVR-1250.
The video files are like 300GB large for obvious reasons, I do not wish to process them today and leave enhancing them in the future once software improves further.
That said, I downloaded a test video from the internet in the format of YUV4MPEG2 4:2:0 in y4m container, and used
ffmpeg -i elephants_dream_1080p24.y4m -c:v libx265 -preset slow -x265-params lossless=1 -pix_fmt yuv420p I:\output_lossless_x265.mkv
The video size went from 47,661,505 KB to 5,581,389 KB, which is great
Then to test restoring it back to the original form I used
ffmpeg -i I:\output_lossless_x265.mkv -c:v rawvideo -f yuv4mpegpipe elephants_dream_restored.y4m
The restored file had the exact same original size of 47,661,505 KB, media info matched, but MD5 checksum didn't, I assumed it had something to do with the header, so I extracted the same frames on both source and restored videos and checksummed those, the hashes matched, meaning they're pixel perfect :)
Anyway, I tried archiving with winrar as well using these settings
create solid arhive
add recovery record (10% record)
Lock archive
1GB dictionary size
split volumes in 2GB
compression method: best
I ended up with a total of 9.64GB file size after compression.
So,
Source: 47.6GB
x265 lossless: 5.5GB
Winrar: 9.64GB
I will be storing on an unraid array running XFS with a parity disk, I do not have ECC rams, I am not familiar with bit rot and those factors about data storage, I am just wondering which would be the best choice between winrar with recovery archives or x265 lossless, I just don't want to come back 5-10 years later and find them damaged. It's worth nothing I don't always replace hdd's with slight errors, don't always have the budget.
Here's media info if anyone is interested
Source video:
Complete name : E:\elephants_dream_1080p24.y4m
Format : YUV4MPEG2
File size : 45.5 GiB
Duration : 10 min 53 s
Overall bit rate : 597 Mb/s
Frame rate : 24.000 FPS
Video
Format : YUV
Duration : 10 min 53 s
Bit rate : 597 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 24.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Scan type : Progressive
Compression mode : Lossless
Bits/(Pixel*Frame) : 12.000
Stream size : 45.5 GiB (100%)
X265 --lossless
Unique ID : 228437949554887390664296614558885315335 (0xABDB8C7AC3616332E7D3FDA223B18307)
Complete name : E:\elephants_dream_1080p24_output_lossless_x265.mkv
Format : Matroska
Format version : Version 4
File size : 5.32 GiB
Duration : 10 min 53 s
Overall bit rate : 69.9 Mb/s
Frame rate : 24.000 FPS
Writing application : Lavf61.5.101
Writing library : Lavf61.5.101
ErrorDetectionType : Per level 1
Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main@L8.5@Main
Codec ID : V_MPEGH/ISO/HEVC
Duration : 10 min 53 s
Bit rate : 68.5 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 24.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0 (Type 1)
Bit depth : 8 bits
Bits/(Pixel*Frame) : 1.377
Stream size : 5.22 GiB (98%)
Writing library : x265 4.0+4-28de550a2:[Windows][GCC 14.2.0][64 bit] 8bit+10bit+12bit
Encoding settings : cpuid=1111039 / frame-threads=4 / numa-pools=16 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=1920x1080 / interlace=0 / total-frames=0 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=4 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-eob / no-eos / no-hrd / info / hash=0 / temporal-layers=0 / open-gop / min-keyint=24 / keyint=250 / gop-lookahead=0 / bframes=4 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=25 / lookahead-slices=4 / scenecut=40 / no-hist-scenecut / radl=0 / no-splice / no-intra-refresh / ctu=64 / min-cu-size=8 / rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=2 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=3 / limit-refs=3 / limit-modes / me=3 / subme=3 / merange=57 / temporal-mvp / no-frame-dup / no-hme / weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / sao / no-sao-non-deblock / rd=4 / selective-sao=4 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=1.00 / no-rd-refine / lossless / cbqpoffs=0 / crqpoffs=0 / rc=cqp / qp=4 / ipratio=1.40 / pbratio=1.30 / aq-mode=0 / aq-strength=0.00 / no-cutree / zone-count=0 / no-strict-cbr / qg-size=64 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=1 / overscan=0 / videoformat=5 / range=0 / colorprim=2 / transfer=2 / colormatrix=2 / chromaloc=1 / chromaloc-top=1 / chromaloc-bottom=1 / display-window=0 / cll=0,0 / min-luma=0 / max-luma=255 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr10 / no-hdr10-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=0 / analysis-save-reuse-level=0 / analysis-load-reuse-level=0 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=1 / refine-ctu-distortion=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / no-svt / no-field / qp-adaptation-range=1.00 / scenecut-aware-qp=0conformance-window-offsets / right=0 / bottom=0 / decoder-max-rate=0 / no-vbv-live-multi-pass / no-mcstf / no-sbrc
Default : No
Forced : No
Color range : Limited
Restored from x265 --lossless to YUV
Complete name : E:\elephants_dream_restored.y4m
Format : YUV4MPEG2
File size : 45.5 GiB
Duration : 10 min 53 s
Overall bit rate : 597 Mb/s
Frame rate : 24.000 FPS
Video
Format : YUV
Duration : 10 min 53 s
Bit rate : 597 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 24.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Scan type : Progressive
Compression mode : Lossless
Bits/(Pixel*Frame) : 12.000
Stream size : 45.5 GiB (100%)