My scenario is:
- 4TB nvme drive
- want to use thin provisioning
- don't care so much about snapshots, but if ever used they would have limited lifetime (e.g. a temp atomic snapshot for a backup tool).
- want to understand how to avoid running out of metadata, and simulate this
- want to optimize for nvme ssd performance where possible
I'm consulting man pages for lvmthin, lvcreate, and thin_metadata_size. Also thin-provisioning.txt seems like it might provide some deeper details.
When using lvcreate to create the thinpool, --poolmetadatasize can be provided if not wanting the default calculated value. The tool thin_metadata_size I think is intended to help estimate the needed values. One of the input args is --block-size, which sounds a lot like the --chunksize argument to lvcreate but I'm not sure.
man lvmthin has this to say about chunksize:
- The value must be a multiple of 64 KiB, between 64 KiB and 1 GiB.
- When a thin pool is used primarily for the thin provisioning feature, a larger value is optimal. To optimize for many snapshots, a smaller value reduces copying time and consumes less space.
Q1. What makes a larger chunksize optimal for primary use of thin provisioning? What are the caveats? What is a good way to test this? Does it make it harder for a whole chunk to be "unused" for discard to work and return the free space back to the pool?
thin_metadata_size describes --block-size as:
Block size of thin provisioned devices in units of bytes, sectors,
kibibytes, kilobytes, ... respectively. Default is in sectors without a
block size unit specifier. Size/number option arguments can be followed by
unit specifiers in short one character and long form (eg. -b1m or
-b1mebibytes).
And when using thin_metadata_size, I can tease out error messages block size must be a multiple of 64 KiB and maximum block size is 1 GiB. So it sounds very much like chunk size but I'm not sure.
The kernel doc for thin-provisioning.txt says:
-
$data_block_size gives the smallest unit of disk space that can be allocated at a time expressed in units of 512-byte sectors. $data_block_size must be between 128 (64KB) and 2097152 (1GB) and a multiple of 128 (64KB).
-
People primarily interested in thin provisioning may want to use a value such as 1024 (512KB)
-
People doing lots of snapshotting may want a smaller value such as 128 (64KB)
-
If you are not zeroing newly-allocated data, a larger $data_block_size in the region of 256000 (128MB) is suggested
-
As a guide, we suggest you calculate the number of bytes to use in the metadata device as 48 * $data_dev_size / $data_block_size but round it up to 2MB if the answer is smaller. If you're creating large numbers of snapshots which are recording large amounts of change, you may find you need to increase this.
This talks about "block size" like in thin_metadata_size, so still wondering if these are all the same as "chunk size" in lvcreate.
While man lvmthin just says to use a "larger" chunksize for thin provisioning, here we get more specific suggestions like 512KB, but also a much bigger 128MB if not using zeroing.
Q2. Should I disable zeroing with lvcreate option -Zn to improve SSD performance?
Q3. If so, is a 128MB block size or chunk size a good idea?
For a 4TB VG, testing out 2MB chunksize:
lvcreate --type thin-pool -l 100%FREE -Zn -n thinpool vg results in 116MB for [thinpool_tmeta] and uses a 2MB chunk size by default.48B * 4TB / 2MB = 96MB from kernel doc calcthin_metadata_size -b 2048k -s 4TB --max-thins 128 -u M = 62.53 megabytes
Testing out 64KB chunksize:
lvcreate --type thin-pool -l 100%FREE -Zn --chunksize 64k -n thinpool vg results in 3.61g for [thinpool_tmeta] (pool is 3.61t)48B * 4TB / 64KB = 3GB from kernel doc calcthin_metadata_size -b 64k -s 4TB --max-thins 128 -u M = 1984.66 megabytes
The calcs agree within the same order of magnitude, which could support that chunk size and block size are the same.
What actually uses metadata? I try the following experiment:
- create a 5GB thin pool (
lvcreate --type thin-pool -L 5G -n tpool -Zn vg)
- it used 64KB chunksize by default
- creates an 8MB metadata lv, plus spare
- initially Meta% = 10.64 per
lvs
- create 3 lvs, 2GB each (
lvcreate --type thin -n tvol$i -V 2G --thinpool tpool vg)
- Meta% increases for each one to 10.69, 10.74, then 10.79%
- write 1GB random data to each lv (
dd if=/dev/random of=/dev/vg/tvol$i bs=1G count=1)
- 1st: pool Data% goes to 20%, Meta% to 14.06% (+3.27%)
- 2nd: pool Data% goes to 40%, Meta% to 17.33% (+3.27%)
- 3rd: pool Data% goes to 60%, Meta% to 20.61% (+3.28%)
- take a snapshot (
lvcreate -s vg/tvol0 -n snap0)
- no change to metadata used
- write 1GB random data to the snapshot
- the device doesn't exist until
lvchange -ay -Ky vg/snap0
- then
dd if=/dev/random of=/dev/vg/snap0 bs=1G count=1
- pool Data% goes to 80%, Meta% to 23.93% (+3.32%)
- write 1GB random data to the origin of the snapshot
dd if=/dev/random of=/dev/vg/tvol0 bs=1G count=1- hmm, pools still at 80% Data% and 23.93% Meta%
- write 2GB random data
dd if=/dev/random of=/dev/vg/tvol0 bs=1G count=1- pool is now full 100% Data% and 27.15% Meta%
Observations:
- Creating a snapshot on its own didn't consume more metadata
- Creating new LVs consumed a tiny amount of metadata
- Every 1GB written resulted in ~3.3% metadata growth. I assume this is 8MB x 0.033 = approx 270KB. With 64KB per chunk that would be ~17 bytes per chunk. Which sounds reasonable.
Q4. So is metadata growth mainly just due to writes and mapping physical blocks to the addresses used in the LVs?
Q5. I reached max capacity of the pool and only used 27% of the metadata space. When would I ever run out of metadata?
And I think the final Q is, when creating the thin pool, should I use less than 100% of the space in the volume group? Like save 2% for some reason?
Any tips appreciated as I try to wrap my head around this!
