meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
linux:fs:zfs:dedup [2021/02/17 08:07] – niziaklinux:fs:zfs:dedup [2021/07/05 12:58] (current) – niziak
Line 2: Line 2:
  
 For deduplication, it is recommended to have L2ARC cache size of 2-5GB per 1TB of disk. For deduplication, it is recommended to have L2ARC cache size of 2-5GB per 1TB of disk.
- 
- 
- 
   * For every TB of pool data, you should expect 5 GB of dedup table data, assuming an average block size of 64K.   * For every TB of pool data, you should expect 5 GB of dedup table data, assuming an average block size of 64K.
   * This means you should plan for at least 20GB of system RAM per TB of pool data, if you want to keep the dedup table in RAM, plus any extra memory for other metadata, plus an extra GB for the OS.   * This means you should plan for at least 20GB of system RAM per TB of pool data, if you want to keep the dedup table in RAM, plus any extra memory for other metadata, plus an extra GB for the OS.
Line 17: Line 14:
  
 Huge CPU usage by over 96 ZFS kernel threads noticed with open-zfs v8.0.6 (ZFS On Linux), when some big parts of data deleted (auto snapshot rotation). It is connected with deduplication enabled and causes system to almost freeze because of high CPU usage! Huge CPU usage by over 96 ZFS kernel threads noticed with open-zfs v8.0.6 (ZFS On Linux), when some big parts of data deleted (auto snapshot rotation). It is connected with deduplication enabled and causes system to almost freeze because of high CPU usage!
 +
 +==== WARNING! ====
 +
 +Issue when deleting large portion of data and deduplication enabled.
 +ZFS driver creates 96 ''z_fr_iss'' threads. Load average of system goes immediatelly to 100 (soft watchdog can be set to reboot system when LA is too high).
 +These threads kills CPU and IO.
 +
 +  * [[https://github.com/openzfs/zfs/issues/6783|Large Deletes & Memory Consumption ]]
 +  * [[https://github.com/openzfs/zfs/issues/118|Reduce ZFS related processes/tasks]]
 +  * [[https://github.com/openzfs/zfs/issues/3976|High CPU usage by "z_fr_iss" after deleting large files]]
 +  * [[https://github.com/openzfs/zfs/issues/4227|rm / remove / delete a large file causes high load and irresponsiveness]]
 +
  
  
Line 31: Line 40:
  
 Once all deduped datasets are destroyed the dedup table will be removed and the performance impact is cleared. Once all deduped datasets are destroyed the dedup table will be removed and the performance impact is cleared.
 +NOTE: after removal data set with dedup enabled, it takes some time until ZFS removes all DDT entries internally.
 +
  
 ===== status ===== ===== status =====