meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
zfs [2015/03/24 09:10] – niziaklinux:fs:zfs [2021/05/10 20:42] (current) – niziak
Line 1: Line 1:
 ====== ZFS ====== ====== ZFS ======
  
-===== Creating ZFS dataset =====+[[https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/|ZFS 101—Understanding ZFS storage and performance]] 
 + 
 +[[https://lists.debian.org/debian-user/2012/05/msg01026.html]] 
 + 
 +Features: 
 +  * data pools (tanks) are abstraction aggregate block devices (simple, mirror, raidz, spares, etc) 
 +  * data set is created on data pool or another (parent) data set. 
 +  * whole data pool space is shared between dataset (no fixed partition size problem). Size of data set (and its descendants) can be limited using quota 
 +  * compression 
 +  * block level deduplication (not usable for emails with attachment, where attachment are shifted to different offset) 
 + 
 +OpenZFS2.0.0 (Dec 20) [[https://github.com/openzfs/zfs/releases/tag/zfs-2.0.0]]: 
 +  * Sequential resilver (rebuild only used by data portions) 
 +  * Persistent L2ARC cache (survives between reboots) 
 +  * ZSTD 
 +  * Redacted replication (replicate with some data excluded) 
 +  * FreeBSD and Linux unification 
 + 
 +Proposed use case: 
 +POOL created on encrypted LUKS block device.
  
 <code> <code>
 + POOL
 +   |-- /filer (quota)
 +   |     |- foto 
 +   |     |- mp3 (dedup)
 +   |     |- movies
 +   |     +- backup (copies=2, compression)
 +   |
 +   |-- /home (compression, dedup, quota)
 +   +-- /var (quota)
 +         +- log (compression)
 +</code>
 +
 +===== ZFS implementations =====
 +
 +ZFS-Fuse 0.7 is using old pool version 23, where [[http://zfsonlinux.org|ZFSonLinux]] is using pool version 28.
 +[[http://exitcode.de/?p=106|zfs-fuse vs. zfsonlinux]]
 +
 +
 +
 +===== Creating ZFS dataset =====
 +
 +<code bash>
 zpool create INBOX /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3 zpool create INBOX /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3
 </code> </code>
  
-<code>+<code bash>
 # zpool list # zpool list
 NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
Line 13: Line 54:
 </code> </code>
  
-<code>+<code bash>
 # zpool status # zpool status
   pool: INBOX   pool: INBOX
Line 32: Line 73:
 Dataset "INBOX" is also automatically created based on zpool name "INBOX". It is mounted as /INBOX Dataset "INBOX" is also automatically created based on zpool name "INBOX". It is mounted as /INBOX
  
-<code>+<code bash>
 # zfs list # zfs list
 NAME    USED  AVAIL  REFER  MOUNTPOINT NAME    USED  AVAIL  REFER  MOUNTPOINT
Line 61: Line 102:
 zfs set dedup=on INBOX zfs set dedup=on INBOX
 </code> </code>
 +New attributed applies only to newly written data.
  
 Tests Tests
Line 134: Line 176:
 Only beginning of both files |0|B1| was deduplicated (16MB saved) Only beginning of both files |0|B1| was deduplicated (16MB saved)
  
-By default ZFS provider block level deduplication based on checksums wchich we got almost for free. +ZFS provides block level deduplication based on block checksums which we got almost for free.
-To perform full byte comparison, to catch shifted data like e-mail attachment: +
-<code> +
-zfs dedup=verify INBOX +
-</code>+
  
  
Line 148: Line 186:
 zfs set compression=on INBOX zfs set compression=on INBOX
 </code> </code>
 +Possible parameters for compression=on | off | lzjb | gzip | gzip-[1-9] | zle
 +New attributed applies only to newly written data. For test data I was using Maildir with some huge e-mails.
  
-But new attributed applies only to newly written data.+^compression ^ logical size  ^ physical size^ ratio |  
 +|off         |   702 MB      | 703 MB       | 1.0   | 
 +|on = lzjb   |   702 MB      | 531 MB       | 1.32  | 
 +|gzip-1      |   702 MB      | 374 MB       | 1.87  | 
 +|gzip=gzip-6 |   702 MB      | 359 MB       | 1.95  | 
 +|gzip-9      |   702 MB      | 353 MB       | 1.96  | 
 +|-- 
 +|squashfs    |               | 365 MB       |       |
  
-<code> 
-NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT 
-INBOX   975M   724M   251M    74%  1.00x  ONLINE  - 
-</code> 
- 
-The same data copied again to dataset after compression enabled 
- 
-<code> 
-NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT 
-INBOX   975M   563M   412M    57%  1.00x  ONLINE  - 
-</code> 
  
 zdb -S INBOX zdb -S INBOX
- 
 zdb -b INBOX zdb -b INBOX
  
-====== Tests: ====== 
  
-Filesystem was tested with 648MiB of e-mail stored in Maildir format (lots of binary attachment encoded as BASE64). 
  
-SquashFS=365MB vs ZFS=563MB +<code bash> 
- +zfs get compressratio 
-Deduplication: +</code>
- +
-Deduplication on file level works on ZFS and SquashFS (the same folder copied again). +
- +
-Deduplication of 2 different 32MB blobs, with file concatenated from | 0 | blob1 | 0 | blob2 | 0 | blob1 | 0 | blob2 |0| +
- +
-Deduplication on the same attachment inside different email doesn't work in ZFS.+
  
-====== References: ======+===== References: ===== 
 +[[http://docs.oracle.com/cd/E19253-01/819-5461/6n7ht6qu6/index.html]] 
 +[[https://wiki.freebsd.org/ZFSQuickStartGuide]]
  
 [[http://www.funtoo.org/ZFS_Fun|http://www.funtoo.org/ZFS_Fun]] [[http://www.funtoo.org/ZFS_Fun|http://www.funtoo.org/ZFS_Fun]]