Selfhosted

50253 readers

586 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago

MODERATORS

HybridSarcasm@lemmy.world

HybridSarcasm@lemmy.hybridsarcasm.xyz

Apparent issues with ZFS on RPi 5 (lemmy.world)

submitted 3 days ago* (last edited 3 days ago) by ramenshaman@lemmy.world to c/selfhosted@lemmy.world

27 comments fedilink hide all child comments

I tried to find a more applicable community to post this to but didn't find anything.

I recently set up a NAS/server on a Raspberry Pi 5 running Raspberry Pi OS (see my last post) and since then I've got everything installed into a 3D printed enclosure and I've got RAID set up (ZFS RAIDz1). Prior to setting up RAID, I could transfer files to/from the NAS at around 200MB/s, but now that RAID is seemingly working things are transferring at around 28-30 MB/s. I did a couple searches and found someone suggesting to disable sync ($ sudo zfs set sync=disabled zfspool). I tried that and it doesn't seem to have had any effect. Any suggestions are welcome but keep in mind that I barely know what I'm doing.

Edit: When I look at the SATA hat, the LEDs indicate that the drives are being written to for less than half a second and then there's a break of about 4 seconds where there's no writing going on.

you are viewing a single comment's thread
view the rest of the comments

[–] calamityjanitor@lemmy.world 10 points 3 days ago (3 children)

sync=disabled will make ZFS write to disk every 5 seconds instead of when software demands it, which maybe explains your LED behavior.

Jeff Geerling found that writes with Z1 was 74 MB/sec using the Radxa Penta SATA HAT with SSDs. Any HDD should be that fast, the SATA hat is likely the bottleneck.

Are you performing writes locally, or over smb?

Can try iostat or zpool iostat to monitor drive writes and latencies, might give a clue.

How much RAM does the Pi 5 have?

[–] calamityjanitor@lemmy.world 5 points 3 days ago* (last edited 3 days ago)

After some googling:

Some Linux distributions (at least Debian, Ubuntu) enable init_on_alloc option as security precaution by default. This option can help to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.

Unfortunately, it can lower ARC throughput considerably (see bug).

If you’re ready to cope with these security risks 6, you may disable it by setting init_on_alloc=0 in the GRUB kernel boot parameters.

I think it's set to 1 on Raspberry Pi OS, you set it in /boot/cmdline.txtI think.

Exhaustive ZFS performance tuning guide

[–] ramenshaman@lemmy.world 1 points 3 days ago* (last edited 3 days ago) (2 children)

I haven't seen more than maybe 32MB/s. The transfers I've done are all on my local network from my desktop to my NAS which is plugged into my router. I have samba installed, my NAS shows up as a network drive just fine and I've just been dragging/dropping on my desktop GUI. Before setting up RAID, when I did this I would get around 200MB/s. The Pi has 16GB of RAM and I'm using less than 1.5GB while making a large transfer from my desktop to the NAS.

[–] Lemmchen@feddit.org 3 points 3 days ago

Those 200MB/s probably weren't synchronous transfers. The OS tells you the write was complete, but it actually hasn't committed the data to disk yet. (Wild guess)

[–] calamityjanitor@lemmy.world 2 points 3 days ago* (last edited 3 days ago)

Ah kay, definitely not a RAM size problem then.

iostat -x 5 Will print out per drive stats every 5 seconds. The first output is an average since boot. Check all of the drives have similar values while performing a write. Might be one drive is having problems and slows everything down, hopefully unlikely if they are brand new drives.

zpool iostat -w Will print out a latency histogram. Check if any have a lot above 1s and if it's in the disk or sync queues. Here's mine with 4 HDDs in z1 working fairly happily for comparison:

Here's mine with 4 HDDs in z1 working fairly happily for comparison

The init_on_alloc=0 kernel flag I mentioned below might still be worth trying.