[xasop]

So this is more of a medium-to-long-term thing, but I want to try to build a proper distributed Plan 9 system, with all the services running on different machines, and I've been thinking about what hardware to use.

So far, I'm leaning towards:


File server - HPE ProLiant MicroServer Gen10

• https://www.hpe.com/nl/en/product-catalog/servers/proliant-servers/pip.specifications.hpe-proliant-microserver-gen10.1009955178.html
• Supports hardware RAID 5 with an internal SSD, which is the ideal use case for Plan 9's cwfs.
• I'll probably stuff it full of cheap Seagate Archive disks for 24 TB total capacity, with at least a 120 GB SSD cache.
• I can get a second one running Linux for backups.

Auth server - Raspberry Pi

To be honest, I actually don't know much about the auth server or what resources it needs, but I can't imagine it being too intensive, so a RPi should suffice.


Cpu server - Intel NUC7i7BNH

• https://www.intel.com/content/www/us/en/products/boards-kits/nuc/kits/nuc7i7bnh.html
• Straightforward to run diskless.
• 32 GiB of RAM means plenty of compute capacity.

Terminal - any web browser

This is actually the fun bit. I plan to install 9webdraw, which is an HTML5 implementation of a Plan 9 terminal. As a result, I will be able to use any web browser on any Internet-connected computer anywhere as a terminal for this whole system.

Of course, I'll probably also install a real Plan 9 terminal, most likely on my X201, but having the ability to make any public computer into a terminal for my own OS seems like a neat feature. This will also make it easy to hand out accounts to anyone else who may be interested in the system.


Software is a no-brainer, of course. 9front is the only competently maintained Plan 9 derivative. As a bonus, it has an x86 hypervisor, so I can run OpenBSD in a VM on the cpu server for all my Unix crapware.

[Dr David Thork]

My 2 cents. Don't be filling it with cheap seagate drives. You are making a server with closely packed always on disks. Get NAS disks or you'll regret it down the line.
Western digital red are good. I have those on my NAS server.
https://www.wdc.com/products/internal-storage/wd-red.html


Cache ... I'd be tempted to lose the ssd cache and run optane. You already picked intel NUC ... optane is compatible. The read / write speeds on the 32Gb version are swift and intel's software for caching is very smart.
https://www.intel.com/content/www/us/en/architecture-and-technology/optane-memory.html
If you have more money, the PCI optane drives are insanely fast. They also cost a fair bit more, but they piss all over any m2 ssd.
https://www.intel.co.uk/content/www/uk/en/products/memory-storage/solid-state-drives/gaming-enthusiast-ssds/optane-900p-series/900p-280gb-aic-20nm.html


Whilst you undoubtedly have your heart set on Plan 9, might I suggest you at least look at FreeNAS. Its open source, its free as in beer and speech and most importantly, it has tons of apps so you can get it running straight away such as Plex, itunes, mobile phone back ups, time machine if you have anything apple, all kinds of stuff. You can still hack the crap out of it yourself.

[xasop]

My 2 cents. Don't be filling it with cheap seagate drives. You are making a server with closely packed always on disks. Get NAS disks or you'll regret it down the line.
Western digital red are good. I have those on my NAS server.
https://www.wdc.com/products/internal-storage/wd-red.html

Cache ... I'd be tempted to lose the ssd cache and run optane. You already picked intel NUC ... optane is compatible. The read / write speeds on the 32Gb version are swift and intel's software for caching is very smart.
https://www.intel.com/content/www/us/en/architecture-and-technology/optane-memory.html

You might want to research how Plan 9's cwfs filesystem works before you give advice along these lines. All writes and the vast majority of reads from user programs will only ever hit the SSD. Data is then asynchronously copied to the hard drives so that the SSD doesn't fill up. The performance of the hard drives is almost irrelevant, and having an SSD is what allows cheap, slow drives to be used.

Also, the NUC will not have any storage in it at all, since it isn't the file server.

Further reading: http://man.9front.org/4/cwfs

Whilst you undoubtedly have your heart set on Plan 9, might I suggest you at least look at FreeNAS. Its open source, if free as in beer and speech and most importantly, it has tons of apps so you can get it running straight away such as Plex, itunes, mobile phone back ups, time machine if you have anything apple, all kinds of stuff. You can still hack the crap out of it yourself.

FreeNAS is based on FreeBSD, which ranks fairly low on my list of open-source Unix systems I would consider running for any purpose, just above MINIX and Hurd. Trust me, I have considered running a Linux file server quite thoroughly, and while the idea does have some appeal, Plan 9 is a lot simpler to configure and maintain in the long run.

Also, you may have missed that there will be more than one computer involved in this build, since you seem to be confusing the compute server with the file server. Plan 9 is not just a NAS.

[Dr David Thork]

Can I ask what you want this for?

I use my NAS as a mirrored (raid 1) back up server for my business, I stream videos from it using Plex and all my devices run back ups to it. I also use it as it has a wordpress plug in so I can mount client websites to it during development so that my slaves can ftp into it or mount on their filesystems. Its a really simple system, with low set up investment from a time perspective. Basically a personal cloud.

What do you intend to do with it that needs all these extra layers for?


PS- Red NAS drives aren't fast. They are reliable. You are paying for reliability, not performance.

Also, the NUC will not have any storage in it at all, since it isn't the file server.

My bad. I was thinking about my own requirements where I can access my NAS from anywhere including a coffee shop, and the limiting factor is wifi speed. Not drive speed. If your NUC is wired, no problem.

[xasop]

Can I ask what you want this for?

In the long term, as a complete replacement for OpenBSD for my daily computing (except for crapware that will likely never run on Plan 9, like modern web browsers). In the medium term, as a complement to my existing computing which provides for easy remote access to storage and compute resources.

it has a wordpress plug in

Thanks for convincing me never to use it. Also, you can mount directories from Unix servers on Plan 9 as well using sshfs, except that works regardless of whether you're using Wordpress or not.

What do you intend to do with it that needs all these extra layers for?

It doesn't have "extra layers". It's a distributed operating system, that is to say, one OS consisting of multiple computers communicating with each other. It does not, in principle, do anything any other OS cannot do. It just does it better and simpler.

PS- Red NAS drives aren't fast. They are reliable. You are paying for reliability, not performance.

The hard drives will be inactive most of the time. Reliability also isn't a very important concern.

My bad. I was thinking about my own requirements where I can access my NAS from anywhere including a coffee shop, and the limiting factor is wifi speed. Not drive speed. So I put optane in my NUC for example and when I go to my girlfriend's house and develop a bit there, I get the cache with me and use it with her wifi.

This sounds backwards. The way Plan 9 works is that if you need rapid access to the storage, you log into the compute server (which would be the NUC) which is located on the same local network as the file server (which would be the MicroServer). Because all IPC in Plan 9 uses the network-transparent 9p protocol, there are no disadvantages to this (apart from the obvious latency imposed by the network). You can still access all the devices on your laptop even though the programs are running on a server at home, because device access on Plan 9 does not assume that the device is local.

This is also how using a web browser as a terminal works. Instead of running Plan 9 programs locally, the HTML5 application presents the browser window to the Plan 9 compute server as a terminal device (or more precisely, a set of input and display devices). You can think of it as being like Remote Desktop for Windows, except that instead of being a separate protocol that you need to configure, it just falls naturally out of the OS's design.

My NUC is just a desktop lite for travel.

I use a laptop for that, but whatever floats your boat.

[Dr David Thork]

I can see the point of this if you have employees and you want them all on the same locked-down OS, but for personal use?

Surely distributed storage is all you need?

I think this is overkill, but its your hobby so knock yourself out. 


Also the bit you think I have backwards? Most of the devices I use connect via wifi. Again, that being the bottleneck. Caching serverside doesn't speed that up. So I put the cache in the device (my desktop and NUC). Everything else just runs at the speed of the NAS over wifi ... which is ok for a mobile phone or an iPad. I can stream video to my mobile for example and the NAS/wifi is OK to keep up. Its fine for consumer requirements.

[xasop]

I can see the point of this if you have employees and you want them all on the same locked-down OS, but for personal use?

Surely distributed storage is all you need?

"Distributed storage" is another matter entirely. In both your suggestion and in Plan 9, storage is centralised. What is distributed is the resource access model, where any resource on any computer can be used from any other computer as if it were local. That is not the same thing as an individual resource being distributed.

And no, having a server that you can run programs on rather than just access files is very useful. I'm surprised this needs to be said.

Also the bit you think I have backwards? Most of the devices I use connect via wifi. Again, that being the bottleneck. Caching serverside doesn't speed that up. So I put the cache in the device (my desktop and NUC). Everything else just runs at the speed of the NAS over wifi ... which is ok for a mobile phone or an iPad. I can stream video to my mobile for example and the NAS/wifi is OK to keep up. Its fine for consumer requirements.

You don't seem to have understood correctly. I didn't say anything at all about "caching serverside", I was talking about running your programs on the server itself. You don't need to worry about transferring large files over WiFi if your web browser and your e-mail client are physically running on the server. All that needs to be transferred over WiFi is the contents of the application's window, and your input.

[Dr David Thork]

I'm still non the wiser what you need such a system for.

[xasop]

I'm still non the wiser what you need such a system for.

I don't need it. My current computing environment is more than sufficient for what I need. I am simply interested in trying to make things better and more efficient.

[Dr David Thork]

I don't need it. My current computing environment is more than sufficient for what I need. I am simply interested in trying to make things better and more efficient.

I'll rephrase. I still don't know what you want it for? Like an example thing you want to do, that this set up and not something simpler would achieve. Now I'm just thinking illegal porn server, crypto exchange, illegal torrent server ... nothing good. This is your fault for being secretive.

[xasop]

I'll rephrase. I still don't know what you want it for? Like an example thing you want to do, that this set up and not something simpler would achieve. Now I'm just thinking illegal porn server, crypto exchange, illegal torrent server ... nothing good. This is your fault for being secretive.

Your lack of imagination does not make me secretive.

An example thing I would do is walk up to any computer at uni when I return to study, fire up a web browser and log in remotely to my home PC. No need to carry a laptop everywhere.

Or, I could fire up my laptop on a flight between here and Sydney to search my old e-mails remotely without needing to download my entire inbox over in-flight WiFi.

One thing I am looking forward to not doing is maintaining more than one OS installation ever again. Every Plan 9 system uses the same filesystem, the one on the file server, so installing updates on any other computer won't even be a thing anymore.

Along that vein, another thing is that the system becomes trivial to extend. If I move into a bigger house someday and I want a computer upstairs and one downstairs, I just buy some cheap shit with video out and a second monitor, plug it in, network boot from the file server and now I have two interfaces to the same OS.

Also, you are still talking as though this is a complex setup. It is not. As I already explained, Plan 9 is a distributed OS. It is no more complicated to install on multiple computers than it is to install on a single one. The various system services do not know whether they are running together on the same host, or separated by 1000 km of optical fibre. That is the whole point of it; to provide a simple and generic way to network computers, instead of the clunky and ad hoc solutions in other OSes.

[Dr David Thork]

Yeah, no. The bit i don't understand is why you have to have a highly controlled OS for a one size fits all solution. On your laptop, run Chrome OS or Windows. Its a laptop. You are on a plane. What is the problem. You can watch a video or read your e-mails on any OS.

Thin client for your living room ... My smart TV runs from ... my NAS and runs off the plex server. So would my playstation if I owned one. The TV has a browser and can access all the files I'd need on it. I don't want to program in my living room. I'm only interested in content consumption in that room, not content creation.

Maintaining an OS ... again for content consumption most OS like Windows, MacOs, iOS, Android, Chrome OS etc have this thing called automatic update. They maintain themselves. Sure, sometimes something small goes awry, but it rarely stops you doing the basic things you do on such devices such as browse, listen to music and watch youtube.

Mailbox ... again you can get your mails from a regular NAS or cache out the last 50 or so like your mobile phone does.


I'm just not seeing this. Unless you are a business with lots of power users needing access for all devices, this isn't necessary. You'll always do 99% of your power user work on your desktop with the big monitor, your comfy computer chair and your expensive mouse mat. The rest of the time you are using a computer like joe public and just pushing a mouse about the desk or liberally swiping at a glossy screen.

Just say ... its my hobby and I'm doing it because I can. I'll be happy with that.

[xasop]

Yeah, no. The bit i don't understand is why you have to have a highly controlled OS for a one size fits all solution.

"Highly controlled OS"? What? Where did you get that from?

On your laptop, run Chrome OS or Windows. Its a laptop. You are on a plane. What is the problem. You can watch a video or read your e-mails on any OS.

Are you volunteering to pay my €10000 in-flight WiFi bill for all the data I download from my NAS?

Thin client for your living room ... My smart TV runs from ... my NAS and runs off the plex server. So would my playstation if I owned one. The TV has a browser and can access all the files I'd need on it. I don't want to program in my living room. I'm only interested in content consumption in that room, not content creation.

That's great. I am not you.

Maintaining an OS ... again for content consumption most OS like Windows, MacOs, iOS, Android, Chrome OS etc have this thing called automatic update. They maintain themselves. Sure, sometimes something small goes awry, but it rarely stops you doing the basic things you do on such devices such as browse, listen to music and watch youtube.

Again, I am not you. But instead of asking me what I find irksome about maintaining my OS, you decided to lecture me on how easy it is to update yours.

Mailbox ... again you can get your mails from a regular NAS or cache out the last 50 or so like your mobile phone does.

This is sounding more and more like an offer to pay for data usage.

I'm just not seeing this.

Fortunately for me, you don't need to see why it's useful in order for it to be useful.

Unless you are a business with lots of power users needing access for all devices, this isn't necessary. You'll always do 99% of your power user work on your desktop with the big monitor, your comfy computer chair and your expensive mouse mat. The rest of the time you are using a computer like joe public and just pushing a mouse about the desk or liberally swiping at a glossy screen.

Thanks for explaining to me how I use a computer, Thork.

Just say ... its my hobby and I'm doing it because I can. I'll be happy with that.

That's not the whole reason. Part of it is to learn about Plan 9, because it is interesting. But I wouldn't spend the money on building a distributed computing environment only for that.

[Dr David Thork]

No further questions, your honour.