*

Offline Parsifal

  • Administrator
  • *****
  • Posts: 5494
  • Professional computer somebody
    • View Profile
We have our first cloud-hosted subdomain!
« on: December 07, 2013, 09:11:44 AM »
For those who have been keeping up in S&C, I've been researching different CDN providers to reduce the dependency of the various tfes.org services on my VPS. I've just moved the least critical of these services into this new cloud thing you kids keep talking about these days.

The Flat Earth Library is now served by CloudFlare, a global CDN which caches web content to provide high-speed page loads around the world. The backing store for this is an Amazon S3 bucket here in Sydney, Australia, which provides high availability for these files at low cost.

The directory indices look very similar to the output from nginx's autoindex directive. That's because I like nginx's indexing format, and I wrote about 30 lines of horrible shell script in a Makefile to emulate it.

I'd appreciate it if any volunteer guinea pigs could try using the new library, and let me know if it performs reasonably well for you, or if you find any weird problems.
when you try to mock anyone while also running the flat earth society. Lol

Thork

Re: We have our first cloud-hosted subdomain!
« Reply #1 on: December 07, 2013, 02:45:06 PM »
I don't know what this means, but I appreciate it. Whatever it is. Because I know its for my benefit, even if I don't know how.


Ok, Mr Parsifal, I have what I suspect is a related question, but it might not be because I don't understand the OP.


Will you, have you, won't you, consider hosting images on this website? I ask for two reasons.
1) We all post billions of them.
2) If they are on your server, there is limited risk of privacy issues to the users here.
3) It adds to the overall size and content of this site making it bigger and regularly updated as far as google is concerned.

So basically I'm suggesting an area of the site like tinypic where we can upload images. I don't know how you can store that (obviously have file size limits), captchas etc to prevent abuse. If we each had a folder linked to our account, it would actually serve as an archive for us. So if I post a flat earth map, I can find the b*st*rd again next time easily. I suspect you could then serve such pictures to the forums at stupid efiiciency.

But mostly this is about FES owning content and growing.

*

Offline Parsifal

  • Administrator
  • *****
  • Posts: 5494
  • Professional computer somebody
    • View Profile
Re: We have our first cloud-hosted subdomain!
« Reply #2 on: December 07, 2013, 02:59:51 PM »
Will you, have you, won't you, consider hosting images on this website?

Yes. However, it's not a priority at the moment.

I suspect you could then serve such pictures to the forums at stupid efiiciency.

Like you wouldn't believe.
when you try to mock anyone while also running the flat earth society. Lol

Thork

Re: We have our first cloud-hosted subdomain!
« Reply #3 on: December 07, 2013, 03:05:44 PM »
Your geekishness is going to make this one of the most over engineered websites the world has ever known. Ironic for the Flat Earth Society, really. We are supposed to be backward.

Yes. However, it's not a priority at the moment.
Would you consider a construction thread? Bascially one where you guys tell us what you are doing and the timeframe you expect it to take. Just a sticky with an OP of objectives and estimates.

Its not like before where we know barely anything is happening. More just of interest so we can see what is coming and what has just been done. The kind of transparency we never got from Daniel et al.

Re: We have our first cloud-hosted subdomain!
« Reply #4 on: December 07, 2013, 03:29:26 PM »
I just tested it here in NY and each item downloaded at 1.3-1.5mb/s which is almost as fast as steam games download.  So I'd call it a success.


So if I understand correctly, this cloud-host basically takes the contents of your library.tfes.org and caches it.  Then, when someone requests it, that host sends it to them.  And since said host is bigger and more powerful than your VPS, it can send them with greater speed while also allowing your VPS to not have to worry about it.  Like say if someone wrote a script to download the entire library 1,000 times the VPS wouldn't notice a damn thing.

*

Online junker

  • Planar Moderator
  • *****
  • Posts: 8611
    • View Profile
Re: We have our first cloud-hosted subdomain!
« Reply #5 on: December 07, 2013, 03:49:16 PM »
I am getting 40Mb-50Mb per second downloads.  Latency varies wildly.  18ms on the low side, 100+ on the high side.
Wait, is Thork gay or does he just have a thing for lipstick?

*

Offline Tom Bishop

  • Zetetic Council Member
  • **
  • Posts: 6407
  • Flat Earth Believer
    • View Profile
Re: We have our first cloud-hosted subdomain!
« Reply #6 on: December 08, 2013, 04:13:17 AM »
What is our hosting limit?

*

Offline Parsifal

  • Administrator
  • *****
  • Posts: 5494
  • Professional computer somebody
    • View Profile
Re: We have our first cloud-hosted subdomain!
« Reply #7 on: December 08, 2013, 06:36:35 AM »
What is our hosting limit?

What do you mean by "hosting limit"? Data transfer, storage, something else? Do you mean in general or specifically for the library?
when you try to mock anyone while also running the flat earth society. Lol

*

Offline Parsifal

  • Administrator
  • *****
  • Posts: 5494
  • Professional computer somebody
    • View Profile
Re: We have our first cloud-hosted subdomain!
« Reply #8 on: December 08, 2013, 06:40:45 AM »
Incidentally, I've turned off CloudFlare for the library for the time being, meaning it will be served directly from Amazon S3. I'm being cautious about violating section 10 of their terms of use in doing this.

It shouldn't be a big deal, since the library is probably going to be infrequenty accessed anyway. I'll still use CloudFlare for the main website, and (eventually) the forum's static assets.
when you try to mock anyone while also running the flat earth society. Lol

*

Offline Tom Bishop

  • Zetetic Council Member
  • **
  • Posts: 6407
  • Flat Earth Believer
    • View Profile
Re: We have our first cloud-hosted subdomain!
« Reply #9 on: December 08, 2013, 08:37:53 AM »
What is our hosting limit?

What do you mean by "hosting limit"? Data transfer, storage, something else?

Yes.

Specifically, can we upload large feature-length films there pertaining to the Flat Earth and NASA Hoax Documentaries?

Quote
Do you mean in general or specifically for the library?

Library
« Last Edit: December 08, 2013, 08:50:20 AM by Tom Bishop »

*

Offline Parsifal

  • Administrator
  • *****
  • Posts: 5494
  • Professional computer somebody
    • View Profile
Re: We have our first cloud-hosted subdomain!
« Reply #10 on: December 08, 2013, 09:24:31 AM »
Specifically, can we upload large feature-length films there pertaining to the Flat Earth and NASA Hoax Documentaries?

S3 has no limits, you just get charged for how much you use. So yes, you can, but it will start to cost more.


Also, I've realised that the reason CloudFront is slow for me is because my local CloudFront node is apparently in Japan. I've enabled CloudFront for the library, since that specific use case violates CloudFlare's terms of use, but I'll still use CloudFlare for the main website since it sucks less.

And yes, these names are confusingly similar.
when you try to mock anyone while also running the flat earth society. Lol

Thork

Re: We have our first cloud-hosted subdomain!
« Reply #11 on: December 08, 2013, 11:46:47 AM »
Specifically, can we upload large feature-length films there pertaining to the Flat Earth and NASA Hoax Documentaries?
Maybe would be better having our own youTube channel and then embedding the videos? This gives us a presence on yet another massive social media platform.

If you have such documentaries already, would you care to set us up with a page Tom? We can provide assets to decorate the page at your request.


Edit:
I have just remembered that we already have a shiny new Google+ account. Perhaps Secret User would consider allowing Tom access to populate a connected youtube page?
https://plus.google.com/101206158176997811430/posts
« Last Edit: December 08, 2013, 12:08:00 PM by Thork »

*

Online junker

  • Planar Moderator
  • *****
  • Posts: 8611
    • View Profile
Re: We have our first cloud-hosted subdomain!
« Reply #12 on: December 08, 2013, 05:00:00 PM »
I haven't entrusted SU with Google+ (remember, she is a girl...) just yet, but I am happy to give Tom the login if he wants it.
Wait, is Thork gay or does he just have a thing for lipstick?

*

Offline Tom Bishop

  • Zetetic Council Member
  • **
  • Posts: 6407
  • Flat Earth Believer
    • View Profile
Re: We have our first cloud-hosted subdomain!
« Reply #13 on: December 09, 2013, 02:09:22 AM »
Whenever I post files somewhere, they seem to eventually disappear from the internet after a time. For example, blip.tv deleted the 'How they did it' videos discussing the moon landing hoax. Google Video deleted the 'In Search of the Edge' by Bullfrog Films and 'Flat Earth' by Pantheon Movies.

I have In Search of the Edge on DVD, but some of the other videos above may be irrevocably lost.
« Last Edit: December 09, 2013, 02:44:53 AM by Tom Bishop »

*

Offline Parsifal

  • Administrator
  • *****
  • Posts: 5494
  • Professional computer somebody
    • View Profile
Re: We have our first cloud-hosted subdomain!
« Reply #14 on: December 09, 2013, 02:21:50 AM »
Tom, I can arrange for you to have upload access to the library. I can't promise when, but once everything is settled with the homepage it will be feasible for anyone who wants to upload large files to have access to the library S3 bucket. I want to keep backups of everything in S3 outside of Amazon's sphere of influence for cross-provider redundancy as well, so you can rest assured your uploads won't disappear short of the Internet going up in flames. We already have regular backups being taken of the forum and the wiki.

I'll let you know when this is ready to go forward.
when you try to mock anyone while also running the flat earth society. Lol