Fosshost domains down?

I currently can’t reach www.octave.org, hg.octave.org, wiki.octave.org, or buildbot.octave.org.
Is this just me? Or are the Fosshost domains having issues currently?

Edit: The main website can temporarily be reached via: GNU Octave (gnu-octave.github.io)

I can’t load any of our Fosshost domains either.

Fosshost - Network Status says that the physical server is down. It’s called pve01-chi1. Our Octave servers run as a virtual machine on that host.

I’ll tag @peer and @sr229 who are the Fosshost volunteers who helped us get set up. Let’s hope it’s temporary downtime.

2 Likes

I was able to ping www.octave.org and also login to fosshost.octave.org using ssh. Once logged in, top showed the load average to be 180+. There were a lot of cron and apache jobs running. I don’t know what was causing that. I tried rebooting. I was disconnected. The system still responds to ping, but now I can’t connect with ssh. Is there another way to force a restart of the system?

FYI: Fosshost Support

From my end there is also no chance to connect to FossHost.

@arungiridhar, can you do anything from the hypervisor? In parallel I create a ticket as @dasergatskov suggested.

The hypervisor kept giving me “incorrect login” messages after waiting some 8-10 seconds. The network status page says the host is down though, so it might not have a server to go to after authentication.

Just in case this goes on for a few days, it’s best to keep the site backups handy in case we need to restore anything from there.

I received bad news today within the FossHost ticket:

The hypervisor seems to be in a state where I cannot even login to it or better yet, I can in theory login, but it will not give me a shell.

Contacted the sponsor regarding it.

More than 24h downtime is barely acceptable. Either Fosshost can sponsor a backup instance on a different sponsored-node, or I try to ask NumFocus (Octave’s new sponsor?!) for a paid solution back on DreanHost, DigitalOcean, … Maybe we can discuss this tonight in the meeting?

That’s bad news indeed. Speaking for myself, I support your asking NumFocus about hosting mechanisms. Have you had fast response time from them?

I don’t know if the current incident can be compared with previous ones.
Anyway: Another server downtime took about a week to get resolved:
pve02-lon2 status page (updown.io) (Jul 9 - Jul 16, 2022)

Is there an alternative location for the Mercurial repo (other than hg.octave.org/octave) which I can use while octave.org is down?

Afaict, the repository on Fosshost is (was) a mirror of the repository on savannah:
https://hg.savannah.gnu.org/hgweb/octave/

2 Likes

I’d be glad to set up one or more VMs on digital ocean under my account. You can see options here: Droplet Pricing | DigitalOcean.

We have the following services on fosshost now:

We could host all of these on a single digital ocean VM (as we have things set up now on fosshost) or split it up to avoid having problems with any one service causing all to be unavailable. We could group the simple static pages and redirects on one system (what could go wrong?) and the things that require more resources (the wiki, maybe the mercurial archive or the buildbot server) on separate systems.

I’m not using the digital ocean account for anything else, so I would have no problem sharing access to the control panel with a few other Octave developers. With that, maybe we could avoid extended downtime in the future.

Adding to that list: hg.octave.org is also (the only?) host of the repository for MXE Octave, not just a mirror of the Octave repository on savannah.

2 Likes

We also probably do not need to put all eggs in one basket. E.g. there are few options for free
wiki hosting (miraheze, Ourproject.org, gitlab? ). I do not have any experience with them.
It looks like we need a provider with a shell access for buildbot master.

Reposting from octave-maintainers@gnu.org:

*************** Start ***************
bill-auger via RT webmasters-comment@gnu.org
9:06 PM (1 hour ago)

Reply to all

to octave-maintainers

i should have clarified, that this thread is related to a ticket received by GNU webmasters - please CC the GNU webmasters (webmasters@gnu.org), if there is anything for us to do - the ticket is remaining open for now

Dmitri wrote:

See: Fosshost domains down?

based on that forum thread, it appears that the host has not responded, and so it could be some time before octave’s infra is operational once again - at the very least, we would like to see the gnu.org URL (http://www.gnu.org/software/octave/), directing users to something interesting

if hosting is a problem, you could consider adjusting the DNS records, and utilizing the GNU infra, at least temporarily, and/or as a permanent backup plan to ease with such unexpected emergencies in the future

off-hand, at least these appear to be relevant:

www.octave.org - mostly (all?) static web pages
docs.octave.org - all static pages?

  • savannah could host a copy of the main website
  • a simple static web page could be hosted in the GNU website VCS

bugs.octave.org - static page(s) or redirect?

  • the savannah bug tracker is already active and is heavily used

hg.octave.org - mirror of savannah mercurial archive

  • savannah already has the source code VCS

buildbot.octave.org - buildbot master

  • i beleive that GNU fencepost could accommodate the project’s needs

ftp.octave.org - redirect to gnu.org

  • that’s an easy one (just change the DNS)

*************** End ***************

Is it possible to view that support ticket to keep track of any development?

There was only a single reply from Tuesday… not much progress there. However, I was not yet given permission to make the link to the ticket public, but I sent the link to you by PM.

If anyone else is interested, please write me a PM.

1 Like

@jwe, @arungiridhar, and I prepare to move to Digitial Ocean this weekend :sweat:

Do we have records for the amount of traffic going (GB per month) on to the static web pages (www.octave.org and docs.octave.org)?

No records that I know of, but when I sampled network traffic speed for all the Fosshost-based *.octave.org sites in the past, it was 10 to 20 GB per day outgoing. This was docs + wiki + everything else. Why do you ask btw?