ISPStatus

DreamHost - luanda filesystem check fun!

luanda was rebooted earlier this today and the fielsystem came back read-only. So, unmounted and did a regular filesystem check and all was happy, till we rebooted and the filesystem started throwing up erros again. So now we’re a thorough filesystem check to sort this whole mess out. All the user data is intact so no need to worry there (and of course, there is the nightly backups too).

This filesystem check will likely take a few hours to complete and is dependent on a number of factors that’ll determine how long the filesystem check will run. PETA has been going after us after word leaked out that we were using entrails to determine fsck times for these posts, so we can’t be any more accurate or inaccurate than that for an ETA.

This affects less that 1% of our shared web hosting customers, so please check your web panel to figure out what server your on. For any other servers, please create a support ticket.

Ebay - Scheduled Maintenance For This Week

The eBay system will be undergoing general maintenance from approximately 23:00 PT (11pm) on Thursday, September 9th to 01:00 PT (1am) on Friday, September 10th. During this maintenance period, certai...

Zen - #1940 Routing & Core Network - New BTW Ethernet London Bearers (Update)

Start: Fri, 10th Sep 2010 13:00
End: Fri, 10th Sep 2010 14:30
Update: Fri, 10th Sep 2010 14:30
Edited: Wed, 8th Sep 2010 15:08
Status: Up
Maintenance: Planned

Zen - #1941 Voice - Continued issues with inbound voip (New)

The problems detected yesterday affecting customers who have VOIP from ourselves have returned. This problem would only affect larger customers who would have a lot of users on inbound calls only.

This issue comes from a problem with one of our third party suppliers and we are currently working with them for a resolution.

Further updates will be posted here when available.

Zen regret any inconvenience this may cause.

Start: Wed, 8th Sep 2010 14:43
Update: Wed, 8th Sep 2010 16:45
Edited: Wed, 8th Sep 2010 14:44
Status: Partial
Maintenance: Emergency

Zen - #1940 Routing & Core Network - New BTW Ethernet London Bearers (New)

Maintenance work is being performed on our London Metro ring used to service DSL customers on the BT 21CN platform.

No disruption to service is expected, this service alert is being raised as a precautionary measure.

Start: Fri, 10th Sep 2010 13:00
End: Fri, 10th Sep 2010 14:30
Update: Wed, 8th Sep 2010 14:30
Edited: Wed, 8th Sep 2010 14:40
Status: Up
Maintenance: Planned

Zen - #1939 Shared Linux Hosting - rscp01.hosting.zen.net.uk server maintenance. (New)

Our engineers have identified a potential service impacting issue on the server and as such we need to perform emergency maintenance.

The server will be taken off-line for a short period whilst software patches are applied to the server.

Customer will loose access to all services on the server 5-10 minutes whilst it is restarted.

Zen regret any inconvenience this may cause.

Start: Thu, 9th Sep 2010 00:01
End: Thu, 9th Sep 2010 01:00
Update: Thu, 9th Sep 2010 01:00
Edited: Wed, 8th Sep 2010 14:07
Status: Outage
Maintenance: Emergency

madasafish - Access to Care4Free websites and email (63356) - RESOLVED

This is an update to yesterday's post regarding Care4Free customers' access to our email and web servers. Our domain registrar has resolved the underlying problem and DNS appears to have fully propagated. As such, access to email and websites has been restored for all customers. Please accept our sincere apologies for any inconvenience caused by this problem. Regards, Ben Trimble Customer Support

madasafish - Inbound Email Delays (63388) - RESOLVED

This is an update to this morning's post about delays that may have affected the arrival of incoming email. The underlying problem has been resolved and the mail queues have returned to normal levels. No email has been lost and all messages should now be arriving in a timely fashion. Kind Regards, Bob Pullen Customer Support

madasafish - Inbound Email Delays (63388) - NEW

Our monitoring systems have identified a rise in the volume of messages queued on our inbound mail platform. This could result in delays in affecting the arrival of customers' email to their mailboxes. Our engineers are investigating and we'll provide another update once further information is made available. Apologies for the inconvenience. Kind regards, Bob Pullen Customer Support

Zen - #1930 Broadband (xDSL) - Problems on Basildon exchange (Close)

BT have completed works overnight to restore connections to the remaining customers affected. Any customers continuing to have problems please contact Technical support for further assistance.

Zen regret any inconvenience this may cause.

Start: Fri, 3rd Sep 2010 11:28
Clear: Wed, 8th Sep 2010 09:04
Edited: Wed, 8th Sep 2010 09:04
Status: Up
Maintenance: None

Jumba - Scheduled: SYD1: Network Maintenance [postponed]

Advisory #3387 has been updated. The details are below.

Status: Scheduled, Low Impact, Postponed
Start: -

Update at Tue, Aug 24 2010 11:52 am

During this time we will be performing maintenance on our Sydney network.

This maintenance window will have the following impacts:
- PIPE Sydney connectivity will be interrupted for up to 1 hour.
- Vocus Sydney connectivity will be interrupted for up to 1 hour.

There may be a brief period of packet loss while traffic flowing through the affected links automatically reroutes. No further impact is expected.

This maintenance window also involves bringing on additional upstream carrier circuits however no customer impact is expected when this change is made.

------

Update at Thu, Sep 02 2010 10:15 pm

This maintenance has been postponed. A revised ETA will be posted shortly.

------

Jumba - Scheduled: Updates to vCenter / vSphere service (CBR) [current]

Advisory #3399 has been updated. The details are below.

Status: Scheduled, Low Impact, Current
Start: 08/09/2010 11:00
ETA: 08/09/2010 18:00

Update at Sat, Aug 28 2010 07:39 pm

Customers are advised that due to upgrades on our VMware virtual infrastructure in Canberra, access to vSphere will be limited during this window.
At the completion of the upgrades, customers are advised that the access point for vCenter will change from vc.vde.net.au to the new address of vcenter1.cbr1.aussiehq.net.au.

Additionally, customers using the vSphere client to access their VM's will need to upgrade their client to the newer 4.1 version.
This can be obtained from our mirror:
http://vmware.mirror.aussiehq.net.au/VMware-viclient.exe

If VM actions are required during this time, customers are advised to contact the service desk.

We apologise in advance for the inconvenience caused.

------

Update at Wed, Sep 08 2010 11:33 am

The updates are now in progress.

------

Jumba - Scheduled: SYD1: Network Maintenance [postponed]

Jumba - Scheduled: Updates to vCenter / vSphere service (CBR) [current]

Advisory #3399 has been updated. The details are below.

Status: Scheduled, Low Impact, Current
Start: 08/09/2010 11:00
ETA: 08/09/2010 13:00

Update at Sat, Aug 28 2010 07:39 pm

Customers are advised that due to upgrades on our VMware virtual infrastructure in Canberra, access to vSphere will be limited during this window.
At the completion of the upgrades, customers are advised that the access point for vCenter will change from vc.vde.net.au to the new address of vcenter1.cbr1.aussiehq.net.au.

Additionally, customers using the vSphere client to access their VM's will need to upgrade their client to the newer 4.1 version.
This can be obtained from our mirror:
http://vmware.mirror.aussiehq.net.au/VMware-viclient.exe

If VM actions are required during this time, customers are advised to contact the service desk.

We apologise in advance for the inconvenience caused.

------

Update at Wed, Sep 08 2010 11:33 am

The updates are now in progress.

------

Linode - RESOLVED - Packet loss affecting some Linodes in Fremont

10:06am (EDT): Connectivity has returned to normal and we'll continue to monitor at this time.

10:00am (EDT): We are looking into packet loss affecting some Linodes in the Fremont facility. There is no reason to issue reboots as Linodes are still running.

Linode - RESOLVED - Connectivity issues affecting some Linodes in Newark

8:06am (EDT): Reports indicate that connectivity has returned to normal. We will keep monitoring at this time.

5:13am (EDT): We have received reports of intermittent connectivity to our Newark facility. While our network appears healthy, we are working with the NOC to determine if this is an issue upstream from us.

Linode - COMPLETED - Scheduled Network Maintenance in Newark: August 8, 2010

Saturday, August 8th, 2010

8:20pm (EDT): Maintenance completed. This concludes a major upgrade to our Newark facility's network. Enjoy!

8:00pm (EDT): Maintenance commencing.

Wednesday, August 4th, 2010

4:45pm (EDT): The Newark facility will undergo network maintenance between 8:00pm and 10:00pm EDT, August 8th. We do not expect any downtime during this maintenance window.

Linode - RESOLVED - Network issues in Dallas, Fremont, Atlanta, and Newark

8:59am (EDT): The attack against Atlanta has been mitigated and connectivity has returned to normal. We will continue to monitor all four locations at this time.

8:49am (EDT): The attack against Fremont has been mitigated and connectivity has returned to normal.

8:40am (EDT): The attacks against both Dallas and Newark have been mitigated and connectivity has returned to normal.

8:38am (EDT): This appears to be a malicious DDoS attack against all four Linode facilities. We are currently working on mitigating them.

8:25am (EDT): We are looking into packet loss affecting some Linodes in all four facilities. There is no reason to issue reboots as Linodes are still running.

Linode - Packet Loss affecting some Linodes in London

1:46pm (EDT): Connectivity has returned to normal and we will continue to monitor at this time.

1:41pm (EDT): We are looking into a network issue in the London facility. There is no reason to issue reboots as Linodes are still up at this time.

Second Life - Emerald Viewer to be Blocked From Second Life

As of 10am PT Wednesday, September 8, the Emerald Viewer will be blocked from logging in to Second Life as a result of violations of our Policy on Third Party Viewers. Residents who have been using any version of the Emerald Viewer will need to use a different Viewer to access Second Life. You can download the official Second Life Viewer, developed by Linden Lab, here. Or you can learn more about alternative Viewers, developed by third parties, here. There are several new Viewers listed in the TPV Directory, so there are many alternatives available to you.

We take Residents’ privacy, safety, and security very seriously and will take action to enforce the policies that help protect it. As our CEO, Philip Rosedale, has blogged about, we recently removed the Emerald Viewer from our Third-Party Viewer Directory due to violations of our Policy on Third-Party Viewers.

Since then, we have been in communication with the Emerald development team and have requested several changes in order to remedy violations of our policy, including changes necessary to meet our privacy requirements, and to address GPL license violations. Unfortunately, the team was unable to comply within a stipulated time frame. As a result, we have decided to block logins from the Emerald Viewer in order to protect our Residents. All versions of the Emerald Viewer will be blocked from logging in to Second Life as of tomorrow at 10am. Please be aware that attempting to circumvent our blocking to access Second Life with a banned Viewer is a violation of the Policy on Third-Party Viewers and may result in the loss of one's account.

DreamHost - Stringray unresponsive

We’ve dispatched one of our admins to the physical location of the server to work on getting it back up and we’ll update this post as soon as we have a handle on the situation and service is restored.

madasafish - Planned Telephony System Maintenance - Tuesday 7th September 5.30pm-Wednesday 8th September 6.00am

When's this work happening? This evening, 7th September into tomorrow morning. What does it affect? The telephone system used by our Customer Support Centre. How long will it take? Up to ten hours. What does the work involve? We are migrating our internal telephony platform across to a new system with improved functionality. Am I likely to notice the work? It's unlikely. Some of our support numbers will not work for a brief period of time during the maintenance window, however this part of the work has been scheduled for the early hours of the morning to minimise disruption. Is there anything else I need to know? We originally attempted this work last month but encountered some unforeseen problems that meant we had to roll back the changes. Kind Regards, Bob Pullen Customer Support

AAISP BT Incident and Planned - Broadband Incident Report : ST BURYAN DSL 001 : DEGRADED LOSS OF SERVICE : Reference 24310 : Issue 3 :

--------------------------------------------------------------
Incident Manager - Automatic Mailer.
*** PLEASE DO NOT REPLY TO THIS E-MAIL ***
--------------------------------------------------------------

BT Wholesale - Broadband Incident Report

Please read the following incident report as it could have an impact on some of your end users

BT Incident Reference:

24310

Issue :

Detected Date:

29/08/2010

Detected Time:

09:16

Start Date:

29/08/2010

Start Time:

09:16

Anticipated Clear Date:

29/08/2010

Anticipated Clear Time:

11:01

Incident Headline:

ST BURYAN DSL 001 : DEGRADED LOSS OF SERVICE

Incident Details:

At the moment some of your Broadband End Users will be experiencing a degradation of Broadband Service.

Progress Details:

BT is making progress towards full restoration of Service.
Latest Update:
An Engineer has been tasked to site.
Further updates will be provided as soon as possible.

Dialling Codes Affected:

01736

--------------------------------------------------------------
Incident Manager - Automatic Mailer.
*** PLEASE DO NOT REPLY TO THIS E-MAIL ***
System Time Stamp: 29/08/2010 10:04

AAISP BT Incident and Planned - Broadband Incident Report : ST BURYAN DSL 001 : DEGRADED LOSS OF SERVICE : Reference 24310 : Issue 4 :

BT Wholesale - Broadband Incident Report

Please read the following incident report as it could have an impact on some of your end users

BT Incident Reference:

24310

Issue :

Detected Date:

29/08/2010

Detected Time:

09:16

Start Date:

29/08/2010

Start Time:

09:16

Anticipated Clear Date:

29/08/2010

Anticipated Clear Time:

13:30

Incident Headline:

ST BURYAN DSL 001 : DEGRADED LOSS OF SERVICE

Incident Details:

At the moment some of your Broadband End Users will be experiencing a degradation of Broadband Service.

Progress Details:

BT is making progress towards full restoration of Service.
Latest Update:
An Engineer has been tasked to site.
Further updates will be provided as soon as possible.

Dialling Codes Affected:

01736

--------------------------------------------------------------
Incident Manager - Automatic Mailer.
*** PLEASE DO NOT REPLY TO THIS E-MAIL ***
System Time Stamp: 29/08/2010 10:19

AAISP BT Incident and Planned - Broadband Incident Report : ST BURYAN DSL 001 : DEGRADED LOSS OF SERVICE : Reference 24310 : Issue 5 :

BT Wholesale - Broadband Incident Report

Please read the following incident report as it could have an impact on some of your end users

BT Incident Reference:

24310

Issue :

Detected Date:

29/08/2010

Detected Time:

09:16

Start Date:

29/08/2010

Start Time:

09:16

Anticipated Clear Date:

29/08/2010

Anticipated Clear Time:

15:00

Incident Headline:

ST BURYAN DSL 001 : DEGRADED LOSS OF SERVICE

Incident Details:

At the moment some of your Broadband End Users will be experiencing a degradation of Broadband Service.

Progress Details:

BT is making progress towards full restoration of Service.
Latest Update:
An Engineer is now on site.
Further updates will be provided as soon as possible.

Dialling Codes Affected:

01736

--------------------------------------------------------------
Incident Manager - Automatic Mailer.
*** PLEASE DO NOT REPLY TO THIS E-MAIL ***
System Time Stamp: 29/08/2010 13:21