September 24, 2008

Network Connectivity Issues : Possible DoS attack

Some of our clients may experience intermittent difficulty with their services. We are seeing random and widespread outages of very short durations. They appear to be lasting from one to fifteen minutes and are not limited to any specific service. We suspect the network is under a large scale "denial of service" attack and are working to identify and isolate it.

October 23 , 2007

cPanel Maintenance : Server Software Upgrade

We will be upgrading our cPanel web servers' software today after 5pm. Web hosting customers can expect approximately 15 minutes of downtime.  We thank you for your patience during this maintenance window.

October 23, 2007

SBC DSL : Network maintenance

Our SBC carrier will be swinging all of our DSL subscribers in the Los Angeles area to a new POP at 12:01 a.m. on Tuesday, October 23rd. We expect approximately one hour of downtime as circuits are migrated. This deployment brings with it decreased latency to the Internet for our LA customers.  We thank you for your patience during this maintenance window.

July 11, 2007

Level 3 : Network outage

At 2:49 PM, one of our primary backbone T3 connections experienced a two and a half minute outage. We have opened a trouble ticket with our backbone provider, and will schedule emergency testing after midnight on
07-12-07, pending the results of the monitor which is now in place.

June 23, 2007

Emergency Maintenance: Keyway Mail and Dialup Servers

1:30 pm -- Customers using mail.keyway.net to send or receive email might experience a temporary delay or an inability to connect while we perform unscheduled maintenance on that mail server.   Also, customers dialing in might experience login failures while we perform maintenance on our authentication server. We expect the maintenance to last no more than half an hour, and apologize for any inconvenience this might cause.

June 22, 2007

Emergency Maintenance: DSL Maintenance

12:01am on Friday, June 22nd -- Our SBC carrier will be performing emergency maintenance on a portion of their DSL aggregation infrastructure in San Francisco. Customers in Los Angeles as well as a large portion of the LATA 1 customers will be affected by this operation. While expected downtime is approximately 15 minutes, they are reserving a one hour maintenance window in case of further trouble.   We thank you for your patience and apologize for any inconvenience this might cause.

March 23, 2007

Level 3 : Network outage resolved

All services have been restored as of 3:45PM today. The total outage duration was one hour and forty-five minutes. The outage was caused by a failed Verizon fiber-optic high speed line. Keyway's engineers will be researching ways to avoid future similar outages and add more redundancy to our network.

March 23, 2007

Level 3 : Network outage

Keyway Internet Services is currently experiencing an outage with our connection to the rest of the Internet. The outage occured at 2PM today. The outage is somewhere in the fiber-optic high-speed circuit connections that connect our network to the Internet. Keyway Engineers are currently working with Engineers to locate and repair the issue. We do not yet have an official Estimated Repair Time but the issue appears to be minor and will likely be repaired within an hour or so.

February 19, 2007

Office Hours : 02/19/07

Keyway's business office will be closed on Monday 2/19/2007 in observance of President's Day.  Clients with premier support services should use their 24/7 contact numbers.

October 27, 2006

Scheduled Maintenance: Los Angeles SBC Area DSL

On Friday, October 27th at 12:01 a.m., our SBC carrier will be performing routine maintenance on the equipment that aggregates our Los Angeles SBC area DSL customers. This operation is intended to increase performance and reliability for all our LA area customers. Downtime will be approximately 15 minutes. We apologize for any inconvenience this might cause.

October 5 , 2006

Emergency Maintenance: Power interruption

9:15 PM -- We have been notified by the building management at our Los Angeles Facility that they need to perform Emergency Maintenance on a power circuit there. This maintenance will interrupt service to some of our
clients in that facility. If you you believe that you are experiencing a
service interruption due to this maintenance, please call our support
department at 909.933.3699. We apologize for the short notice.

August 18 , 2006

Hardware failure: Power outage

In the morning hours of 8/18/06 (approx. 4:30AM), a Southern California Edison power surge tripped three of the breakers in the Keyway Ontario
Datacenter. The Los Angeles Datacenter was completely unaffected. The
power surge permanently damaged one UPS and one DNS server and temporarily disabled a backbone switch and two more UPS'. All Internet services were restored at 5AM with the exception of DSL and some T1 clients who were restored at 6:30AM.

August 5 , 2006

Hardware failure: Spam Filtering Outage

This morning at approximately 8am , the server which performs our spam-filtering went offline due to hardware problems.  This issue is being worked on, and we expect to have it resolved by the end of today.  We apologize for any inconvenience this may cause.

July 17, 2006

Keyway office - phone outage

Today at approximately noon, our office lines went down.  We have opened a trouble ticket with our telephone line carrier and hope to have the issue resolved very soon. 

If you need sales or support assistance, please email us at either sales@keyway.net or support@keyway.net.    We apologize for any inconvenience this may cause.

June 30, 2006

Network Issues : DNS

Today at 10AM our primary DNS server ("name server"), ns1.keyway.net stopped responding to name lookup requests due to configuration file corruption. We are repairing and determining the cause of the corruption to effect preventative measures. We have three name servers online at this time and so this should not impact your performance at all. Our ETA for repair is 11AM this morning.

If you find your connection does not seem to be performing properly, please call our technical support at 909.933.3699 for assistance.

June 14, 2006

SBC DSL : Network Issues

We received a report from our SBC carrier that they were experiencing network instability issues and that they are working to find the cause and resolve the problem.   We apologize for any inconvenice that this may cause.  This issue is only impacting some clients who have SBC as their phone company and have Keyway DSL, no other services are affected at this time.  The issue started today, June 14th at 10AM and was corrected shortly thereafter.

June 12, 2006

Mail Issues : DNS caching

We are receiving reports of clients unable to send or receive email when
they are using non-Keyway DNS servers. The cause is due to bad information that was distributed to the root domain servers on Sunday. The problem was identified and corrected on Sunday, but some DNS servers are caching (carrying over) the bad information. There are two solutions, first to 'refresh' the non-Keyway DNS server so that it stores the current valid data, or second, to enter a Keyway DNS server into the settings for your computer.

May 27, 2006

Mail Issues : SORBS

Some of our client's who run their own mail servers located on the IP
blocks listed below may experience some difficulty sending email to recipients who choose to use SORBS to filter inbound email. The reason for this is that SORBS has erroneously listed the affected IP addresses as "dynamically issued" when in fact they are permanently assigned. We have opened a ticket with SORBS to correct the erroneous listing but they take several days to respond to each mailnote, so we do not expect this issue to be resolved until 6/9/06.

Here are some actions you can take in the meantime:

1. Use Keyway's primary mailhost, mail.keyway.net, to send your email. This host accepts email from any Keyway issued IP address and will allow you to remain fully operational until the SORBS issue can be resolved.
   
2. Contact the recipient and ask them to either 'whitelist' you, or
   
3. Contact the recipient and ask them to temporarily stop using SORBS until SORBS corrects the erroneous listing.
   
4. Request a SORBS 'exclusion' entry. To request an exclusion, please contact our technical support.

Keyway has three team members dedicated full time to correcting this issue and it is our current highest priority item. 

Affected IP addresses:
----------------------
216.117.192.0 to .255
216.117.193.0 to .255
216.117.194.0 to .255
216.117.195.0 to .255
216.117.196.0 to .255
216.117.197.0 to .255
216.117.198.0 to .255
216.117.200.0 to .255
216.117.202.0 to .255
216.117.203.0 to .255
216.117.204.0 to .255
216.117.205.0 to .255
216.117.207.0 to .255
216.117.211.0 to .255
216.117.212.0 to .255
216.117.213.0 to .255
216.117.214.0 to .255
216.117.215.0 to .255
216.117.216.0 to .255
216.117.217.0 to .255
216.117.218.0 to .255
216.117.219.0 to .255
216.117.220.0 to .255
216.117.221.0 to .255
216.117.222.0 to .255
216.117.223 0 to .255