TOCICI

System Status

Service Status1) Date@Time2) & Notes
Cooling
Electrical
Email (SMTP)
Network
VoIP
SAN (storage)
VPS
Web (HTTP/S)

Recurring maintenance windows are from 9:00PM until 1:00AM, daily.3)

A 100% dual modular redundant computing, networking, power, and cooling infrastructure ensures that most maintenance has zero impact on core production services. If we expect any impact, notices are posted here, and an email is sent to affected parties.

No Scheduled Maintenance

Notices will be posted here, when scheduled maintenance is arranged.

Automated System Status Report (updates once per minute)

VM Hosts
(tcp\80) pdxvmh01pdxvmh01
(tcp\80) pdxvmh02pdxvmh02
(tcp\80) pdxvmh03pdxvmh03
(tcp\80) pdxvmh04pdxvmh04
(tcp\80) pdxvmh05pdxvmh05
(tcp\80) pdxvmh06pdxvmh06
(tcp\80) pdxvmh07pdxvmh07
(tcp\80) pdxvmh08pdxvmh08
(tcp\80) pdxvmh09pdxvmh09
(tcp\80) pdxvmh10pdxvmh10
(tcp\80) pdxvmh11pdxvmh11
(tcp\80) pdxvmh12pdxvmh12
(tcp\80) pdxvmh13pdxvmh13
(tcp\80) pdxvmh14pdxvmh14
(tcp\80) pdxvmh15pdxvmh15
(tcp\80) pdxvmh16pdxvmh16
(tcp\80) pdxvmh17pdxvmh17
(tcp\80) pdxvmh18pdxvmh18
(tcp\80) pdxvmh19pdxvmh19
(tcp\80) pdxvmh20pdxvmh20
(tcp\80) pdxvmh21pdxvmh21
(tcp\80) pdxvmh22pdxvmh22
(tcp\80) pdxvmh23pdxvmh23
(tcp\80) pdxvmh24pdxvmh24
(tcp\80) pdxvmh25pdxvmh25
(tcp\80) pdxvmh26pdxvmh26
(tcp\80) pdxvmh27pdxvmh27
(tcp\80) pdxvmh28pdxvmh28
Infrastructure
(tcp\80) PDX 10GigE Switch #1PDX 10GigE Switch #1
(tcp\80) PDX 10GigE Switch #2PDX 10GigE Switch #2
(tcp\80) PDX 10GigE Switch #3PDX 10GigE Switch #3
(tcp\80) PDX 10GigE Switch #4PDX 10GigE Switch #4
(tcp\80) SMTP #1SMTP #1
(tcp\80) PDX SAN #1PDX SAN #1
(tcp\80) PDX SAN #2PDX SAN #2
(tcp\80) PDX IDS #1PDX IDS #1
(tcp\80) PDX IDS #2PDX IDS #2
(tcp\80) VoIP #1VoIP #1
(tcp\80) PDX Router #1PDX Router #1
(tcp\80) PDX Router #2PDX Router #2
(tcp\80) PDX iSysMon #1PDX iSysMon #1
(tcp\80) PDX DB #1PDX DB #1
(tcp\80) PDX DB #2PDX DB #2
(tcp\80) PDX HTTP #1PDX HTTP #1
(tcp\80) PDX HTTP #2PDX HTTP #2
(tcp\80) DNS #1DNS #1
(tcp\80) DNS #2DNS #2
(tcp\80) PDX Mirror #1PDX Mirror #1

History

Day@Time4) Service Notes
29SEPT2016@22:28 VPS The host PDXVMH11 encountered a CPU failure. All affected auto-failover environments switched to their secondaries without issue. Non-failover environments were back online by 23:12.
10MAR2016@02:30 VPS Just deployed 14 new SSD nodes. Customers on our retiring nodes will be live-migrated to new systems over the next few months. Watch your email for details.
30JAN2015@23:30 VPS Four new SSD nodes are now online, two more nodes will follow once the darned RAM shows up!
15OCT2014@21:45 VPS The VPS host node PDXVMH18 reported unusual hardware-related log entries. As a precaution, automated systems redistributed affected VPS environments onto other VPS host nodes. PDXVMH18 has since been taken offline for a CPU replacement and followup burn-in testing; automated systems will return PDXVMH18 to the host node resource pool upon completion of CPU burn-in.
28APR2014@00:35 VPS PDXVMH18 encountered a kernel panic, and was automatically restarted by its watchdog processes, with all affected auto-failover environments switched to their secondary nodes without issue. Non-failover environments where brought back online once each environment completed a necessary automated filesystem check.
07APR2014@09:00 ALL We were never affected by the OpenSSL software bug commonly known as Heartbleed. In response to this bug, you DO NOT need to change your password(s) on our systems.
05NOV2013@09:15 VPS PDXVMH15 encountered a kernel panic, and was automatically restarted by its watchdog processes, with all affected auto-failover environments switched to their secondary nodes without issue. Non-failover environments completed necessary filesystem checks, where brought back online through automated processes.
16AUG2013@23:00 VPS Five new SSD nodes are now online.
14MAY2013@11:27 NET An ingress DDoS attack exceeded 20Gps; impacting uplink latency and packet loss across Boise, Denver, Seattle, Sacramento, and Portland backhaul connections, and affecting multiple upstream networks. At approximately 15:10 the attack ended and services returned to normal.
03APR2013@00:01 ALL It has been a quiet and uneventful past few months. Just putting something here - so that folks stop asking if we're still alive :-)
13JAN2013@21:35 VPS PDXVMH14 encountered a kernel panic, and was automatically restarted by its watchdog processes, with all affected auto-failover environments switched to their secondary nodes without issue. Non-failover environments completed necessary filesystem checks, where brought back online through automated processes.
31DEC2012@21:12 VPS PDXVMH23 experienced a watchdog timeout caused by actions within an abusive VPS environment. PDXVMH23 was automatically restarted, with all affected auto-failover environments switched to their secondary nodes without issue. Non-failover environments where all back online within ~30 minutes (after completing automated filesystem checks). The offending VPS environment has been suspended.
11DEC2012@20:01 VPS PDXVMH23 experienced an unexpected watchdog timeout, and was automatically restarted. All affected auto-failover VPS environments automatically switched to their secondary nodes without issue. Non-failover environments where all back online within about 1 hour (after completing automated filesystem checks).
05DEC2012@22:10 NET 10GbE install completed, no issues encountered. Using bonded network interfaces, all systems now leverage an aggregate 20Gbps core network fabric.
01DEC2012@00:35 NET Unusual ARP traffic was spotted on the network; causing intermittent connectivity issues for a few hosts. Issue has been mitigated; returning networking services to normal.
19NOV2012@23:37 SAN Completed the retirement of our old SAN array; offlined and removed equipment. New SAN array is nicely buzzing along.
30OCT2012@05:00 NET An upstream provider completed scheduled router maintenance, without any measurable interruption of network/IP services.
07OCT2012@13:00 VPS A “soft lockup” kernel bug forced the restart of PDXVMH23. All affected auto-failover VPS environments automatically switched to their secondary nodes without issue. All non-auto-failover environments where brought back online after completing their respective filesystem checks.
30JUL2012@23:00 NET Completed scheduled router upgrades.
01JUN2012@15:00 VPS A “soft lockup” kernel bug forced a restart of PDXVMH21. All affected auto-failover VPS environments automatically switched to their secondary nodes without issue. Non-failover environments where automatically brought back online after completing filesystem checks.
24APR2012@18:50 VPS PDXVMH11 experienced an unexpected watchdog timeout, and was automatically restarted. All affected auto-failover VPS environments automatically switched to their secondary nodes without issue. Non-failover environments where automatically brought back online after completing filesystem checks.
10APR2012@23:45 VPS PDXVMH23 encountered a kernel panic, and its watchdog timer initiated a restart. After over 9 hours of non-stop troubleshooting and diagnostics efforts by our systems engineers, and by IBM systems engineers, they fingered an apparent firmware bug previously unknown to them, as the root cause. Replacement equipment has already been shipped by our vendor, and all affected VPS environments have been migrated to a standby VPS host node in the meantime.
01APR2012@12:30 VPS A failed hard disk drive was replaced in PDXVMG23. Hotswap hardware ensured that no downtime was necessary during an emergency maintenance window.
27MAR2012@14:45 NET A DDoS impacted uplink latency and packet loss; internal mitigation tools and processes promptly detected the inbound attack, and returned services to normal.
09MAR2012@15:00 NET A DDoS affected our upstream links; increasing network traffic latency and packet loss. After two rounds of attacks, automated mitigation tools successfully returned systems and networking status back to green.
07DEC2011@17:31 VPS PDXVMH14 was the target of a DDoS that spanned about 4 hours in duration. After a few attack rounds, enough information was gathered to successfully suppress further attempts.
06DEC2011@22:38 VPS A software bug caused a kernel panic on PDXVMH14 this evening, and an automated restarted occurred. Please see suspended_support for details.
28NOV2011@23:15 SAN Access to the primary and secondary SAN arrays, by VPS host nodes PDXVMH11 and PDXVMH14, where hindered by a multi-switch networking fault. Once the SAN's filesystem repairs had completed, all affected VPS environments where back online by 22:33.
27NOV2011@12:42 NET A DoS across our primary and secondary uplinks affected networking services until about 13:45 today.
22NOV2011@10:16 VPS At 07:35 PDXVMH14 and PDXVMH11 encountered network driver kernel panics. Both nodes are now back online.
19NOV2011@21:40 VPS The VPS host PDXVMH11 had encountered a critical hardware fault this evening, and has been replaced. All VPSes where brought back online by 21:40pm this evening.
27OCT2011@23:05 VPS All scheduled VPS host server restarts have been completed without issue.
30SEPT2011@23:00 NET One of our upstreams experienced a DDoS that targeting core equipment within their systems. This manifested as slower than normal connectivity, while affected routes to our subnets where adjusted amongst secondary/tertiary uplinks.
21SEPT2011@21:00 VPS A hardware failure on PDXVMH12 required an unscheduled node replacement. All auto-failover VPSes promptly switched to their backup server without issue, non-redundant VPSes were brought back online once the replacement hardware was installed at 21:00 (US/Pacific).
11AUG2011@18:59 VPS On August 11th at 18:41 an external DDoS attack degraded network performance, by saturating one of our IP transit backhauls. Mitigation procedures were immediately put into motion, and as-of 18:59 this evening, the attack stopped.
09AUG2011@23:35 VPS On August 9th at 21:11 an external DDoS attack degraded network performance, by saturating one of our IP transit backhauls. Mitigation procedures were immediately put into motion, and as-of 23:35 this evening, network performance across all backhauls returned to normal.
18JUL2011@22:10 VPS The VPS host server PDXVMH04 encountered a critical hardware fault, and needs to be taken offline for further attention. To ensure continued services, all hosted VPSes are being actively migrated onto other servers. Impact to migrated VPSes is a restart; if all VPS services autostart, no manual intervention is needed.
17JUL2011@14:45 NET A denial of service attack affected networking on PDXVMH09, and restart of this VPS host server was necessary in order to restore networking services.
05JUL2011@09:00 NET Completed scheduled nameserver migrations.
01JUL2011@17:00 NET As planned, our new direct ARIN IPv4 allocation has been in full production without a single incident throughout the entire month. As a result, the planned retiring of two legacy IP netblocks (199.223.127.0/24 & 65.49.94.0/24) is effective today. If you encounter access issues, please double-check that you're using the information emailed to you earlier this month. Your control panel interface also details your new IP addressing information.
25May2011@16:39 NET For reasons that are not yet clear, an upstream provider began filtering our BGP route announcements for 199.223.127.0/24. This would have interrupted communications for services hosted within this subnet for the past hour. The upstream has since corrected their issues.
19May2011@11:30 VPS A hardware fault on PDXVMH10 caused an unscheduled watchdog timeout and restart. Services are back online & a detailed hardware review is underway.
10May2011@20:35 VPS The primary network interface in PDXVMH09 failed this evening. The secondary interface is currently providing networking services, until the next scheduled maintenance window where well be swapping the entire server for an alternate system.
09MAY2011@14:44 NET An external DDoS attack degraded network performance by saturating one of our IP transit backhauls. Mitigation procedures were immediately put into effect, and as-of ~15:45 this afternoon, network services returned to normal.
21APR2011@23:55 VPS A hardware failure necessitated the replacement of PDXVMH10 this evening. No data was lost & the server is back online.
05APR2011@22:40 VPS Scheduled maintenance completed on PDXVMH01 & PDXVMH04; all systems back online
17MAR2011@21:16 VPS A kernel panic on pdxvmh09 forced a watchdog timeout and system restart.
15MAR2011@20:20 VPS A hardware fault on pdxvmh10 caused a watchdog timeout and system restart.
02FEB2011@12:00 NET One of our network back-hauls was experiencing high latency and packet loss. Traffic to/from the datacenter was promptly routed around their issues.
02FEB2011@09:00 VPS PDXVMH01 auto-restarted due to an automated watchdog timeout that's in place to ensure prompt recovery from kernel panics.
24JAN2011@01:00 VPS PDXVMH09 auto-restarted due to an automated watchdog timeout that's in place to ensure prompt recovery from kernel panics.
08JAN2011@22:00 VPS Scheduled maintenance on PDXVMH09 was completed without issue.
22DEC2010@13:00 VPS The hardware RAID card in PDXVMH09 died at 11:50 today. Replacement equipment was installed and online by 12:32.
20DEC2010@22:00 NET A DDoS attack degraded network performance by saturating our IP transit backhauls. As-of ~19:00 this evening, the effects were mitigated.
16DEC2010@08:39 VPS The VPS host pdxvmh09 experienced a sudden restart. Cause was due to a very unusual bc_ioprio_queue kernel panic.
07DEC2010@22:00 VPS Updates applied issue-free
05DEC2010@22:00 VPS Maintenance completed without issue
25OCT2010@21:35 VPS One of our IP backhauls, Stephouse Networks, continues to encounter internal network issues (as-of 31OCT2010). We've since re-routed all traffic away from their network; across our other backhauls. With the re-routing, your quality of services remains unaffected, we're also in the process of securing a replacement IP transit provider.
24OCT2010@21:35 VPS Completed scheduled equipment upgrade, incident-free & ahead of schedule!
24OCT2010@16:45 VPS An update was rolled out to the autobuild engine - everything looks great.
18OCT2010@14:11 VPS The VPS Host Server PDXVMH02 encountered a kernel panic. Cause appears to have been from a stray alpha particle - we're monitoring this situation closely, but so-far everything looks great since then.
21SEPT2010@17:00 NET Yesterday's ~2% IPv6 specific packet loss has been corrected upstream - all clear now!
19SEPT2010@23:45 VPS Incident-free kernel upgrades completed, on schedule!
15SEPT2010@23:00 VPS Firmware updates and hardware restart completed on schedule & without issue.
12SEPT2010@11:00 VPS The automated VPS provisioning system threw a tantrum this morning; backlogging almost a dozen new orders received over the past two hours. The backlog was just cleared - thanks for your patience.
31AUG2010@22:00 VPS Incident-free kernel upgrades completed, on schedule!
29AUG2010@23:00 ALL Incident-free kernel upgrades completed, on schedule…laying the groundwork for some exciting announcements soon!
08AUG2010@22:00 ALL Incident-free electrical upgrades completed, on schedule; all eight BladeCenter power supplies are now online.
02AUG2010@00:01 VPS Scheduled hardware upgrade in PDXVMH18 has been completed. 2nd IBM BladeCenter Chassis is now online.
22MAY2010@22:55 VPS Scheduled hardware upgrades in VPS host servers have been completed.
14MAY2010@21:35 VPS Replaced problematic hardware, without service interruptions.
25MAR2010@20:00 VPS VPS host server replacement completed (failed baseboard management controller).
19FEB2010@09:00 VPS Completed replacement of PDXVMH10 (failed power supply).
07JAN2010@09:00 Electrical The city of Portland experienced a major substation fire and subsequent 8 hour long power outage throughout most of the downtown area on January 7th 2010. Included in the outage area was our datacenter. Although thanks to the redundant power provided via the backup battery bank and two large diesel generators, TOCICI had zero service interruptions.
28DEC2009@00:05 ALL Whew…time to go home; just completed the installation of a few new servers, another network switch, and a backup IP-KVM. YAY for scheduled maintenance windows!
09DEC2009@21:00 VPS The kernel panic appears unrelated to any hardware issues. We'll continue to monitor the situation, but we're convinced that the kernel panic was a non-reproducible software fluke.
09DEC2009@02:55 VPS A kernel panic on PDXVPS10 forced a watchdog restart on . Contingency plans in effect; while the machine is back online, and all VPSes are running, we've taken the primary network interface offline, the secondary is now servicing network traffic. In addition, we're running diagnostics on the primary network port to determine if its a hardware or software quirk which caused the kernel panic.
07DEC2009@21:00-23:00 Network Scheduled network switch replacement completed without complications.
29NOV2009@10:27 ALL All systems consolidated and online! Thanks for the awesome data circuit Stephouse!
29NOV2009@09:00-11:00 ALL This morning we'll be finalizing the last phases of our system consolidation project
28SEPT2009@03:14 VPS We just completed the migration of three existing Google Health connected apps to our infrastructure. WOOT!
10SEPT2009@16:49 Network Latency issues seem resolved - we're staying on top of this folks!
05SEPT2009@13:25 Network Latency continues to settle back into historical norms, but we're still not fully-satisfied
03SEPT2009@11:19 Network Our historical graphs are showing a very undesirable trend of increasing latency - looks like its time for a new bandwidth provider
10AUG2009@19:00-23:59 Network Primary network provider is replacing their Portland/Salem long-haul cable; expect intermittent 1-5minute network interruptions throughout this evening.
08AUG2009@07:03 Network After a brief unscheduled provider/network outage, we're back online w/ latency back to pre-August 7th trends. WOOT!
07AUG2009@14:17 Network Latency via two back-hauls remains unusually high
07AUG2009@12:19 Network Latency is creeping up again - nothing critical, but we're not amused. Broadcast email details forthcoming.
02JUN2009@07:23 Network latency graphs now trending back to normally low levels
01JUN2009@09:37 Network Crazy days ahead…latency graphs are showing an unexpected upward trend
07JAN2009@16:24 ALL Our equipment is being migrated to the Pittock; an impressive datacenter in downtown Portland with three redundant commercial power feeds, diesel beasts providing backup power, and a very intimidating battery bank. Watch our upcoming broadcast email for more info.
18DEC2008@03:00-06:00 VoIP VoIP equipment upgrades are going in…watch email for details.
03NOV2008@23:12 Cooling 82F in the datacenter…we're raising hell, and looking for alternative spaces
15OCT2008@08:27 Cooling 75F in the datacenter. Unless this is resolved soon, we're packing up our equipment and moving out.
02OCT2008@14:48 Cooling 77F in the datacenter. Still leaning on the provider to do something about their cooling failures.
17SEPT2008@16:47 VPS …aaaaannnnddd POP! goes the disk drive. Thank goodness for redundant hotswap equipment; emergency equipment replacement in PDXVPH07 completed w/ zero downtime
17SEPT2008@16:32 Cooling 79F in the datacenter…aaaaannnnddd, yep, the disk drive is dead…
04APR2008@07:16 ALL This morning we were rudely awakened to screaming phones & pagers…our colocation provider was caught in a floor-wide power failure, not once, but twice (6:27am & 6:52am). So much for promises of redundant power! We're onsite now, and it's amazing how many iPhones are around…scary…the room is full of very upset SysAdmins. Yet-again, the datacenter space is far warmer than acceptable for a datacenter. Details forthcoming via an email broadcast.
27MAR2008@21:12 Cooling Our temperature probes are quite upset…although not nearly as upset as we are; it's nearly 84F in the datacenter
13MAR2008@13:01 Network reverse DNS (PTR) has finally been restored. Service provider has given us a generous refund which we're now passing onto the affected customers. Watch your email for respective partial service refunds.
12MAR2008@08:12 Network reverse DNS (PTR) is still not functioning as promised via contracts…we're very unhappy with the service provider, and making that well known. Also seeking alternative providers now.
11MAR2008@08:14 Network Our network provider has FUBARed reverse DNS (PTR) records - working with them to resolve
19FEB2008@12:05 VPS Just added 5TB to our storage array - yay for growth!
03AUG2007@12:37 VPS We're adding 1TB of additional storage capacity to our infrastructure
22JUL2007@23:53 Network Whew! Systems migrated completed just seven minutes before the deadline. Enjoy the new services folks!
20JUL2007@08:15 Network It's time to say goodbye to a very unprofessional and frustrating hosting provider. We're growing, we demand service excellence, and we will not tolerate the sillyness of needing to call the CEO's personal cell phone for resolution of frequent and fundamental core competency failures. Details forthcoming in emails to affected customers.
06FEB2006@11:19 Network ICMP RTT latency through two of our three multi-homed routes has skyrocketed…network provider is working to resolve
30JUL2005@14:18 ALL Looks like our colocation provider forgot how to use copy/paste … FUBARed reverse DNS is being resolved now
29JUL2005@23:47 ALL Systems migrations completed w/out issue.
28JUL2005@14:18 ALL We're finally reaching “non-low-budget” status - equipment migrations will commence soon!
1) All OK: All OK
Investigating: Investigating
Service Interruption: Service Interruption
Status Unknown: Status Unknown
2) , 4) All times are US/Pacific
3) All times are US/Pacific. Time exceptions are made for urgent/emergency issues