|January 5, 2018: Meltdown and Spectre Mitigation|
SRO Hosting is working to reduce the attack footprint on our servers related to the 'Meltdown' and 'Spectre' vulnerabilities
All customer facing Virtual Machines have already been updated with new kernel and microcode patches providing protections against anticipated attack vectors. Physical hosting servers are also being updated this weekend. Due to the severity of these vulnerabilities, SRO Hosting is updating affected systems with minimal consideration to site downtime. We appreciate our hosting customers understanding that security is THE paramount consideration and a few minutes of downtime is insignificant compared to potential exposure to imminent attacks.
|January 29, 2017: SQL Server Outage|
Some customers may have experienced an hour or more of SQL server downtime as one of our virtual hosting servers crashed this afternoon.
We restored backed up databases to another server and all sites should be up and running again.
If your database is out of date, we appreciate your patience while we restore the dead server backup and import a more current version of your database to the new server.
|May 19, 2016: Mail filter updates|
Due to a recent dramatic increase in phishing emails, we have updated our mail scanners to be more aggressive in scanning and removing malicious ZIP attachments. The end result will be far fewer virus attachments getting through, however be advised that delivery of emails with large legitimate file attachments may be delayed.
If this causes delivery troubles, please contact us about white-listing specific senders or recipients.
|May 8, 2016: PHP Version upgrade|
The DR1 server PHP version was upgraded to 5.5.35 for security reasons this evening. There should be no compatibility issues with this update, however you are advised to check the php release page for more information.
|August 28, 2015: DNS Outage|
The "Two inch grinch" server failed Thursday evening. TIG hosts the SRO customer portal and master DNS server. The outage did not affect customer sites or services however the subsequent corruption of our secondary DNS server Friday morning caused a system wide outage as customer sites were unable to resolve via DNS.
This outage was caused by a lack of knowledge of changes between the BIND DNS software versions on two of our servers and persisted until both primary and secondary servers were replaced in a few hours.
A tertiary dns server has been provisioned to avoid any similar issue in the future.
|April 8, 2015: Temporary outage|
SRO Hosting experienced an outage on the DR1 hosting server after a routine security audit triggered an emergency software upgrade. This incident impacted services from April 7, 11:53:41pm to April 8, 01:18:40am at which time some customer web sites and email services may have been temporarily unavailable. All services returned to normal after the incident and no data loss is anticipated.
It should be stressed that in spite of its immediacy, this outage was preventative in nature and no customer data was known to be compromised by the repaired vulnerabilities.
|March 3, 2015: Webmail improvements|
Today we upgraded our Roundcube installations with Memcache and database message caching. Users with huge (multi-GB) mailboxes should see a dramatic improvement in webmail loading speed on subsequent logins.
|March 1, 2015: Software updates|
PHP and Apache versions were updated today. This caused a temporary slowdown as we removed PHP opcode cache until we found one which did not cause segfaults when running software (ie: Wordpress) with 3rd party caching plugins. Servers running older PHP versions now use Xcache instead of eaccelerator or APC.
|November 11, 2014: TIG server outage|
The TIG server was down for a few hours on Nov 11th and 12th. TIG hosts DNS and some SRO internal services. The outage did not impact client access, however DNS may have been slightly delayed during the outage.
|March 7, 2014: IMAP server issues|
Our apologies for the brief email outage this evening. An IMAP server authentication module failed to load and it took some digging to determine that a prior software installation had updated authentication libraries, causing the IMAP server to fail after a routine restart. The old authentication libraries were restored and IMAP service resumed at approximately 6:45pm this evening. We thank you for your patience.
|September 3, 2013: vBulletin security fix|
In response to a recent vBulletin security bulletin, we have pre-emptively removed the default /install/ folder from most vBulletin 4.x sites on our servers. If you are running a vBulletin forum and have a vB installation folder with a different name or location, please do yourself a favor and delete the installation folder yourself. Removing the /install/ folder will not harm your vbulletin forum.
|March 30, 2013: Network Outage|
Saturday morning sometime before 8AM (when we first noticed the problem), IP addresses in the 64.235.233.x range failed to resolve from outside our datacenter. This resulted in some client email and web services being unavailable. We are currently unsure if this is a IP provider or peering issue, however our quick resolution has been :
Please note that DNS records may take from 30 minutes to several hours to propagate across the net, so some services may still be unavailable at this time. Please do not hesitate to call or email if you continue to have issues connecting to SRO hosted services.
- Update DNS records on all domains using legacy IP addresses to use the 216.193.197.x IP range.
- Verify server software (http/ftp/email) is utilizing the 216.193.197. range.
- Change server default gateways so internal software/scripts continue to function.
|July 27, 2012: Power Outage|
As you may have noticed, ALL our Los Angeles servers went down this morning at 9:20am due to a UPS failure at our data center. Power and networking were restored at approximately 10:30am, at which time we discovered serious data corruption on our customer storage drives, wiping out all user /home folders.
Thank heaven for backups ... and early warning! In spite of the drive failure, by approximately 11:50am, we restored nearly 1TB of user accounts from a backup. Customer web sites and email should be working now, but please understand there may be continuing issues as a result of this power failure. If you have any trouble, please do not hesitate to email email@example.com or call us at (877) 276-HOST.
We hope you understand how fortunate it was our backup drives were not affected in the same way as our primary storage. We cannot emphasize enough that you should HAVE YOUR OWN BACKUPS!!
|October 5, 2011: New Security Rules|
Please be advised that we have updated our mod security ruleset on the DR1 server. As mod security attack detection rules can cause problems with complex web applications, we request that you report any unusual errors to SRO Hosting so we may adjust the ruleset accordingly. As always, we can individually configure or disable modsecurity on a per-site basis until rules can be adjusted to play nice with your applications.
|September 9, 2011: Software updates (cont)...|
Due to security vulnerabilities in previous Apache and PHP versions, SRO has been 'forced' (by Trustwave and other auditing agencies) to upgrade to PHP 5.3.8 on all servers. While SRO STRONGLY disagrees with PCI compliance policies based on comparing software versions against the CVE rather than testing for actual exploitable vulnerabilities, we have little choice but to comply lest our clients lose card processing privileges. The end result is many SRO customers will be forced to update their sites to ensure compatibility with the newest PHP versions long before it should be necessary and we are sorry for this inconvenience this may cause.
It is important to state that SRO independently patches/hardens our servers against known vulnerabilities using our own code and third party modules such as Suhosin and mod security (why SRO has NEVER experienced a server software version related compromise affecting customer data), however the PCI compliance cartel continues to force upgrades with no evidence of actual vulnerabilities.
While there is no arguing the importance of standards such as PCI DSS for general security, the fact is: most substantial compromises to consumer card security in recent memory have had NOTHING to do with the metrics tested by the PCI compliance industry in this case (php/apache versions) but rather social engineering, card skimming and good old fashioned burglary (stolen trash/flash drives).
|September 8, 2011: Server software update notice|
SRO Hosting will be performing an Apache/PHP software upgrade this weekend on Friday and Saturday, September 9-10 at 11PM Pacific time. The upgrade window is from 11PM to midnight and web sites may be unavailable for several minutes at a time while various software packages are re-compiled and re-installed.
|June 7, 2011: Server crash update|
Status: Web sites and email have been succesfully migrated to the DR1 hosting server. Most sites and email accounts were up and running again within an hour or two of the crash. Unfortunately, DNS issues, loss of some MySQL passwords and "legacy" (PHP 5.x incompatible) code caused delays with some sites. Please see the new server software versions below (old versions in parenthesis).
Server Software versions
Apache: 2.2.19 (1.3.27)
PHP: 5.2.17 (4.4.8)
MySQL: 5.1.45 (4.1.45)
Note that SRO discontinued PHP 4.x support in May of 2010. While we will attempt help customers who need PHP assistance, it is a webmaster`s responsibility to maintain and update web sites security and manageability.
|June 6, 2011: DB1 Server crash|
The DB1 hosting server crashed for unknown reasons at about 8:00am this morning. Clients who were hosted on DB1 are being moved to the 'DR1' hosting server ASAP.
Status: Email service has been transferred to the new server and missing email account re-created. Web site DNS has also been pointed to the new server and the Apache configuration replicated. Sites using Mysql databases might still be non-functions (see below).
Please understand there will be some delays and glitches to resolve during the transfer, specifically:
1) DNS takes time to propagate. Some people may be unable to see your site or send you mail for a few hours.
2) Due to a sync error, Mysql database passwords may not have transferred. We have backups of the DB1 mysql config and will be manually updating mysql passwords through the evening.
3) Web sites, database and email accounts were all mirrored to a new server. Some data was not included in the backups, such as webmail contacts, mailing lists and global software (webmail, cp) preferences.
3) Some sites are still PHP 5.x incompatible and will have trouble on the new server. SRO discontinued official PHP 4.x support over a year ago, but we will do our best to assist customers who need help converting code.
SRO Hosting is doing everything possible to restore sites, email and databases as quickly as possible. We appreciate your patience during this transition.
|May 15, 2010: DR1 Server installed|
Just a quick note that the new DR1 server is now installed and has passed initial burn-in testing. Like most of our new machines, DR1 has dual quad Xeon cpus and 100mb/sec SATA3 drives, so it's nearly double the speed in every respect than our older shared hosting servers. DR1 is running Apache 2.2.x, PHP 5.2.x and Mysql 5.x, so it will be perfectly appropriate for newer sites not relying on legacy code.
|April 13, 2010: PHP 4.x support ending|
SRO Hosting support for PHP 4.x will be ending as of May, 2010.
Tests indicate some clients may still be running scripts which do not work under PHP 5. While we do not wish inconvenience, we can't continue to support PHP 4 as neither it nor its extensions are supported by the authors.
We will attempt to contact customers whom we expect to be affected by the end of PHP 4.x support, however it is the responsibility of site owners/managers to keep software up to date and all applications not already updated for PHP 5.2/5.3 compatibility should be updated immediately.
|March 24, 2010: Network/server maintenance|
IMPORTANT: There will be a 3-hour maintenance window this evening from 9:00pm to 12:00am PST for network and server repairs/upgrades. A primary ethernet switch is being replaced, so all users will experience some network down time. Two servers also require cooling system maintenance, so these servers may experience outages of up to an hour.
Status messages will be posted at http://bk1.srohosting.com during maintenance.
03/25/2010: Maintenance complete. All servers are online and operating normally. Thanks for your patience.
|January 17, 2010: (3:15pm) Mail recovery|
One more step in return to normalcy: IMAP folders are being synchronized from the DG1 backup server to DB1. We weighed a number of options for doing such a large transfer without scheduled planning and in the end opted to err to the side of redundancy. ALL mail on the DG1 backup server is being copied to DB1. That means you may encounter duplicate messages and you may see folders/mail which you deleted after Jan 16th return.
We appologize for any inconvenience this may cause, however it is the best way to ensure nobody loses mail which was received between the last time they checked mail and the date the server went down.
|January 17, 2010: (4:00am) Database recovery|
In spite of drive array failure, we have made progress in recovering Mysql databases from the DG1 backup server. As installed, DB1 only had data from Thursday, January 14, however new data recovered from the failed MYSQL array is currently being uploaded. This data spans to a few hours before the failure on January 16th around noon.
Thank you for your patience during this time. Having a primary and backup server fail within weeks has been a bit trying, however we will continue to work to preserve our customer data to the greatest extent possible.
|January 17, 2010: (Midnight) DB1 server installed|
The DB1 hosting server which was scheduled to be re-installed later this month was installed tonight (Sat, Jan 16 at 10:30pm). This was necessary as the DG1 backup server's MYSQL storage failed unexpectedly a few hours prior. DB1 was fortunately synchronized during testing a few days ago, but there are several days of missing data. It will take some time to determine if/when the missing data on DG1 from the last few days will be restored.
News will be updated as data is restored.
|January 16, 2010: (6:30pm) Failover server failure|
Around 5pm this evening, the DG1 (failover) server experienced an unknown problem causing MYSQL database failure. This will cause database driven sites to fail and cause mail authentication errors.
Fortunately, we had already finished repairs to the DB1 server and will simply move the installation date forward (to tonight). Status will be updated here as soon as more information is available.
|January 4, 2010: Outage Status update|
All clients on the db1.srohosting.com server should have been successfully migrated to the mirror server ('dg1') by now. For most sites, this was automatic and painless.
IMPORTANT: Shared hosting customers not using SRO DNS need to point their domain DNS (A records) to the new server IP '18.104.22.168'. Basically, wherever you see '22.214.171.124', change the last number from '35' to '36'). If you have a dedicated IP address, it will not need to be changed.
UPDATE: According to our logs, some customers are still having trouble with IMAP over SSL on the new server. If your mail client is displaying errors, please let us know.
|January 2, 2010: Server Outage|
Due to problems on the DB1 hosting server, customers have been migrated to the DG1 failover server at this time. There may be intermittent outages for some time, so please be patient during the migration process. Please do not hesistate to email firstname.lastname@example.org with any additional information or trouble and we will respond as soon as possible. We will update this news page as status changes.
Note: As this is the second failure on DB1 in less than two months, the server is being removed from service and critical components replaced.
|November 10, 2009: Outage Status update|
Repairs have been made to TIG and DB1 servers and customers who were temporarily moved to failover servers have been quickly moved back to their primary server to avoid lengthy database resynchonization issues. We have temporarily suspended mirroring, so if you require resyncronization with the mirror server database (used between 7pm and 11pm November 9th), please let us know as soon as possible.
Please be advised, this outage may cause some email to be re-downloaded several times as switching between servers confuses the 'last read' timestamps.
|November 9, 2009: Outage Status update|
We are in the process of migrating customers to backup servers at this time. Some live failover backups were corrupted, so we are doing everything manually and this may take until morning to complete.
We thank you for your patience during this difficult time.
|November 9, 2009: Outage Status|
Outages have been reported on multiple servers (TIG, TC1, DB1) beginning at around 3pm this afternoon. Please be advised that we are still investigating the cause and are working to restore customer services to normal.