At Zillow we are trying to launch new code to our website 24 times a year, or approximately every other week. By new code I’m not including bug fixes. I’m talking about new features, services, views, reports, and such thing that really change the customers experience on our site. This also includes major software backend services, that may not be plainly visible but fundamentally and/or architecturally change the way we provide access to our vast amounts of data. Bug fixes get rolled out each day for a couple of days following the Release.
Today was a Release day, but the new code and data deployment had to be backed out before it actually Released To Web (RTW). We usually pull half our “services” from live use and deploy the new bits. When the new stuff is “prop’ed” and ready we do functional testing. It was during this time that problems were discovered and everything that had been updated needed to be reverted back to the old software. Time consuming and sometime error prone after an already long days work.
Data had also been updated but it was first thought those changes were 100% backward compatible with the old software release code. The short of it is, “wrong. we are seeing all kinds of data exceptions!”
Now a long day has become a real long day, and sliding downhill into the abyss of exhaustion.
Thursday is going to be an interesting day in the office.