Sign Up

May 17, 2011

State of DropDAV at Version 2.0

A Brief Historical Aside

Well, this has been a long time coming (three months, to be exact). When we launched DropDAV last December, it worked well for our use-cases, so we thought it was great. Then thousands of you started using it, and while a lot of you thought it was great too, many of you were kind enough to expose our faulty assumptions and point out the glaring bugs. Three months ago, we began work in earnest on a new version of our WebDAV server that would address all those bugs, and more.

We knew many WebDAV clients have idiosyncrasies, but we never anticipated the breadth of clients, each with its own oddities, that would be used with DropDAV. In terms of user-base and per-user-usage growth, they've both exceeded our estimates by ten fold. And to say we've been floored by the international response would be a tremendous understatement. About 40% of our users are in the United States (with us), and more than half are in non-English speaking countries.

In designing a new version we knew we needed extensible request parsing and response formatting to address all our users' different WebDAV clients, much more durable infrastructure to cope with increasing demand, and vastly improved support for UTF-8 character sets to enhance our international users' experience.

So, Version 2.0? What's new?

The WebDAV codebase was a mess. It inconsistently supported UTF-8 and XML Special Characters. It didn't support WebDAV Class 2. It needed a complete re-write, a re-think that included the first real designing of the service.

We identified inconsistencies in the way PHP and the OAuth Pear plugin worked with UTF-8 strings. We created a taxonomy of WebDAV clients, sorting out which ones behaved similarly. We integrated in support for WebDAV locking to comply with the Class 2 specifications.

In the end, we bridged our legacy implementations of WebDAV's hairier functions (written in PHP) with the native UTF-8 support and better Dropbox client in Python. We settled on an implementation that includes extensible request parsing in PHP, communication with Dropbox in Python, then extensible response formatting back in PHP. It's a lot cleaner than it sounds, and it works really, really well for the entire UTF-8 character set and XML Special characters.

Class 2 support is still a work in progress. In truth, it's somewhat of a hack in that Dropbox doesn't support locking, so we could only create the illusion of locking for our WebDAV clients. We think we've done that well enough to roll out and will continue improving it to eventually pass the Litmus test in the coming weeks.

Expertly designing and implementing all the Version 2.0 improvements was very time consuming. Combined with providing first-class customer support, DropDAV quickly grew into a full-time operation. Our estimates about the distribution of users we'd serve with paid-for Dropbox accounts were way off, and the "freemium" revenue model wasn't sufficient support for our team. Further, DropDAV usage was poorly correlated to Dropbox account size, so non-paying users were sopping up resources, and performance degraded for our paying users.

In response, we did some soul searching about the utility of our improved offering and decided that bucking the freemium trend by going with a flat-fee for all of our users was a better fit. We've changed our pricing model to a fixed $5 monthly fee, following our no-commitment 14-day free trial, and we're confident we can deliver more than $5 each month in utility to all of our users. Your monthly fees go directly towards supporting new infrastructure costs (discussed further down), continuing WebDAV improvement, and, as always, first-class customer support.

In service of our new flat-fee pricing, we've implemented a super simple recurring billing system. We're using Stripe's rock-solid payment processing infrastructure to perform payments and store sensitive credit card data. Nevertheless, we've contracted an independent security consultant to verify our PCI-DSS compliance on a quarterly basis so we can all sleep soundly knowing DropDAV users' payment information will never be in jeopardy.

Keeping with the security theme, we've re-designed our signup flow to support connecting to Dropbox via OAuth. DropDAV users have long been able to change their passwords to anything they desire, and now new users won't need to reveal their Dropbox passwords to us during signup. We're still encrypting passwords with strong, unique 40-character random salts using the bcrypt hashing scheme, which, unfortunately, is much more secure than industry standards. DropDAV user data has never faced compromise, and thanks to our continued vigilance (bordering on paranoia), it's very, very unlikely that it ever will be.

In the five months since we launched publicly, we've had two incidents of un-expected downtime. The first lasted about 30 hours, initially due to SoftLayer's poorly implemented CloudLayer hosting service, and prolonged by our immature backup strategy. The second, about five days later, lasted about six hours while the public internet connection between our WebDAV server in Dallas and our database server in Virginia was unreliable. As a result, our uptime has been 99%.

Version 2.0 was as much about improving the WebDAV code as it has been about improving the underlying server infrastructure. We started on a single virtual private server at Media Temple in Virginia. After a few weeks we transitioned the operation to a pair of CloudLayer instances in Dallas to be closer to Dropbox's servers. Last weekend we completed a migration to a cluster of venerable Linode instances for the WebDAV services, and a (monster) dedicated web and database server hosted by DigitalOne.

Reduced physical proximity to Dropbox and aggressive credential caching have boosted WebDAV speed by a factor of two since our introduction. Industry-best-practice cluster architecture, with load-balancing and automated failover protection will help us add some more 9's to our uptime percentage. Database replication, and frequent backups will prevent data loss. Standardized server environments and code deployment mean we can scale WebDAV services in as little as 15 minutes any time resource demand spikes.

We recognize DropDAV has become an essential tool in many of our users' professional work-flows, and we're committed to maintaining a fast, reliable, premium service now and in the future.

Closing Sentiments

We'd all like to thank the expert development and support team at the OmniGroup. They've been instrumental in helping us improve our WebDAV service with regard to XML Special Character encoding. They're great partners who produce stupendous software. We've also had terrific luck interacting with the fine people at Midnight Inbox; their iPad app is gorgeous, and we look forward to the release of its iPhone and Mac companions. It's been a pleasure servicing our mutual users.

To the more than 35,000 past and present DropDAV users, we are sincerely grateful for the struggle supporting your almost insatiable demand for a broadly functional, highly available connection between WebDAV and Dropbox. You are each directly responsible for immeasurable growth in our team's technical knowledge. We've never been pushed harder to produce something really amazing. We thank you; our careers thank you. We look forward to serving each of you in the future, and continue to welcome your feedback on twitter @DropDAV and via email sent to contact@dropdav.com (tell us what WebDAV clients you're using and how we can support them better!).

Sincerely yours,

Zane and Delilah

The DropDAV Team.