Design
TEST FROM HOME UNDER LOAD
I learned an important lesson today: never put on hold a book from the library that's already listed on the shelves. Just go get it. If it's on hold, they'll take it off the shelves and put it "into the system," which means you won't actually be able to get it for 24 hours.
I also learned another important lesson. Always start a few load-creating automated tests, then go to the most hostile wi-fi enabled cafe you know and try to use your web application from there.
The King County Library System recently spent a lot of money upgrading their computerized system. It looks like there was a desperate attempt to make it "more 2.0" by throwing a thin layer of AJAX into the mix: searches now update page partials rather than whole pages, and various divs for holds, fines, checkouts, and wishlists come and go as you click on the appropriate tabs. All fairly unimpressive, by my standards. The developers, whomever they were, used Adobe's library as their basis as the Javascript is full of Dreamweaver-related ("MM_") tags. Where they didn't use Dreamweaver, they wrote the code themselves; there's a strong Not Invented Here feel to much of the Javascript they used.
And I'm sure, when they first started out, it looked okay. They tried to search for a few books, the spinner spun for a second and then the page rendered. Only now, under load, the spinner spins for a several seconds, and then the page goes blank. A long pause later, the page begins drawing again. The clue that the spinner needs to stop and the signal that the data is ready to be drawn are two different signals.
There are several examples of this, but the book search is the most egregious. All in all, the upgrade has made a lot of people mad: renderers don't work, often fail to deliver, time out without excuse. And this doesn't have to be hard. KCLS has slightly under a million items, and had 21 million check-ins last year, or about 3 a second during business hours. Prodigious, but seriously middle-tier as far as businesses go. A modern implementation of the website should not be as difficult as they've made it.