RSS Verzeichnis - Holovaty.com
Adrian Holovaty's blog.
We've intentionally never put anything in Django that identifies the framework. There are no custom
Back in the day, Django's session cookie was called "hotclub" by default (a reference to the name of Django Reinhardt's band, the Quintette du Hot Club de France), and checking for a presence of that cookie was a reasonable way of seeing whether a site used Django. But we changed the default on that back in 2005 because "hotclub" had been misinterpreted by some people as having some sort of porn-site meaning. :-)
I've always considered this anonymity a sign of a good framework -- Django stays out of the way and doesn't insert cruft -- but in some cases, I really want to know whether a site uses Django! In those cases, my usual plan is to go to /admin/ and see whether I get the classic Django admin login screen.
This isn't a perfect technique, because there's no guarantee that the site (a) uses the Django admin or (b) hooks it to /admin/ as opposed to some other URL. But it's pretty decent.
And that brings me to revealing an easter egg I've put in various sites over the years: the /admin/ redirect. I'm proud of the framework and want people to know that my sites use it, but I haven't used the Django admin on a site since working for washingtonpost.com back in 2007! So I implement /admin/ in my sites -- but only as a redirect...to the Django admin documentation. :-)
Hit holovaty.com/admin or soundslice.com/admin to see what I mean.
If you're looking for a way to send a wink to your fellow Django developers, here's how you can do it in your own urls.py:
If you do actually use the Django admin, just hook it to a separate URL. This gives you a tiny little benefit of security-by-obscurity.
Happy easter egging!
I recently began using a newish HTML5 feature called prerendering in the Soundslice registration process, and I'm very happy with how it's been working.
Prerendering is a way for you (as a web developer) to tell a web page to load (and render!) a second page in the background -- where the second page is one that's highly likely to be visited next.
Here's how it works. Say you have Page A and Page B, where the user always visits them in that order. You can put a bit of code in Page A's markup that says "prerender Page B right now." Thus, when the user visits Page A, her browser will download and render Page B in the background, such that when she clicks through to Page B, it's displayed nearly instantly, because all of the loading and rendering work has already been done. It's so fast it seems magical -- it must be experienced to be believed.
Prerendering is easy to do: just put a line of markup in Page A's
Note this apparently only works in Google Chrome at the moment, so the benefit is limited to that browser until other browsers implement it. But it's such an easy change that it's hard to justify not taking the few minutes to do it.
A multi-page form, as in a registration process, is a great use-case for prerendering. When you sign up for a Soundslice account, for example, first you get a form where you specify your username and password. Then, when you submit that, you get a form where you optionally upload an avatar and enter a profile. Then, when you submit that, you get the welcome/tutorial page. This is a very linear thing, where there's a very high probability that a user is visiting those pages directly in succession.
(Comparing the benefits of a single-page mega form vs. multi-page smaller forms is a separate discussion. Let's assume you have at least two pages in your site's registration process.)
In a multi-page registration process like this, prerendering works great -- as long as the pages don't depend on another. If the submitted data in Page A influences the markup of Page B, don't use prerendering between those two pages. (For example, this might be the case if the form on Page A includes a "username" field, and you then display "Welcome, [username]" on Page B.) On Soundslice registration pages, I'm in the clear because the pages don't depend on another.
Finally, there is a small downside to prerendering -- you're making your users load and render that next page whether they end up visiting the page or not. But in a tightly controlled multi-page experience like a registration process, you can be confident that most users will visit that next page. So if your pages don't depend on each other and there's a high probability they'll be visited in succession, I highly recommend this technique.
On Friday, we migrated Soundslice from Heroku to direct use of Amazon Web Services (AWS). I'm very, very happy with this change and want to spread the word about how we did it and why you should consider it if you're in a similar position.
My Heroku experience
Soundslice had been on Heroku since the site launched in November 2012. I decided to use it for a few reasons:
While I was getting Soundslice up and running on Heroku, I ran into problems immediately. For one, their automatic detection of Python/Django didn't work. I had to rejigger my code four or five times ("Should settings.py go in this directory? In a subdirectory? In a sub-subdirectory?") in order for it to pick up my app -- and this auto-detection stuff is the kind of thing that's very hard to debug.
Then I spent a full day and a half (!) trying to get Django error emails working. I verified that the server could send email, and all the necessary code worked from the Python shell, but errors just didn't get emailed out from the app for some reason. I never did figure out the problem -- I ended up punting and using Sentry/Raven (highly recommended).
These experiences, along with a few other oddities, made me weary of Heroku, but I kept with it.
To its credit, Heroku handled the Soundslice launch well, with no issues -- and using heroku:ps scale from the command line was super cool. In December, Soundslice made it to the Reddit homepage and 350,000 people visited the site in a period of a few hours. Heroku handled it nicely, after I scaled up the number of dynos.
But over the next few months, I got burned a few more times.
First, in January, they broke deployment. Whenever I tried to deploy, I got ugly error messages. I ended up routing around their bug by installing a different "buildpack" thanks to a tip from Jacob, but this left a sour taste in my mouth.
Then, one April evening, I deployed my app, and Heroku decided to upgrade the Python version during the deploy, from 2.7.3 to 2.7.4. (In itself, that's vaguely upsetting, as I didn't request an upgrade. But my app code worked just as well on the new version, so I was OK with it.) When the deployment was done, my site was completely down -- a HARD failure with a very ugly Heroku error message being shown to my users. I had no idea what happened. I raced through my recent commits, looking for problems. I looked at the Heroku log output, and it just said some stuff about my "soundslice" package not being found. I ran the site locally to make sure it was working. It was working fine. I had deployed successfully earlier in the day, and I had made no fundamental changes to package layout.
After several minutes of this futzing around, with the site being completely down, after I had just sent the link to some potential partners who, for all I know, were evaluating the site that very moment -- I deployed again and the site worked again. So it was nothing on my end. Clearly just something busted with the Heroku deployment process.
That's when Heroku lost my trust. From then on, whenever I deployed, I got a little nervous that something bad would happen, out of my control.
Around the same time, Soundslice began using some Python modules with compiled C extensions and other various non-Python code that was not deployable on Heroku with their standard requirements.txt process. Heroku offers a way to compile and package binaries, which I used successfully, but it was more work using that proprietary process than running a simple apt-get command on a server I had root access to.
With all of this, I decided it was time to leave Heroku. I'm still using Heroku for this blog, and I might use it in the future for small/throwaway projects, but I personally wouldn't recommend using it for anything more substantial. Especially now that I know how easy it is to get a powerful AWS stack running.
My AWS setup
I'm lucky to be friends with Scott VanDenPlas, who was director of dev ops for the Obama reelection tech team -- you know, the one that got a ton of attention for being awesome. Scott helped me set up a fantastic infrastructure for Soundslice on AWS. Despite having used Amazon S3 and EC2 a fair amount over the years, I had no idea how powerful Amazon's full suite of services really were until Scott showed me. Unsolicited advertisement: You should definitely hire Scott if you need any AWS work done. He's one of the very best.
The way we set up Soundslice is relatively simple. We made a custom AMI with our code/dependencies, then set up an Elastic Load Balancer with auto-scaling rules that instantiate app servers from that AMI based on load. I also converted the app to use MySQL. In detail:
Step 1: "Bake" an AMI. I grabbed an existing vanilla Ubuntu AMI (basically a frozen image of a Linux box) and installed the various packages Soundslice needs with apt-get and pip. I also compiled a few bits of code I needed that aren't in apt-get, and I got our app's code on there by cloning our Git repository. After that instance had all my code/dependencies on it, I created an AMI from it ("Create Image (EBS AMI)" in the EC2 dashboard).
Step 2: Set up auto-scaling rules. This is the real magic. We configured a load balancer (using Amazon ELB) to automatically spawn app servers based on load. This involves setting up things called "Launch configurations" and "scaling policies" and "metric alarms." Check out my Python code here to see the details. Basically, Amazon constantly monitors the app servers, and if any of them reaches a certain CPU usage, Amazon will automatically launch X new server(s) and associate them with the load balancer when they're up and running. Same thing applies if traffic levels go down and you need to terminate an instance or two. It's awesome.
Step 3: Change app not to use shared cache. Up until the AWS migration, Soundslice used memcache for Django session data. This introduces a few wrinkles in an auto-scaled environment, because it means each server needs access to a common memcache instance. Rather than have to deal with that, I changed the app to use cookie-based sessions, so that session data is stored in signed cookies rather than in memcache. This way, the web app servers don't need to share any state (other than the database). Plus it's faster for end users because the app doesn't have to hit memcache for session data.
Step 4: Migrate to MySQL. Eeeek, I know. I have been a die-hard PostgreSQL fan since Frank Wiles showed me the light circa 2003. But the only way to use Postgres on AWS is to do the maintenance/scaling yourself...and my distaste for doing sysadmin work is greater than my distate for MySQL. :-) Amazon offers RDS, which is basically hosted MySQL, with point-and-click replication. I fell in love with it the moment I scaled it from one to two availability zones with a couple of clicks on the AWS admin console. The simplicity is amazing.
Step 5: Add nice API with Fabric. Deployment was stupidly simple with Heroku, but it's easy to make it equally simple using a custom AWS environment -- I just had to do some upfront work by writing Fabric tasks. The key is, because you don't know how many servers you have at a given moment, or what their host names are, you query the Amazon API (using the excellent boto library) to get the hostnames dynamically. See here for the relevant parts of my fabfile.
Ongoing: Update AMI as needed. Whenever there's a new bit of code that my app needs -- say, a new apt-get package -- I make a one-off instance of the AMI, install the package, then freeze it as a new AMI. Then I associated the load balancer with the new AMI, and each new app server from then on will use the new AMI. I can force existing instances to use the new AMI by simply terminating them in the Amazon console; the load balancer will detect that they're terminated and, based on the scaling rules, will bring up a new instance with the new AMI.
Another approach would be to use Chef or Puppet to automatically install the necessary packages on each new server at instantiation time, instead of "baking" the packages into the AMI itself. We opted not to do this, because it was unnecessary complexity. The app is simple enough that the baked-AMI approach works nicely.
Put all this together, and you have a very powerful setup that I would argue is just as easy to use as Heroku (once it's set up!), with the full power of root access on your boxes, the ability to install whatever you want, set your scaling rules, etc. Try it!
My friend and fellow Chicagoan Evan Miller wrote an excellent blog post over the weekend: Why I Develop For The Mac. It's full of great reasons why his software (which is also excellent, by the way) was written for the desktop, despite the fact that he's a web developer, even the creator of an Erlang web framework.
But I'm compelled to respond to it, specifically his statements about
So I'm left with
I have become intimately familiar with
Here's what I've learned:
On Soundslice, we're drawing guitar-chord charts completely on the fly (again, see an example), which is a relatively involved drawing routine -- and it's still near-instant performance. That's across all modern browsers (Chrome, Safari, Firefox and IE 10).
Here are some specific tips I've picked up to make
Above all else, do this.
For example, say you have an event such as mousemove that results in a
The requestAnimationFrame API solves this by letting you say, "Execute this code the next time a redraw happens." Which saves your browser from having to do unnecessary work.
When I added this to Soundslice, the site became dramatically faster and more responsive. Here's more info about how to use the API.
Above, I linked to a video of a tech talk I gave about Soundslice. In that talk, I demonstrated how Soundslice uses several different
I'm planning to write a separate blog post about this, but the Cliff's Notes version is that you can stack transparent
For example, on Soundslice, there's a separate
Bunch calls to fillStyle
When you draw on
For example, Soundslice, which is all about annotating YouTube videos, needs to draw dozens, sometimes hundreds, of annotations on the screen at a time. Each annotation might use several different colors -- the text color, the border color, the line color, etc.
My original implementation looped over each annotation and drew each one independently, which resulted in two to five fillStyle calls for each annotation. I changed this to bunch the fillStyles across all annotations -- so that all of the light grays were drawn at the same time, then all the dark grays, etc. -- and the drawing got a few dozen milliseconds faster.
For more background, see the "Avoid unnecessary canvas state changes" section in this great HTML5 Rocks article.
Cache text rendering
In profiling, I've found that rendering text on
A decent argument in Evan's favor is: "Well, if
Two thoughts on that.
First, well, sure! I'd love it if
Second, there's the bigger question -- a defining question for the current generation of web developers -- which is: web or native app? I am squarely in the web camp, both for philosophical reasons (such as openness) and practical reasons (such as the fact that Soundslice has only one developer and one designer, and we can't justify building separate apps for separate platforms).
What I love about
UPDATE, May 7, 2013: Evan has posted a thoughtful follow up, reacting to this.
Today's explosions at the Boston Marathon were heartbreaking and horrifying. But a mundane detail in one site's news coverage put me over the edge.
I saw a few tweets saying that Boston.com, the hometown newspaper, had high-quality video of the explosion. So I clicked through to one of the links (warning: disturbing video) and couldn't help but notice: the page's
I clicked through several related videos. All had the same title and URL pattern.
Putting aside the fact that if you have to say a video is viral then you've already lost, advertising these explosion videos as "viral" is sad. It's cheap, it's wildly inappropriate -- and, frankly, for a news organization with the power of the Boston Globe -- it's pathetic and beneath them.
Now, this is obviously a content-management system problem, not a case of a Boston.com editor saying, "Here's hoping this goes viral!" Clearly there's a template somewhere with a default title "Boston.com viral video page," and the URLs are hard-coded to include "viral_page". And let's keep some perspective: overall, what's important here is the news event itself, not the details of how one particular CMS titled a video.
And I don't mean to berate Boston.com. As somebody who used to work for news sites, I have been in their shoes. There are many, many more important things for a news organization to do in this situation -- vet new information, feverishly post updates and, well, keep the site up and running under huge traffic levels! And to their credit, after I posted a snarky tweet about it, a few Boston.com folks responded within minutes, saying they were migrating videos to YouTube and didn't have much control over their software.
But I'm writing this because I think there's an important lesson here for makers of Web products. That lesson is: tools will be repurposed, and they should be designed with that in mind.
Whoever designed this video gallery system (sounds like it might've been a vendor) was too narrow-minded. Judging by the URL and default page title, it was meant to be a clearinghouse for viral videos, but it ended up being used for all types of videos. They ought to have designed for broad use, not specific use.
If they truly wanted to build a "viral video" platform, one that lived up to its rather lofty URL, it should have included specific features that made it work particularly well for viral videos and not for other videos. But that doesn't make any sense, because, when it comes to publishing-system needs, nothing distinguishes a viral video from your run-of-the-mill news video. It's completely reasonable that Boston.com would start using their "viral video" publishing tool to publish any video. Which I assume is what happened.
So if you're building a CMS-ish thing, keep that in mind. Think of the most wildly inappropriate thing that your system might have to handle some day, and make sure your system is still robust and appropriate.
(See Peter Harkins' rules of database app aging for some excellent related points from a database perspective.)