RSS Verzeichnis   Online RSS Reader   RSS Reader   RSS Generatoren   Was bedeutet RSS?   Interessante RSS Feeds   Impressum

RSS Verzeichnis - Holovaty.com

<< zurück

Holovaty.com

Adding /admin/ as a wink to Django developers

We've intentionally never put anything in Django that identifies the framework. There are no custom X-Generated-By headers, no hard-coded "Powered by Django" default templates -- no reliable way for a user of a site to know whether that site uses Django behind the scenes.

Back in the day, Django's session cookie was called "hotclub" by default (a reference to the name of Django Reinhardt's band, the Quintette du Hot Club de France), and checking for a presence of that cookie was a reasonable way of seeing whether a site used Django. But we changed the default on that back in 2005 because "hotclub" had been misinterpreted by some people as having some sort of porn-site meaning. :-)

I've always considered this anonymity a sign of a good framework -- Django stays out of the way and doesn't insert cruft -- but in some cases, I really want to know whether a site uses Django! In those cases, my usual plan is to go to /admin/ and see whether I get the classic Django admin login screen.

This isn't a perfect technique, because there's no guarantee that the site (a) uses the Django admin or (b) hooks it to /admin/ as opposed to some other URL. But it's pretty decent.

And that brings me to revealing an easter egg I've put in various sites over the years: the /admin/ redirect. I'm proud of the framework and want people to know that my sites use it, but I haven't used the Django admin on a site since working for washingtonpost.com back in 2007! So I implement /admin/ in my sites -- but only as a redirect...to the Django admin documentation. :-)

Hit holovaty.com/admin or soundslice.com/admin to see what I mean.

If you're looking for a way to send a wink to your fellow Django developers, here's how you can do it in your own urls.py:

from django.views.generic import RedirectView
from django.conf.urls import patterns, url

ADMIN_URL = 'https://docs.djangoproject.com/en/dev/ref/contrib/admin/'

urlpatterns = patterns('',
    # ...
    ('^admin/$', RedirectView.as_view(url=ADMIN_URL)),
)

If you do actually use the Django admin, just hook it to a separate URL. This gives you a tiny little benefit of security-by-obscurity.

Happy easter egging!

Using HTML5 prerendering to speed up a multi-page registration process

I recently began using a newish HTML5 feature called prerendering in the Soundslice registration process, and I'm very happy with how it's been working.

Prerendering is a way for you (as a web developer) to tell a web page to load (and render!) a second page in the background -- where the second page is one that's highly likely to be visited next.

Here's how it works. Say you have Page A and Page B, where the user always visits them in that order. You can put a bit of code in Page A's markup that says "prerender Page B right now." Thus, when the user visits Page A, her browser will download and render Page B in the background, such that when she clicks through to Page B, it's displayed nearly instantly, because all of the loading and rendering work has already been done. It's so fast it seems magical -- it must be experienced to be believed.

Prerendering is easy to do: just put a line of markup in Page A's , like so:

Note this apparently only works in Google Chrome at the moment, so the benefit is limited to that browser until other browsers implement it. But it's such an easy change that it's hard to justify not taking the few minutes to do it.

A multi-page form, as in a registration process, is a great use-case for prerendering. When you sign up for a Soundslice account, for example, first you get a form where you specify your username and password. Then, when you submit that, you get a form where you optionally upload an avatar and enter a profile. Then, when you submit that, you get the welcome/tutorial page. This is a very linear thing, where there's a very high probability that a user is visiting those pages directly in succession.

(Comparing the benefits of a single-page mega form vs. multi-page smaller forms is a separate discussion. Let's assume you have at least two pages in your site's registration process.)

In a multi-page registration process like this, prerendering works great -- as long as the pages don't depend on another. If the submitted data in Page A influences the markup of Page B, don't use prerendering between those two pages. (For example, this might be the case if the form on Page A includes a "username" field, and you then display "Welcome, [username]" on Page B.) On Soundslice registration pages, I'm in the clear because the pages don't depend on another.

Finally, there is a small downside to prerendering -- you're making your users load and render that next page whether they end up visiting the page or not. But in a tightly controlled multi-page experience like a registration process, you can be confident that most users will visit that next page. So if your pages don't depend on each other and there's a high probability they'll be visited in succession, I highly recommend this technique.

More info on prerendering: Google whitepaper, Ilya Grigorik article, Chris Ruppel article.

Why I left Heroku, and notes on my new AWS setup

On Friday, we migrated Soundslice from Heroku to direct use of Amazon Web Services (AWS). I'm very, very happy with this change and want to spread the word about how we did it and why you should consider it if you're in a similar position.

My Heroku experience

Soundslice had been on Heroku since the site launched in November 2012. I decided to use it for a few reasons:

• Being a sysadmin is not my thing. I don't enjoy it, and I'm not particularly good at it.
• Soundslice is a two-man operation (developer and designer), and my time is much better spent working on the product than doing sysadmin work.
• Heroku had the promise of easy setup and easy scaling in cases of high traffic.

While I was getting Soundslice up and running on Heroku, I ran into problems immediately. For one, their automatic detection of Python/Django didn't work. I had to rejigger my code four or five times ("Should settings.py go in this directory? In a subdirectory? In a sub-subdirectory?") in order for it to pick up my app -- and this auto-detection stuff is the kind of thing that's very hard to debug.

Then I spent a full day and a half (!) trying to get Django error emails working. I verified that the server could send email, and all the necessary code worked from the Python shell, but errors just didn't get emailed out from the app for some reason. I never did figure out the problem -- I ended up punting and using Sentry/Raven (highly recommended).

These experiences, along with a few other oddities, made me weary of Heroku, but I kept with it.

To its credit, Heroku handled the Soundslice launch well, with no issues -- and using heroku:ps scale from the command line was super cool. In December, Soundslice made it to the Reddit homepage and 350,000 people visited the site in a period of a few hours. Heroku handled it nicely, after I scaled up the number of dynos.

But over the next few months, I got burned a few more times.

First, in January, they broke deployment. Whenever I tried to deploy, I got ugly error messages. I ended up routing around their bug by installing a different "buildpack" thanks to a tip from Jacob, but this left a sour taste in my mouth.

Then, one April evening, I deployed my app, and Heroku decided to upgrade the Python version during the deploy, from 2.7.3 to 2.7.4. (In itself, that's vaguely upsetting, as I didn't request an upgrade. But my app code worked just as well on the new version, so I was OK with it.) When the deployment was done, my site was completely down -- a HARD failure with a very ugly Heroku error message being shown to my users. I had no idea what happened. I raced through my recent commits, looking for problems. I looked at the Heroku log output, and it just said some stuff about my "soundslice" package not being found. I ran the site locally to make sure it was working. It was working fine. I had deployed successfully earlier in the day, and I had made no fundamental changes to package layout.

After several minutes of this futzing around, with the site being completely down, after I had just sent the link to some potential partners who, for all I know, were evaluating the site that very moment -- I deployed again and the site worked again. So it was nothing on my end. Clearly just something busted with the Heroku deployment process.

That's when Heroku lost my trust. From then on, whenever I deployed, I got a little nervous that something bad would happen, out of my control.

Around the same time, Soundslice began using some Python modules with compiled C extensions and other various non-Python code that was not deployable on Heroku with their standard requirements.txt process. Heroku offers a way to compile and package binaries, which I used successfully, but it was more work using that proprietary process than running a simple apt-get command on a server I had root access to.

With all of this, I decided it was time to leave Heroku. I'm still using Heroku for this blog, and I might use it in the future for small/throwaway projects, but I personally wouldn't recommend using it for anything more substantial. Especially now that I know how easy it is to get a powerful AWS stack running.

My AWS setup

I'm lucky to be friends with Scott VanDenPlas, who was director of dev ops for the Obama reelection tech team -- you know, the one that got a ton of attention for being awesome. Scott helped me set up a fantastic infrastructure for Soundslice on AWS. Despite having used Amazon S3 and EC2 a fair amount over the years, I had no idea how powerful Amazon's full suite of services really were until Scott showed me. Unsolicited advertisement: You should definitely hire Scott if you need any AWS work done. He's one of the very best.

The way we set up Soundslice is relatively simple. We made a custom AMI with our code/dependencies, then set up an Elastic Load Balancer with auto-scaling rules that instantiate app servers from that AMI based on load. I also converted the app to use MySQL. In detail:

Step 1: "Bake" an AMI. I grabbed an existing vanilla Ubuntu AMI (basically a frozen image of a Linux box) and installed the various packages Soundslice needs with apt-get and pip. I also compiled a few bits of code I needed that aren't in apt-get, and I got our app's code on there by cloning our Git repository. After that instance had all my code/dependencies on it, I created an AMI from it ("Create Image (EBS AMI)" in the EC2 dashboard).

Step 2: Set up auto-scaling rules. This is the real magic. We configured a load balancer (using Amazon ELB) to automatically spawn app servers based on load. This involves setting up things called "Launch configurations" and "scaling policies" and "metric alarms." Check out my Python code here to see the details. Basically, Amazon constantly monitors the app servers, and if any of them reaches a certain CPU usage, Amazon will automatically launch X new server(s) and associate them with the load balancer when they're up and running. Same thing applies if traffic levels go down and you need to terminate an instance or two. It's awesome.

Step 3: Change app not to use shared cache. Up until the AWS migration, Soundslice used memcache for Django session data. This introduces a few wrinkles in an auto-scaled environment, because it means each server needs access to a common memcache instance. Rather than have to deal with that, I changed the app to use cookie-based sessions, so that session data is stored in signed cookies rather than in memcache. This way, the web app servers don't need to share any state (other than the database). Plus it's faster for end users because the app doesn't have to hit memcache for session data.

Step 4: Migrate to MySQL. Eeeek, I know. I have been a die-hard PostgreSQL fan since Frank Wiles showed me the light circa 2003. But the only way to use Postgres on AWS is to do the maintenance/scaling yourself...and my distaste for doing sysadmin work is greater than my distate for MySQL. :-) Amazon offers RDS, which is basically hosted MySQL, with point-and-click replication. I fell in love with it the moment I scaled it from one to two availability zones with a couple of clicks on the AWS admin console. The simplicity is amazing.

Step 5: Add nice API with Fabric. Deployment was stupidly simple with Heroku, but it's easy to make it equally simple using a custom AWS environment -- I just had to do some upfront work by writing Fabric tasks. The key is, because you don't know how many servers you have at a given moment, or what their host names are, you query the Amazon API (using the excellent boto library) to get the hostnames dynamically. See here for the relevant parts of my fabfile.

Ongoing: Update AMI as needed. Whenever there's a new bit of code that my app needs -- say, a new apt-get package -- I make a one-off instance of the AMI, install the package, then freeze it as a new AMI. Then I associated the load balancer with the new AMI, and each new app server from then on will use the new AMI. I can force existing instances to use the new AMI by simply terminating them in the Amazon console; the load balancer will detect that they're terminated and, based on the scaling rules, will bring up a new instance with the new AMI.

Another approach would be to use Chef or Puppet to automatically install the necessary packages on each new server at instantiation time, instead of "baking" the packages into the AMI itself. We opted not to do this, because it was unnecessary complexity. The app is simple enough that the baked-AMI approach works nicely.

Put all this together, and you have a very powerful setup that I would argue is just as easy to use as Heroku (once it's set up!), with the full power of root access on your boxes, the ability to install whatever you want, set your scaling rules, etc. Try it!

In defense of

My friend and fellow Chicagoan Evan Miller wrote an excellent blog post over the weekend: Why I Develop For The Mac. It's full of great reasons why his software (which is also excellent, by the way) was written for the desktop, despite the fact that he's a web developer, even the creator of an Erlang web framework.

But I'm compelled to respond to it, specifically his statements about :

large areas seem laggy on most browsers

So I'm left with , and is slow.

I have become intimately familiar with while developing Soundslice. I'd even venture to say Soundslice is one of the most advanced uses of on the web that's not a tech demo -- i.e., it's an application that normal people use. The site uses not one, but nine elements stacked on top of each other to make a very rich UI, sort of like Photoshop for guitar tabs. (For a flashy demo of how those canvases interact, watch the tech talk I gave at 37signals, specifically the bit starting at 10:20.)

Here's what I've learned: is not slow. In fact, I've been continually surprised by how fast it is -- as long as you take care to do things right. Evan's article mentions the "magical" sensation of instantaneous feedback; I invite you to play with the zoom slider on any Soundslice page (example) to experience this same magic, all drawn dynamically with .

Of course, is certainly not as fast as the lower-level drawing routines that you can use if you develop a desktop app. No question. But it's fast enough that, unless you're doing something relatively insane, you'll be totally fine.

On Soundslice, we're drawing guitar-chord charts completely on the fly (again, see an example), which is a relatively involved drawing routine -- and it's still near-instant performance. That's across all modern browsers (Chrome, Safari, Firefox and IE 10).

Here are some specific tips I've picked up to make performance really shine.

Use requestAnimationFrame

Above all else, do this.

It's a JavaScript API designed to fix a very specific problem: your computer screen can only be redrawn a certain number of times per second (the "refresh rate"), so any calculations that redraw more often than your refresh rate are wasteful.

For example, say you have an event such as mousemove that results in a redraw. A mousemove might happen hundreds of times per second, but your screen might only refresh, say, 75 times per second (75 Hz). That means, if your code is naively written, it will try to redraw several times within each actual opportunity to redraw (hundreds of times per second vs. 75 actual redraw opportunities per second).

The requestAnimationFrame API solves this by letting you say, "Execute this code the next time a redraw happens." Which saves your browser from having to do unnecessary work.

When I added this to Soundslice, the site became dramatically faster and more responsive. Here's more info about how to use the API.

Stack canvases

Above, I linked to a video of a tech talk I gave about Soundslice. In that talk, I demonstrated how Soundslice uses several different elements, stacked on top of each other as layers, for maximum performance -- and for nice, clean code. Definitely watch the demo at around 10:20 in the video to get a sense of it.

I'm planning to write a separate blog post about this, but the Cliff's Notes version is that you can stack transparent elements on top of each other so that you only have to redraw the ones that need to change.

For example, on Soundslice, there's a separate for the playhead -- the vertical orange line that tracks the currently played moment of the video. That's a separate with a z-index above the other ones, so that redrawing it doesn't require redrawing any of the other stuff. The less you have to redraw, the better.

Bunch calls to fillStyle

When you draw on , you first have to tell it which color you're using. You can do that by setting the "fillStyle." It turns out that, each time you change the fillStyle, there's a slight performance penalty. Therefore, you can squeeze out some extra performance by bunching your calls to fillStyle -- that is, rather than drawing a gray thing, then an orange thing, then a gray thing again, you should draw all the gray things, then draw all the orange things.

For example, Soundslice, which is all about annotating YouTube videos, needs to draw dozens, sometimes hundreds, of annotations on the screen at a time. Each annotation might use several different colors -- the text color, the border color, the line color, etc.

My original implementation looped over each annotation and drew each one independently, which resulted in two to five fillStyle calls for each annotation. I changed this to bunch the fillStyles across all annotations -- so that all of the light grays were drawn at the same time, then all the dark grays, etc. -- and the drawing got a few dozen milliseconds faster.

For more background, see the "Avoid unnecessary canvas state changes" section in this great HTML5 Rocks article.

Cache text rendering

In profiling, I've found that rendering text on is my next big rendering-related bottleneck on Soundslice. I haven't done this yet, but I'm planning to come up with a way of caching the results of fillText, possibly using this technique.

Final thoughts

A decent argument in Evan's favor is: "Well, if is only fast if you use these various hacks, it's not really fast, then, is it?"

Two thoughts on that.

First, well, sure! I'd love it if was super fast right out of the box, without needing to use these techniques. No doubt about it. But the reality is, it is fast enough, if you put in the work.

Second, there's the bigger question -- a defining question for the current generation of web developers -- which is: web or native app? I am squarely in the web camp, both for philosophical reasons (such as openness) and practical reasons (such as the fact that Soundslice has only one developer and one designer, and we can't justify building separate apps for separate platforms).

What I love about is that it lets us make desktop-quality apps right in the browser, so we can get the benefits of being "of the web" along with the benefits of amazing, fast graphics. Fear not, my friends: is great.

UPDATE, May 7, 2013: Evan has posted a thoughtful follow up, reacting to this.

Boston.com's viral explosion videos

Today's explosions at the Boston Marathon were heartbreaking and horrifying. But a mundane detail in one site's news coverage put me over the edge.

I saw a few tweets saying that Boston.com, the hometown newspaper, had high-quality video of the explosion. So I clicked through to one of the links (warning: disturbing video) and couldn't help but notice: the page's