Announcing StaticGenerator for Django
January 7, 2008
Want to speed up your Django app? StaticGenerator is a Python class for Django that makes it easy to create static files for lightning fast performance. It accepts strings (URL), Models (class or instance), Managers, and QuerySets in a simple syntax.
Please go to the project page for more details and a download.
Add your comment
No HTML; Only URLs and line breaks are converted.
Comments
Very cool!
This will be great for people looking to scale their Django applications or squeeze more out of a small VPS.
Posted by Pete
Nice. Great example of (making your vps) work smarter, not harder.
Posted by nikolaj
Which django caching are you using? If it isn't much work (which it shouldn't be, you can just change the type in the settings), could you add memcache, filesystem and db?
Looks great.
Posted by mrtron
The caching used is memcached, which is fastest as far as I can tell.
Posted by SuperJared
That's amazing. Looks super useful! Thanks! (Any way this might be integreated into Django, or I guess it'll be used by other apps integrating it)
Posted by Eric Holscher
Great results, but why not have Apache cache to memory or flat files using mod_cache for you?
Then you need not modify your application at all?
see
http://httpd.apache.org/docs/2.0/mod/...
Posted by Noah
Firstly I think Apache should do as little work as possible. It is a significant resource hog in general which is why it's best to have Nginx up front anyway.
Secondly the granularity of StaticGenerator is significantly easier to administer than mod_cache.
But that's just my opinion. ;)
Posted by SuperJared
Cool. I was planning to write something just like this...
@Noah: That's certainly an option if you are already using Apache, but what if you want all static files on some other server? Right now most of my static files are hosted from my Amazon S3 account and that's much cheaper than most VPS bandwidth.
Posted by Max Battcher
Any similar plugin for Rails ?
Posted by Lakshan
A brilliant idea! :)
Posted by james
One addition that might also reduce server load would be if the static file was gzipped on generation, then configure nginx to deliver pre-compressed files rather than gzipping on the fly.
Posted by huxley
@huxley: The way that this works, from how I read it, if you have the Gzip middleware (in django.contrib) installed and activated it should seamlessly use that... You should just need to override it a little bit to add in the correct ACCEPT header to the HttpRequest in get_content_from_path. The only issue here is that some legacy browsers still don't support Gzipped files so you need to be careful and balance the server load of server gzip versus statically generated gzip.
Posted by Max Battcher
@ Max,
neat, I'll have to look at the Gzip middleware.
re: legacy browsers, yeah, you'd still need to do content negotiation, with some extra directives to serve uncompressed files to problematic browsers. Depending on the mix of browsers and UAs it might be a wash.
One option might be to decompress the gzipped files on-the-fly to UAs which can't handle gzipped files (mod_gunzip for Apache 1 could do this, but I'm not that familiar with nginx so I couldn't say if it could be configured to do that) and another would be to generate gzipped and uncompressed versions to serve as needed (which unfortunately adds clutter and increases storage requirements by doubling the number of files).
Posted by huxley
How did you use the cache for the benchmark you published ? CacheMiddleWare ? cache the full view ? only caching result sets ?
Posted by Maxime Biais
Or better, can you publish the code that help you to generate this benchmark ?
Posted by Maxime Biais
Do you have a technical rational as to why Apache should do as little work as possible? The static file performance of Apache isn't all that bad when it's tuned properly.
As for the granularity I don't fully agree. Apache allows for cache on a url by url basis and for whole prefixes, allowing you to write 1 line to cache 20 views, and 1 more line for that 1 exclusion.
I think your method would be very good however for having multiple nginx instances serve the static content from a single Apache instance, as it is my understanding that Apache instances don't share cache well.
Posted by Noah
Apache uses more cpu and ram than Nginx in any configuration I've seen. And when you have mod_python loaded for the URL you're looking at, that gap can get even larger. When you're benching Apache and Nginx, take a look at the CPU usage especially.
Posted by SuperJared
On the project page, and in the version string in the generator.py file, it lists the date as January 2007.
Should that be 2008?
Posted by eliott
Well that was dumb of me. Thanks Eliott!
Posted by SuperJared
Look at WebAlchemy
http://www.mysoftparade.com/blog/weba...
it publish Django files and has universal cache invalidation.
Posted by Dima Dogadaylo
WebAlchemy doesn't appear elegant.
Posted by SuperJared
Dude,
I love you, I want to marry you, you are my hero :)
thanks for this great work!
Posted by mama