Summary
This is an interim response to the discussion that followed my previous web frameworks blog. I need to do much more research, but I have some quick thoughts.
Advertisement
Let me thank everybody for participating in the web frameworks discussion. I haven't been able to digest all the comments yet, and I've yet to download a single one of the many recommendations (except for web.py, because it was only one file). But I've skimmed a lot of the discussion, and tried to follow a few tutorials (many of which are unfinished, leaving me with an unsatisfied cliffhanger feeling).
So, while I educate myself more (and take care of other duties), here are some thoughts that are bubbling through my head at this point.
Maybe the current crop of Python web frameworks (as well as Rails BTW) have it all wrong. Maybe the WSGI folks are the only ones who are "getting" it.
ISTM that, contrary to what Rails and its many imitators seem to think, a framework shouldn't be an all-or-nothing proposition. For example, in the case of my Google starter project, I need to roll my own solutions for authentication and persistence (since these must hook into internal Google infrastructure), but I really need a better templating approach than %(name)s. So I should be able to use Django's templates, or Cheetah. I really want to look into Cheetah -- the example I saw on web.py looks "right". (web.py itself OTOH gets an "F", for undocumented code with too much magic behavior. upvars(), bah.)
Maybe we need more standardization efforts like WSGI, that let you plug in different animals, or roll your own, for various pieces of useful web functionality: for example URL dispatch, templates, persistence, authentication, sessions, forms, style sheets, i18n, and client-side scripting (AJAX or not).
Not that I'm particular keen on WSGI -- it reminds me of a trip to East Berlin in the mid '80s. But its point is to provide a minimal interface between the web server and the typical web application, and to support most existing solutions on both side of the fence. The former is the real point, of course, but the latter is extremely important for standardization efforts. (Remember ISO networking? No? See, that proves my point. :-)
Here's another point. Or perhaps a set of chaotically related points. Some frameworks emphasize that the effort of getting started is low. But this is really only so to the extent that you're trying to do something very similar to something that's been done a 1000 times before, so that all the defaults in the framework and the setup utilities are all doing exactly what's needed. This (and very fast typing) is what makes the first Rails movie possible.
But what if you need to into an existing authentication framework? What if you need to hook into an existing database schema that maps to in-memory objects in a different way than the framework's ORM tool? Or if your persistence approach is not based on a relational database at all? What if you want to write an AJAX application like Google Maps or Gmail? Etc. -- for every framework component there's an alternative that you might want or need to use instead of the default.
I remember the time, before the web, when GUI apps were new. We were all learning about event handling, scrolling regions, screen refresh, window placement, text editing, and so on. Soon enough, frameworks emerged that took care of the repetitive stuff, leaving the application developer free to develop an application. But IMO it didn't work all that well. Perhaps it worked well for some apps, that really were very similar to the "ur-application" that the framework designers had in mind. But usually the framework didn't do exactly what you wanted, and then you had to tell it to do things differently. Perhaps you didn't want a horizontal scroll bar on that window. Perhaps you didn't want to automatically change the cursor to an hourglass for certain operations. Or perhaps you didn't want windows at all! Let the whole screen be a canvas. Etc., etc.
For many applications, the choice was between either using a GUI framework and writing lots of overrides to fight (some would say "extend") the framework; or using a more simplistic, lower-level GUI toolkit and writing more low-level code but being more in control. Which approach was the quicklest way to success wasn't always clear, but here's my take: personally, I'd rather use a bottom-up approach, where I can start writing simple code that forms the core of my app, without relying on framework services. Then as I find the need for templates, authentication, sessions, persistence and so on, libraries should be available to do the heavy lifting.
Perhaps frameworks could fit the role of "profiles" -- a set of libraries that is necessary and sufficient for a particular type of application. For example, there could be a profile suitable for building social networking sites, one for setting up on-line stores, one for enterprise intranet applications, one for community websites and so on. These would have different levels of sophistication, and different number of components, but there would be a considerable amount of overlap (e.g. the on-line store profile and the enterprise intranet profile may both provide session management out of the box).
I'm halfway through Joel Spolski's book, User Interface Design for Programmers. It's a great, light read (and a great companion to Steve Krug's Don't Make Me Think). Joel notices a pendumum swing in UI design paradigms: from classic power-apps that give you an empty window and a loaded menu bar, to auto-content-creation wizards focused on specific likely activities that leave the expert frustrated, and back again to a more reasonable mid point. Perhaps WSGI represents the "blank slate" approach; Rails/Django represent the wizard approach; I'm still looking for the ideal mix-and-match solution.
From this perspective, Zope and Twisted are off the scale: they support the mix-and-match approach, offering several alternative solutions for many of the important issues (templating, persistence, authentication, etc.). But they only work if you drink lavish quantities of their particular flavor of kool-aid, and that's not good enough for me. I don't want to depend on any particular flavor of interfaces, adaptation, serialization, discovery, etc. I'm looking for solutions that depend only on the Python standard library, and use accepted Python idioms and patterns. Any takers?
hope you get a chance to look at Myghty, since its close to the Cheetah style of things except its slightly more web-centric, leaves Python syntax pretty much completely intact and is designed for very fast performance, anecdotal evidence puts it at generally faster than Cheetah (probably since there is almost no extra runtime interpretation going on). Its definitely designed for the heavy lifting and stays out of the way of application specifics. I can see that web.py might be more your thing, but Myghty's extra services, particularly component-level caching, are worth checking out. Fully WSGI compliant of course, one of the first engines to be so.
It sounds like you don't necessarily want a framework, just a good way to map your data into HTML. If so, do have a look at XOXO. This is also just one file:
It has 2 functions to call - one that turns Python data structures into HTML, and one that turns the HTML back into Python ones. So, to make your webpage, put the data into a list or dictionary and output it like this: l = getMyData() html = xoxo.toXOXO(l)
Sounds like you want a Python without the standard library?
I think that's an impractical way to look at things; just like Python, it's better to view the "included batteries" as a library full of nice stuff that you can use if you want, or ignore if you feel like it.
Stock Python 2.4, sqlite3, cElementTree, and one hour from start to finish (including software installation).
The first iteration (up and running after ~40 minutes) didn't use neither the database nor the templating system; all it did was to grab pages from disk, fix them up using code from a batch script I had written earlier, and return them to the browser.
The second iteration (another ~20 minutes) used caching/gzip "middleware", a simple database model to map URLs to page files, and a 5-line django template for the HTML boilerplate.
Even without the cache, and running on an old 700 MHz box under Django's test server, the resulting app is fast enough to serve well over a million pages per day. That's pretty cool, and definitely good enough for me.
First, despite Pylons perfectly encompassing almost all the things you listed, it does have a few dependencies. It does not choose an ORM for you, but provides a place should you want to use it, to define your ORM stuff. While it comes with a dispatcher that uses Routes, you can customize the dispatch yourself should you want a different method. Pylons provides many defaults, but leaves them open for easy customization.
It uses WSGI to make it easy to plug-in the authentication system of your choice, and writing your own (completely outside the framework of course, so you can plug it into any other WSGI app of yours) is rather simple.
Despite the "scary" name, middleware is nothing more than a function wrapping a function, passing along the call. It's really simple, and makes it very easy to break down functionality that used to be in frameworks, into convenient parts you can plug-in. This seems to be exactly what you're looking for.
Paste is a set of WSGI parts that throw up a WSGI application, and a few pieces of middleware which allow you to easily build your framework to meet your web applications needs.
Because Pylons utilizes Paste, it means you can easily use a Pylons webapp inside a different framework/webapp entirely, as long as the other one support WSGI. Or you can use a WSGI compatible webapp inside Pylons. This pluggability means you aren't locked into one framework just because you have webapps using it. Pylons also makes middleware rather easy to configure, perhaps seeing the default middleware chain would alleviate your concerns: http://pylons.groovie.org/project/browser/trunk/pylons/templates/paster_template/%2Bpackage%2B/config/middleware.py_tmpl
It's just functions wrapping functions. Cascade just tries to call each app, and continues if it hits a 404, a nice easy way to have your webapp search for static files, then call your app if they aren't there. You could even add multiple static directories rather easily just by making another StaticURLParser.
Pylons merely sets up defaults to get you going, its really little more than Myghty with some Paste middleware and a Routes dispatcher. If you want to break it down more, that's easy too. Nothing is chained to a specific ORM, etc. so its pretty trivial to swap in your preference. Using the template plug-in entry points, Pylons supports Kid, Cheetah, Myghty, STAN, and more.
Ian Bicking has a WSGI URL dispatcher using Routes, and there's the URLDispatcher. Another project called RhubarbTart uses an object-publishing approach similar to CherryPy. So with these parts, you can build your framework to meet your needs, without having to reinvent the wheels.
If you want to start from the bottom-up, I'd highly recommend Paste. It comes with a set of authentication middleware, that should also give you a good set of examples on how to write your own that hooks into Google's auth system. You'd then be able to re-use it with any webapp supporting WSGI. Same goes for the persistence scheme.
If you want a bit more of a jump-start, that still gives you the options of changing any part of the stack with ease, Pylons fills that area.
When the other frameworks say they support WSGI, they mean that the framework can be called via WSGI. You can't then call a WSGI app from within it, nor do they utilize existing WSGI components. Part of the reason for this is because some of them don't understand the utility of it or how to use it, and the other reason is that they don't want to put good features in highly re-usable parts.
Why wouldn't they want to? Because it would make their particular framework less necessary since you'd be able to use it easily anywhere. If more frameworks put things in a re-usable layer like wsgi, there'd be less and less reason to use a particular framework, vs building something that fits exactly what you want.
As we've seen, some people get rather twitchy and excited fighting over frameworks, such choice and re-usability would sure make that tough.
> Sounds like you want a Python without the standard > library?
I must not have been clear (the thoughts in my head about this are still pretty fuzzy). I want libraries. It's fine if a framework provides one of each, which would be a standard library. What I don't want is standard infrastructure (meta stuff) above and beyond what's already in Python. It's possible that I'm just looking for a different approach to teaching this stuff -- instead of the typical tutorial that follows the steps in the first Rails movie, using a generator script that creates models, templates and everything else, I may be looking for tutorials to individual components that don't require you to grok all components together, or at least in a specific order.
> I think that's an impractical way to look at things; just > like Python, it's better to view the "included batteries" > as a library full of nice stuff that you can use if you > want, or ignore if you feel like it.
It's the ignoring where I feel the current crop seems to be falling short. (Of course that is even more the case for Zope.)
I think you meant "of the Python site". No need to confirm if that's what you meant (trying to keep traffic low).
> Stock Python 2.4, sqlite3, cElementTree, and one hour from > start to finish (including software installation).
I presume that sqlite3 is also a package you already know well, like the other two?
> The first iteration (up and running after ~40 minutes) > didn't use neither the database nor the templating system; > all it did was to grab pages from disk, fix them up using > code from a batch script I had written earlier, and return > them to the browser.
So what did Django do for you?
> The second iteration (another ~20 minutes) used > caching/gzip "middleware", a simple database model to map > URLs to page files, and a 5-line django template for the > HTML boilerplate.
The caching in Django actually mildly disturbs me; isn't this something on the "other side" of the web server from the app's POV?
> Even without the cache, and running on an old 700 MHz box > under Django's test server, the resulting app is fast > enough to serve well over a million pages per day. That's > pretty cool, and definitely good enough for me.
Why is there so much emphasis on speed? Several posts mentioned the speed of various rendering/templating engines as one of their most important features. Given the number of web sites being developed, most web sites never see much traffic, so I don't understand the emphasis. Is everyone just hoping that their first project becomes the next Orkut?
> > Why is there so much emphasis on speed? Several posts > mentioned the speed of various rendering/templating > engines as one of their most important features. Given > the number of web sites being developed, most web sites > never see much traffic, so I don't understand the > emphasis. Is everyone just hoping that their first > project becomes the next Orkut?
well we do actually use these things at real jobs, where you might be doing a site somewhere that has to deploy on just a few small webservers, and then one day has to survive a Slashdotting, or an Oprah-ing, a Super-bowl-ing, or some other high traffic event where you suddenly get hundreds of thousands of users trying to get at something; your boss wants you to use something that isnt going to go down even when the server is running hot. Maybe speed then is not the ultimate metric, its memory/CPU usage as well.
However, its been my observation in the Python community that people generally are not looking at the speed of things. A lot of the tools I've looked at, or gotten feedback on, indicate that they run quite slowly and use a lot of memory...Rails included.
> It's possible that I'm just looking for a different approach > to teaching this stuff -- instead of the typical tutorial that > follows the steps in the first Rails movie, using a generator > script that creates models, templates and everything else, I > may be looking for tutorials to individual components that > don't require you to grok all components together, or at least > in a specific order.
Good observation. I suspect you're right; not sure who's going to do anything about it, though.
(fwiw, I've ignored the movies and screencasts -- if I want brightly colored moving objects, I'll watch The Simpsons with my kids. "tin-ton, pappa, tin-ton!")
On the other hand, as long as the frameworks are written in Python, you don't have to do what they say. I mean, I didn't set out to write a Django app, I just wanted to see if I could make it through the tutorial. I just didn't get very far before I started thinking "hey, what would happen if I stuff the render function from my 15-minute batch hack into a django view function?".
I still haven't finished that tutorial.
(this whole experience reminds me a lot of when I first found Python; I didn't mean to write my first Python application, I just wanted to tinker a little to see if it felt right... if your tutorial had been more rigid, I might have chosen Tcl instead.)
> > fwiw, I just wrote my first Django application: > > > > http://effbot.org/pydotorg/PythonOrg.html > > > > (the above is a static copy of the django site) > > I think you meant "of the Python site". No need to > confirm if that's what you meant (trying to keep traffic > low).
What you're seeing is a static copy of a dynamic Django site which renders content on-the-fly from a MoinMoin wiki which contains a wikified snapshot of the python.org HT sources (a "real-time ht2html", sort of). Still with me? ;-)
> So what did Django do for you?
In the first iteration, all it did was to provide me with a web-based development environment: decent debugging and automatic reload/restart (save in editor, reload page in browser, save, reload, save, reload, done).
And a deployment path, of course.
In the second iteration, it gave me some really useful and well-tested stuff, both as middleware (see below) and via libraries. But I just added stuff; the tricky parts are still done in pure Python and ElementTree.
> The caching in Django actually mildly disturbs me; isn't this > something on the "other side" of the web server from the app's > POV?
Caching is "middleware" in Django; the application doesn't have to care if it doesn't want to (for global caching, all you need is to tweak the server settings file. for more control, use decorators on your URL handlers, or using the caching API directly).
> Why is there so much emphasis on speed?
Maybe because things don't have to be slow if they don't have to? Python's darn fast these days. Get used to it ;-)
> I'd rather use a bottom-up approach, where I can start > writing simple code that forms the core of my app, > without relying on framework services. Then as I find > the need for templates, authentication, sessions, > persistence and so on, libraries should be available > to do the heavy lifting.
This is exactly the design direction of CherryPy; the core specifies a URL dispatch mechanism, a configuration API, and an API for starting an HTTP server (e.g. for use with WSGI). Half of the rest is libraries (which have specifically been written to be useful outside of CherryPy) and the other half is optional filters.
CherryPy is not intended to be what people are now calling a "megaframework". CherryPy intentionally wraps HTTP and ignores persistence, templating, etcetera, leaving the individual developer (or megaframework authors) to specify those things.
> Perhaps frameworks could fit the role of "profiles" > -- a set of libraries that is necessary and sufficient > for a particular type of application. For example, > there could be a profile suitable for building social > networking sites, one for setting up on-line stores, > one for enterprise intranet applications, one for > community websites and so on. These would have > different levels of sophistication, and different > number of components, but there would be a > considerable amount of overlap (e.g. the on-line > store profile and the enterprise intranet profile > may both provide session management out of the box).
CherryPy is trying hard to be the "lowest common library" for all of these application types. Note that "common" does not mean "complete"--CherryPy wraps HTTP, and expects other libraries or frameworks to compete above it, adding value for each niche.
Currently, the URL dispatch and filter-invocation mechanisms are the only hard-wired parts of the core (and the former can be replaced wholesale with a custom Request object; Turbogears may do this soon to implement some custom dispatch styles). My hope for CherryPy 3 will be to formalize the separation between "CherryPy as an HTTP server library" and "default CherryPy"; that is, I want to see if CherryPy 3 can become "the library you use to build an HTTP server to your liking"--not through configuration, but through composition.
"URIs are the lynchpin of the Web; to get the full value of the Web infrastructure, you need to be able to identify every interesting part of your Web application with a URI. Unfortunately, common Web APIs dont encourage this, or even actively discourage it.
For example, one of the most prevalent server-side APIs for HTTP [...] the Java Servlet API does things backwards. It dispatches requests first to the HTTP method, and then has the application handle the URI."
So Rails and similar fix this problem by inverting the api but in doing so forget that HTTP already exposes a very simple verb vocab. That leads to an entirely different kind of mess but still a mess.
Whether web.py got an F or not, think it's worth understanding the motivation for it: a middle ground between to the above two API styles.
> I must not have been clear (the thoughts in my head about > this are still pretty fuzzy). I want libraries. It's > fine if a framework provides one of each, which would be a > standard library. What I don't want is standard > infrastructure (meta stuff) above and beyond what's > already in Python. It's possible that I'm just looking > for a different approach to teaching this stuff -- instead > of the typical tutorial that follows the steps in the > first Rails movie, using a generator script that creates > models, templates and everything else, I may be looking > for tutorials to individual components that don't require > you to grok all components together, or at least in a > specific order.
I think this is extremely valid point: there are many ways to explain stuff, and almost all tutorials target complete n00bs. I feel stupid doing tutorials.
For example I started to "get" Django when I read documentation of components (which is excellent, by the way) instead of tutorials, and inspected actual code. It dawned on me that Django is a set of loosely coupled components, which can be used separately in many cases. Templates? Swappable. Some people use Zope3 instead of Django's native one. Models? Swappable. Use plain Python instead or existing libraries. HTTP processing flow is open enough to implement your own logic ("middleware"). You can even substitute URL dispatching (I did that too).
But not everything is a component. Many useful Django features are implemented as "helpers", and thus completely optional. Form processing is done that way, RSS generation, generic views, and a lot of other stuff. I feel better when code is there to assist me, instead of being forced on me.
Downsides of such dismemberment? By using Django models you are getting Admin application for free. Don't underestimate it: it's not a n00b's tool --- you can give it to your clients as soon as you designed models so they can work on content, while you do presentation and business logic parts. It cuts roll-out time significantly.
OTOH all parts of the system are consistent, and feel natural, while 3rd-party code can be based on different paradigms, and may be felt foreign in details. I think you have to pay for flexibility but it may worth it.
> I must not have been clear (the thoughts in my head about > this are still pretty fuzzy). I want libraries. It's > fine if a framework provides one of each, which would be a > standard library
Most of the projects mentioned here are presented as frameworks, so you may have to dig more to find the libraries inside them. My impression is this is/was largely a response to Rails which *is* designed to be used as a cohesive framework, when most everything else out there was just libraries -- many people *do* want frameworks.
So it's two general approaches/views, which cater to different people. But, as others have mentioned, lots of these projects do offer their parts as libs. Even Rails is composed of independent libs (though the controller part/s are largely meant to be used with the whole package. The model libs can be used standalone very easily).
Oh and like Eugene above touches on, a big part of the attraction for many people on the frameworks is a consistent API feel and style between the different libs making up the whole package.
Overall it's probably an 80/20 thing, making it very easy to do the most common 80%. [Or closer to 100% (or 0) depending on the problems one needs to solve.] And I dare say your abilities to effectively tie disparate libs together is probably a bit better than the average dev out there trying to throw something together...
> hope you get a chance to look at Myghty, since its close > to the Cheetah style of things except its slightly more > web-centric, leaves Python syntax pretty much completely > intact and is designed for very fast performance, > anecdotal evidence puts it at generally faster than > Cheetah (probably since there is almost no extra runtime > interpretation going on).
Hi Mike, by 'extra runtime interpretation' I assume you're talking about Cheetah's NameMapper/searchList lookups. They can easily be turned off with a simple setting so Cheetah outputs pure Python code with no extra lookup magic.
Previous Topic
