Summary
Martin Fowler reports on life without database transactions, and on why handling transactions in application code, instead of relying on database transactions, may sometimes be the right solution.
Advertisement
In Transactionless, Martin Fowler comments on a recent presentation by eBay architect Dan Pritchett about how, and why, eBay's architecture does not rely heavily on database transactions. Instead, data integrity is handled by application-level code. Fowler notes that,
The rationale for not using transactions was that they harm performance at the sort of scale that eBay deals with. This effect is exacerbated by the fact that eBay heavily partitions its data into many, many physical databases. As a result using transactions would mean using distributed transactions, which is a common thing to be wary of.
This heavy partitioning, and the database's central role in performance issues, means that eBay doesn't use many other database facilities. Referential integrity and sorting are done in application code. There's hardly any triggers or stored procedures.
Fowler is quick to note that transactions are a very useful data management tool, and that:
Nobody is arguing that we should tear transactions out of our toolkit. I don't know enough details about eBay to judge whether avoiding transactions is the right approach even for them. But eBay's example suggests that living without transactions is far less impossible than many people think...
Apart from the fact that it's always worth thinking about alternatives, it's also the case that transactionlessness is more common than many people think. It's common to have multi-step business processes with multiple resources that either would need long-running distributed transactions, or resources that have no support for transactions.
According to Fowler's blog post, handling data integrity in the application layer requires great care, and some additional code:
You have to pay attention to the order of your commits, getting the more important ones in first. At each commit you have to check that it succeeded and decide what to do if it fails.
As developers look to scale up enterprise applications, the database tier often ends up as the last bottleneck. It seems that two approaches have emerged to scaling the database tier: In-memory, distributed caches, such as Tangosol's Coherence and JBoss Cache, and partitioning a large database into many smaller databases, de-clustering data horizontally across those databases.
In the first solution, the cache becomes the application's primary view into persistent data. As some cache solutions provide their own APIs, they bypassing the traditional enterprise data management APIs, such as JDBC and J2EE transactions.
Declustered databases, on the other hand, require that an application manages multiple database connections, possibly relying on distributed transactions to ensure that operations across those databases succeed. Distributed transactions—and a distributed transaction coordinator, in particular—can then become yet another bottleneck and single point of failure, possibly limiting scalability. As Fowler's article illustrates, coordinating work across multiple databases from an application may increase scalability in the distributed database case.
In your applications, how do you avoid the database becoming a single point of failure and a scalability bottleneck? What do you think of the transactionless application style outlined in Fowler's blog?
I've no doubt that someone as experienced as Martin fowler could do this, but most developers I've come across are simply not skilled or disciplined enough to do this.
As the saying goes, a chain is only as strong as its weakest link.
I'm not sure it's correct to call this completely without transactions. When you update, delete or insert a row, the whole operation succeeds or fails. While they may not be using higher-level transactions, this in itself makes things much easier.
I recently worked on a LDAP backed module and the API had no concept of a commit or a rollback. You have to execute changed to each item of a record individually. If you had two changes to a record that were required to both succeed or both fail, there was no way to guarantee that would work. You would have to try to reset the first if the second failed and there was no guarantee that would work. Ebay is still using transactions, they just only contain single row changes.
Ironically, I know a lot of RPG and COBOL programmers that don't really know what transactions are and use autocommit for everything.
> I'm not sure it's correct to call this completely without > transactions. When you update, delete or insert a row, > the whole operation succeeds or fails.
Fowler writes that "eBay mostly hardly ever uses database transactions" and later "You have to pay attention to the order of your commits, getting the more important ones in first". So, he uses commits but no transactions???
> Fowler writes that "eBay mostly hardly ever uses database > transactions" and later "You have to pay attention to the > order of your commits, getting the more important ones in > first". So, he uses commits but no transactions???
I think what Fowler was talking about were database transactions. Instead of relying on database transactions, he notes the idea of managing transactions from application code.
A bigger question lurking behind Fowler's comments, though, is how to scale up the database tier. As soon as multiple databases are updated from a single application, or even within a single logical unit of work, distributed transactions would have to be used to ensure the transactional integrity of the operation. I'm curious,
(a) How many people are actually updating multiple databases from within a single application or unit of work? and
(b) How many use distributed transactions when doing so?
If not via distributed transactions and multiple databases, then what techniques do you use to scale up the database tier?
> (a) How many people are actually updating multiple > databases from within a single application or unit of > work? and
I think it depends on what you mean by 'multiple databases'. Do you mean a database cluster (a distributed database) or actual distinct databases.
> If not via distributed transactions and multiple > databases, then what techniques do you use to scale up the > database tier?
I think that most architectures don't. Isn't this one of the main reasons for the development of the data grid?
One of the things I am curious about with eBay is how they avoid race conditions. Say I try 'buy it now' and you try to 'buy it now' at the same moment. My guess is that they use optimistic locking style approaches.
> One of the things I am curious about with eBay is how they > avoid race conditions. Say I try 'buy it now' and you try > to 'buy it now' at the same moment. My guess is that they > use optimistic locking style approaches.
Simple - the user with the highest rating wins, with highest balance breaking ties. :-P
Normally you use things like database contraints to enforce your invariants. For example, you define field as a foreign key and not null so that you know that it always points at a record and that record will always exist.
Split your records across several indepedent databases and making that guerantee becomes a lot harder, but for a site like eBay it becomes necessary.
The end result is that existence of that record changes from an invariant to an expected condition, and a dangling reference becomes an exception - most likely a recoverable exception.
At eBay scale you are probably trading one type of failure against another. If you put everything in one database, it can enforce constraints, but it creates a single point of failure - of catastrophic failure. Distribute the data and you face the potential for more little failures, but you reduce the probability of catastrophic failure.
Or it could be a cost trade. The cost/complexity of fault tolerance may be less than the cost/complexity of keeping that giant database alive.
A live without transactions is possible: We had a large business application that worked always in DIRTY_READ. For modifications it used transactions, but the datastructures could do well without that as incomplete data would be simply ignored later. The application allowed structuring the data to achieve that of course.
For a toy project I had to use an old MySQL database which didn't support transactions. This app didn't allow the techniques use in the application above. The solution was to have a redo/undo log that controlled the validility of the data. This worked well and had the benefit that all commands got logged and rolling back replaying sequences of interaction were very easy and natural. A hell of a debugging facility, droping to a stack-frame on your database: Priceless. If I ever had to design the foundations of system again I would go for that. But it is today nearly impossible to convince people that it could be done, perhaps when Martin says it, too it will change
Life without distributed transactions is not only possible, it may be inevitable in practice. This is particularly true in a world in which the scalability requirements of applications grow inexorably.
Pat Helland has a very interesting paper in which he likens traditional protocols for distributed transactions to the Maginot Line. The paper talks about how to cope with not having transactional guarantees, and suggests some patterns for designing such systems.
I hope the details of Dan's presentation do make it out into the wider world, because it's sounds like there might be 'business transaction' has to exist at a level above any one db instance, surely? In which case each step in a business workflow succeeds or fails as it interacts with each application/db instance, and you'd have to manage this at that level. And as Frank rightly says, there are then various ways to deal with this (Coherence, JavaSpaces, Partitioning, or, as I've seen too many times for comfort, trying to build a massive centralised vertically scalable metastore).
I guess what I am saying is that when you hit scale issues, some of the traditionally lower-level concepts, like ACIDity of transactions, take on a wider form, and that's natural and understandable. Exploring how different companies have solved their particular flavour of domain-specific scaling problems is thought provoking and eminently useful. But in solving them the core concept hasn't gone away (so the word 'transactionless' is a bit of a red herring), just maybe changed into something else with the same objectives.
Apologies – that last post was corrupted (lesson: it’s never a good idea to copy/paste your posting through Word to spell check it first). Here it is with the missing text replaced…
I hope the details of Dan's presentation do make it out into the wider world, because it sounds like there might be some intriguing ideas in there. Right now, it's password protected and accessible only to those who attended the conference.
Maybe it's too early in the week and my brain hasn't engaged properly yet, but if your data stores are partitioned by design, or reside in multiple legacy applications because that’s just the way your company grew, then a 'business transaction' *has* to exist at a level above any one db instance, surely? In which case each step in a business workflow succeeds or fails as it interacts with each application/db instance, and you'd have to manage this at that level. And as Frank rightly says, there are then various ways to deal with this (Coherence, JavaSpaces, Partitioning, or, as I've seen too many times for comfort, trying to build a massive centralised vertically scalable metastore).
I guess what I am saying is that when you hit scale issues, some of the traditionally lower-level concepts, like ACIDity of transactions, take on a wider form, and that's natural and understandable. Exploring how different companies have solved their particular flavour of domain-specific scaling problems is thought-provoking, and eminently useful. But in solving them the core concept hasn't gone away (so the word 'transactionless' is a bit of a red herring), just maybe changed into something else with the same objectives.
More absolutely ridiculous nonsense from this clown Fowler. He should stick to churning out more tedious articles on refactoring since he is obviously a dangerous ignoramus when it comes to database management. This emperor truly has no clothes.
Previous Topic
