Summary
As long as we believe that it's possible to deterministically control the development of software, deterministic solutions will be the only ones that we can entertain. Only by letting go of the idea of determinism can we open up to other, less perfect ways of influencing the process.
Advertisement
A previous essay elicited this response:
Commenter:The problem is not that determinism is an illusion...it is not. It is just us that we are not able to calculate the outcome. For example, take a French billiard table with two balls only: we can easily predict their trajectories, when they will collide etc. Add another ball, and the problem becomes very difficult, if not impossible, to predict. But the outcome is deterministic nevertheless because it is defined by physical laws.
This comment describes is the approximation that falls within the realm of Newtonian mechanics. These are the "physical laws" that people believed governed everything in the universe, until they discovered through experimentation and observation of things that are very big, very small, very fast, very cold, or very hot, that the window where Newtonian mechanics worked was the fairly narrow one of our direct experience. It only works that way because some of the factors, like the gravitational effects between small bodies like billiard balls, or the Brownian motion at typical billiard-table temperatures, are negligible. And probability can even come into play -- the nap of the felt could influence an outcome that's on the edge.
Also, I think the commenter is trying to describe the 3-body problem with the pool table example. However, the 3-body problem involves gravity, and that's why it gets so complicated -- so the bodies need to be big enough for gravitation to have an effect in order for it to be a true 3-body problem.
I think this commenter is making my point for me. Physics always involves approximations, and discovering which factors, in that particular situation, are important. The problem occurs when we start ignoring those factors altogether, and pretending that they don't exist and that our model applies everywhere. A useful model is only an approximation that works under the current circumstances. I think a previous commenter already mentioned Goedel; he and others after him have proven, in essence, that all models are incomplete.
What I've been trying to say in these recent essays is that we have been attempting to apply the same logic in all situations, just as this commenter has asserted that the pool table example is "defined by physical laws" (where probability doesn't come into play). We produce success with a certain team of people, and we draw conclusions such as "it was because I used this methodology," or "it was because the team was less than 8 people and we worked in this kind of environment." And those things may actually have a predominant effect ... in some cases. But those things may also be overwhelmed in other cases; a good example is if you have a particularly strong personality, positive or negative, on the team.
The problem is that science in general and physics in particular is all about looking at the situation and understanding what the dominant factors are, and discarding the factors that, in the current situation, do not influence the outcome enough to care about it for whatever we are doing at the moment (the factors are still there -- they may simply not be measurable, or we may not care about their impact). But in physics, these are repeatable phenomenon, so we can run the experiment again and again and duplicate the results.
With software development, you can't run the same experiment twice, because all the factors are always different. Even if you use the same team, that team will change significantly from one experiment to the next. If we were to use the same approximations that physicists do, one argument would be that all the factors can be discarded except for the people factors, which tend to overwhelm everything else (and this argument appears to be true in a large number of cases). Now we're left with a group of individuals who are all changing. And it's impossible to model their behavior.
Maybe there are cases where the people factors can ignored because something more powerful is involved. I think that's the scenario that many managers are pursuing, because those more powerful factors might be controllable and thus the outcome is more predictable. But I don't think this has ever happened on a software project. I think people have believed that they are in control and have tried to steer things based on what they thought controlled the outcome.
In all the cases I've ever seen, however, the people factors overwhelm everything else, and determinism is, in fact, an illusion. That doesn't mean that you have to give up trying to influence the outcome, but just that looking at the problem in a deterministic fashion is a dead end. This is why complexity theory is so interesting, because it looks at problems that are far too complex to model using some kind of equation, and yet it says "let's see what we can do with this anyway."
But deterministic thinking won't allow you to go there, and that's the problem. As long as you're seeking a solution that promises a predictable outcome for a set of inputs, you'll fail to come up with a useful approach, and that's what we've been seeing for many years in the software industry.
But if we can let go of determinism, this allows us to say instead, "The problem is far too complex to solve, but might there be ways we could improve our chances of success?" I think we then open ourselves up to new and potentially much better ways of doing things. That's the step I think the agilists have taken. I think all the agile methods have in common that they shorten the feedback loop and create lots of little experiments with evaluations and course corrections in between. I suspect that a lot of the resistance to this is because it doesn't promise a deterministic solution like methodologies of the past have promised, and if you are very attached to determinism, you can't be satisfied with that.
> But if we can let go of determinism, this allows us to > say instead, "The problem is far too complex to solve, but > might there be ways we could improve our chances of > success?" I think we then open ourselves up to new and > potentially much better ways of doing things. That's the > step I think the agilists have taken. I think all the > agile methods have in common that they shorten the > feedback loop and create lots of little experiments with > evaluations and course corrections in between. I suspect > that a lot of the resistance to this is because it doesn't > promise a deterministic solution like methodologies of the > past have promised, and if you are very attached to > determinism, you can't be satisfied with that.
This section parlays right into something that's been brewing in my brain. I think software development is a lot more like a poker game than a chess game.
Chess is a fairly deterministic (yet exteremly complex) game. All the information is there. In poker there's a lot of information missing. A player can play a hand perfectly and still lose because it's impossible to know how the cards will come out (in a fair game.)
This doesn't mean that there is no strategy, however. In fact poker is a much more interesting game in game theory than chess because of these characteristics. It's also a better simulacrum to real world problems such as software development.
And like you mentioned, it's like software development because every hand you play in a poker game is likely to be unique in your entire life, no matter how often you play. And even if you do run into a very similar scenario, the same action could produce completely different results.
The other thing that's interesting and similar to software development is that very often it's correct to make choices when the odds (based on quantifiable information) dictate that you should do something else. In other words, someone who plays completely based on the statistical odds will be beaten by others who use the unquantifiable information available to them.
P.S. Bruce, have you read "Chaos: Making a New Science" by James Gleick?
The following takes 'deterministic' in what might seem a left turn, but bear with me.
Well, it seems there's a reason that the imperative languages (and the imperative development model: "you vill do it dis vay und you vill liiike it!") still hold sway. In spite of the existence of non-imperative languages, and matching development models. It's a matter of control. Few care whether the thing actually works; just whether it met the delivery date. It's sick, I know. But true.
I have a passing knowledge of Prolog (and some of its variants). One purveyor, Amzi! (not a typo), claims that multi kilo line java/C/etc. suites get replaced by 100 line Prolog programs. And having worked with the language, I can see how that might be. But using such a language requires a suspension of control: the code is working out an answer from some data that you can't see. Side effects, and self modifying code, are designed in. Not my cup of tea.
(SQL/relational model does the same thing, sort of. As Date says, What not How. The difference is that SQL engines don't hide the data; and the syntax is far more transparent than Prolog. And there are folks still building frameworks which generate code from a SQL catalog; the data really can drive the code.)
Think of it this way: would you rather have a human physician diagnose your problem, or punch in some data and let a program choose one in a probabilistic manner? The physician is doing the same thing, but s/he seems to be making the decision based on a Process. The notion of Process is at issue; the imperative approach is the implementation of Process. The declarative approach is about data (or State); there is no Process. This makes some folk Vewwwwwwwy Nervous. (Aside: Eiffel attempts to remove process from OO with design by contract. I like this; the way it should be.) The Sequence Diagram didn't exist at first; it came into being because folk just couldn't let go of Procedural Thinking.
Prolog engines are actually deterministic, not really AI, but they appear to be stochastic to both the user and coder. You get the "best" answer the data has to offer whose tracabilty may be tenuous; you just don't get to do anything but specify the question.
Near as I can tell, the Agilists aren't really changing the playing field; they aren't headed toward declarative (or functional) programming. The languages they support, while they tend to come from the Dynamic Crowd, are still imperative. The process is serially probabilistic (what's the best small thing I can do next? If I get it wrong, there's not much to tear down), rather than (quoting Spolsky) Big Design Up Front. But really not headed in a different direction.
The declarative folks, on the other hand (Prolog et al, SQL and not much else), do change the playing field; to the extent that they choose to make the data drive the answer, rather than the code. This is really a profound difference; and very strongly opposed by both the COBOL and (enterprise)java camps in my experience. Watching them turn XML into a programming language (XSLT) is amusing. So is watching them turn java into COBOL (but that's not an observation I can prove is original).
Now, to tie this back to the Agilists' way of thinking; a lot of baby steps is better than One Big Step. What if we took Date's assertion as true: What not How. And we used a framework which generates whatever source language we want from the database; really just a UI? How would development then proceed?
We now can morph our application simply by changing the data, if our structure is accurate. We can morph it by changing declared constraints. We can morph it by changing the structure if need be; this is not so tough. Long ago, in a country far, far away, Ted Codd devised the relational model. What he asserted from the beginning was that a row was a Business Rule; a True Fact about the real world. And that is still true. Viewed such a way, data is king, and Process not relevant.
Recast: MVC has been au courant for a long time. The SQL database, done well, is MC (it's really an Object Store, in that we have data and its control together; that's another rant). Really. That leaves the View. But why should the language of the UI determine anything? It ought not. The view is but a fungible image of the data, and thus irrelevant to development of the application; the lipstick is not the pig. Generate it to any way that pleases you. But a pretty UI doesn't make the data any more true (or false).
So the big question is then: are the dynamics of software development so unstable that determinism is impractical or is it that software development negates determinism?
In the first case a stochastic method would work just fine. In the second case we need a completely new idea of what software development is and how it should be approached.
An imaginary experiment: Assume you have just finished writing the requirements of a new software system. Based on your estimation, there are 10 man years worth of effort in implementation. You need the system in 6 months, therefore you put together a team of 20 to do the implementation. You know your 20 people are result oriented and that they can self organize in order to get things done.
You now, have to choices: a) You say nothing of a deadline. b) You tell them exactly when the delivery date is.
After 6 months of no interference, you walk in the room and ask for the software.
Is the behavior of your team (and their results) going to change if you choose a) vs b)?
IMO, absolutely. a) your 20 will self-organize. They'll put a process in place which tends to resolve requirements one by one, in the order dictated by dependencies. Their decisions are mostly technical ones. After 6 months, you'll get their best shot of implementing your requirements.
b) the team has to deal with 2 things now: implementing the requirements as in a) and fitting all that in a given timeframe. Every decision they have to make has 2 components: technical + timeframe. Finding the right answers takes longer. A schedule is put in place and its assumptions leaks into the design and further complicate decisions. Chances are, after 6 months you'll have less implementation. Regardless how far a) or b) get, the end result is completely different.
The fact the a) and b) yield different behaviors hints that trying to determine the outcome of software development changes the outcome itself.
This is similar to the two slit experiment in The Feynmann Lectures on Physics. When you don't look at things they behave one way, differently then we you look at them. This is a manifestation of uncertainty and fundamental quantum behavior.
My feeling is that uncertainty is intrinsic to software development in some way. In software development estimation we need to deal with probabilities not because of some sort of instability and they make a better tool, but because randomness is at the very heart of software development.
Your blog has been very active these days. I'm going to need to find some free time to catch up on them and add my 2 cents.
Anyway, I believe at this point in time that software development is an art form but in time it will become a science. Nearly all, if not all areas of science began like art. Over time as relationships and the unknowns are discovered the areas becomes science.
Ten to 20 years ago in the field of heat transfer, any problem outside of the most basic ones were solved with a bunch of rules of thumbs, empirical data, and methods that felt like black magic. So at that point it was more like art but over time our understanding of heat transfer has improved greatly and the approaches we use today to solve these problems are more scientific.
Now the heat transfer example is one where the transition from art to science has occurred recently. In other areas of science you will need to go back in time by reviewing the history of that area and you will find similar transitions that occurred. I'm sure that in each area of science, before they became science, the people who worked in those areas also felt that the solutions to problems were not deterministic or had some random characteristics.
I do not buy the idea that solutions to software problems are random but will buy that as long as there are many unknown factors in the development process that solutions will feel random in nature. We need to do our best to uncover these unknown factors and obtain a good understanding how these factors affect development so that one day we will have deterministic approaches to software development.
I say approaches instead of approach as when we have a deeper understanding of these factors we will be able to determine which factors can be ignored or simplified for a given project/problem. How long will it be before we have this deeper understanding? Who knows, but it's likely to be far more than 10 years. I wouldn't be surprised if it takes more than 50.
I don't buy that declarative languages provide the profound solutions to all types of problems. To create a declarative language for a particular type of problem there needs to be a good understanding on how you go about solving that type of problem. It's this understanding that allows one to create an effective declarative language.
Not all types of problems have known solutions. Creating a declarative language for such a problem will just turn out to be a waste of time and effort and likely result in a project that fails.
Luckily for most developers, the problems they need to solve tends to be very similar in nature (type of problem) and the solutions to them are well known. Declarative languages are a big win for these types of problems. When I say that solutions to these problems are well known I'm not talking about 100% of the solution, as if that was the case most software projects would be successful.
“data is king” can be a very dangerous position to have on some types of problems. For example, if I ask you to build a system that is going to make us all rich by having it trade stocks for us by monitoring the markets in real time and determining trades that have high probabilities of being profitable. In such a problem the structure of the data is nearly meaningless or has no appreciable effect on the solution. In this type of a problem it becomes far more important on how we use the data than what the data is.
On the other hand if you were building a cart for a web application the structure of the data can have a relatively large impact on the solution. Pick a bad structure and the amount of code to deal with the data may in fact become a significant chunk of the solution. Although in reality, differences in structures for this type of problem are not likely to have that much of an impact. It's just that the impact is relatively far greater for this type of problem over the problem type associated with stock trading system.
It's very likely, that if you were to reflect on the projects you have worked on, you would likely find that you have been working on problems of the same type. I only say this as I can see how someone who has not had to work on a project where the “how” you use data becomes significantly important could feel that it's the data that is king.
Try not to take this reply personally as I think very few people work on the types of projects where the “how” you use data becomes a dominate factor. Although, these type of problems are becoming more prevalent as the complexity of systems increase and the ever increase in pressure to increase the efficiencies or cost of these systems (systems here does not necessarily mean software systems).
> <p>With software development, you can't run the same > experiment twice, because all the factors are always > different. > > <p>In all the cases I've ever seen, however, the people > factors overwhelm everything else, and determinism is, in > fact, an illusion. That doesn't mean that you have to give > up trying to influence the outcome, but just that looking > at the problem in a deterministic fashion is a dead end. > This is why complexity theory is so interesting, because > it looks at problems that are far too complex to model > using some kind of equation, and yet it says "let's see > what we can do with this anyway." > > <p>But deterministic thinking won't allow you to go there, > and that's the problem. As long as you're seeking a > solution that promises a predictable outcome for a set of > inputs, you'll fail to come up with a useful approach, and > that's what we've been seeing for many years in the > software industry. > > <p>But if we can let go of determinism, this allows us to > say instead, "The problem is far too complex to solve, but > might there be ways we could improve our chances of > success?" I think we then open ourselves up to new and > potentially much better ways of doing things. That's the > step I think the agilists have taken. I think all the > agile methods have in common that they shorten the > feedback loop and create lots of little experiments with > evaluations and course corrections in between. I suspect > that a lot of the resistance to this is because it doesn't > promise a deterministic solution like methodologies of the > past have promised, and if you are very attached to > determinism, you can't be satisfied with that.
Bruce,
you are mixing determinism and complexity, which are two totally different concepts.
Software development is deterministic, in the same manner that the 3-body problem is: the outcome is predictable through a set of laws, no matter how complex the software is or how many bodies we throw to the problem.
But that does not mean we can solve the equations! maybe we do not have the methodology or the computing power to examine such complex problems.
So, our only solution, is to do our best shot and try to improve our chances to success.
Despite all this, I think that it is doable to statically check even large programs about violation of contracts. I have no mathematical proof about that, but experience has tought me that whatever can I do with my brain, I can do with a program...and since I can catch may different bugs with only using my brain, I think a compiler can be made to automate the task...because my brain gets easily distracted, whereas a compiler does not.
By the way, what is our thought process when we discover a bug? we abstract things away. We do not care how or when a particular function is executed, but we know that if we call it with null the program will crash. It is that kind of abstraction that our tools should have.
> To create a > declarative language for a particular type of problem > there needs to be a good understanding on how you go about > solving that type of problem. It's this understanding > that allows one to create an effective declarative > language.
Prolog, et al do not propose to be declarative languages for a particular type of problem. It is not a DSL, or DSL factory. OTOH, it is not, in practice, a general purpose language given that it's purpose is solving logic problems. Its proponents assert that all user applications devolve into asking questions of a "database", and thus that part of an application is best solved with Prolog. The semantics are general to the problem of logic; the data will be specific to a problem domain. Much as the relational model is separate thing from the data stored in a SQL database.
> “data is king” can be a very dangerous position to have on > some types of problems. For example, if I ask you to > build a system that is going to make us all rich by having > it trade stocks for us by monitoring the markets in real > time and determining trades that have high probabilities > of being profitable. In such a problem the structure of > the data is nearly meaningless or has no appreciable > effect on the solution.
On the contrary: real time regression analysis of trade prices is what you need. Structure of the data (what's happening to price) is what's important. No amount of programming will substitute for the information available. There are all manner of statistical analyses which can be applied; Wall Street C/Lisp/Smalltalk wonks and econometricians do this a lot. (This is pretty much what 'programmed trading' is/was. The folks who can afford to play get very rich indeed.)
> In this type of a problem it > becomes far more important on how we use the data than > what the data is.
"What not How" might be confusing. Date uses the term to differentiate imperative from declarative methods. He likes declarative as expressed in the relational model. In current versions of DB2, one can ask directly for a simple regression of column data, using SQL. One might, say, ask for the correlation of price with the trades of certain size, or trader, etc. or simply attempt to estimate the time series off the end of the data (you're not supposed to do that, but most folk don't know that). This isn't SAS, but you get picture. It's also how the SEC spots insider trading, although not, apparently, in real time.
> I > only say this as I can see how someone who has not had to > work on a project where the “how” you use data becomes > significantly important could feel that it's the data that > is king.
Data is always king. Code bugs reveal themselves when they screw up the data. We don't see them until then; modulo really good code reviews. Code screws up data; data doesn't screw up code. Self-modifying code excepted, but that's a corner case, or should be. It is true that not all applications are amenable to having runtime access to a SQL database; writing linux device drivers or arcade games for instance. On the other hand, operating systems have been written which do: the still born winFS, the long gone PICK/OS, and the still alive and kicking System/38 -> OS/400 -> i/Series machines.
> you are mixing determinism and complexity, which are two > totally different concepts.
I agree to a point.
> Software development is deterministic, in the same manner > that the 3-body problem is: the outcome is predictable > through a set of laws, no matter how complex the software > is or how many bodies we throw to the problem. > > But that does not mean we can solve the equations! maybe > we do not have the methodology or the computing power to > examine such complex problems.
The problem with the n-body problem is not that the computation is difficult it's that it doesn't settle into a clean cycle. Given the weights, starting points and velocities of the objects, determining the forces at that point is fairly straightforward but determining how these properties far in the future requires calculating all the future states between here and there. No matter how fast the computer can calculate the states, we can ask for a state a future point that will infeasible to calculate.
However, I think you are ignoring a key characteristic of software that makes ut unlike the n-body problem. In the n-body problem the intial state provides all the information needed to claculate any future state. This is not the case in software. The information at the beginning of a software project is incomplete partly because there will events during the progress of the project that cannot be predicted and people are very unpredictable.
> you are mixing determinism and complexity, which are two > totally different concepts.
I agree with that.
Software Development is deterministic in my experiance, on my current project we are completing 26-36 "end user stories" a month. A end user story might be a small bug (one-line fix) or a new feature.
>With software development, you can't run the same >experiment twice, because all the factors are always >different.
So not true, the field of Experimental Psychologi has over and over again proven that human behavior can be tested and measured, and be deterministic/predictable.
>As long as you're seeking a solution that promises a >predictable outcome for a set of inputs, you'll fail to >come up with a useful approach, and that's what we've been >seeing for many years in the software industry.
Do you have an example of what "the sets of inputs" is?
If you want to look at more facts and studies of Software Development look at: Facts and Fallacies of Software Engineering by Robert L. Glass - This book answers most of questions about why a project fail, using facts supported by studies.
> So not true, the field of Experimental Psychologi has over > and over again proven that human behavior can be tested > and measured, and be deterministic/predictable.
Are you sure you aren't mixing up terms now?
Deterministic: Referring to events that have no random or probabilistic aspects but proceed in a fixed predictable fashion.
Are you suggesting that you can predict the behavior of an individual or group with 100% accuracy?
I think you can't have lived very long in our world if you believe that.
> >As long as you're seeking a solution that promises a > >predictable outcome for a set of inputs, you'll fail to > >come up with a useful approach, and that's what we've > been >seeing for many years in the software industry. > > Do you have an example of what "the sets of inputs" is? > > The most study suggest that Software Development is > Deterministic : > http://www.developerdotstar.com/mag/articles/software_succe > ss_failure.html > > My favorite part of that article: "Most projects that had > no schedule were successful"
Someone else posted a study like this. I have a big problem with both:
"The findings came from NICTA’s study of 400 projects in the U.S., Australia, and Chile, using questionnaires and interviews to discuss success and failure factors with practitioners."
This is not an experiment testing software development. It's a psycological experiment involving project managers. So you need to re-phrase your favorite part: 'project managers that reported having no schedule reported that they were successful' because that's all this study shows.
I'm not suggesting that these studies are not meaningful but you can't frame this as a scientific study of software deveopment. Scientific experiments are not executed via hearsay. Scientists do not test theories with surveys. I wager that you could do a study like this that 'shows' prayer causes software projects to succeed. There have been many other psycological studies done that show that many people think they are successful when they are in fact unsuccessful.
> Software development is deterministic, in the same manner > that the 3-body problem is: the outcome is predictable > through a set of laws, no matter how complex the software > is or how many bodies we throw to the problem.
This is simply a statement of belief, no different than Einstein's belief in the hidden variable, but far less plausible. I've certainly never heard of nor seen a software project that supports the idea.
On top of that, you're either suggesting that the machines and the processes can completely eliminate the human factors (the holy grail for managers), or you are stating that human beings can be modelled accurately enough to "predict the outcome through a set of laws."
I remember the first time I saw a physics professor take an equation that came from "laws" and start lopping off higher-order factors. I was shocked. The equation was "the truth" and he was throwing pieces of it away. Engineering students are even more prone to thinking that their equations represent the truth, but in fact, they are all just models. And I have personally found that, although disturbing at first, it is far more powerful to understand that models are not reality (the map is not the territory), but are simply convenient approximations, which are good enough to produce helpful results. This is why Godel's Incompleteness Theorem is so important -- it shows that all models are flawed in some way, even if they produce useful results. Godel keeps us from thinking that our models are The Truth.
A belief in determinism is itself a model of the world, and can be useful and comforting, initially. But eventually it becomes restrictive and constraining. So much so that, when I discover that you believe, for example, that in software development, "the outcome is predictable through a set of laws," there is a whole slew of topics that we simply can't discuss, because we won't be able to consider ideas that suggest that software development is not deterministic. And this belief in determinism will lead us to conclusions like "more type checking is always better" or "design by contract spans the problem space." Despite evidence and experience to the contrary.
Not that I expect to talk you out of your belief in determinism. It's a belief, and not one that goes away in a flash of insight, especially if you come from a scientific background where there is always a desire to establish cause and effect. And for me, it's taken much of my life to get to the point where I begin to understand what a model really is.
>Deterministic: Referring to events that have no random or probabilistic aspects but proceed in a fixed predictable fashion.
Maybe I need to brush up my english, English is my second language. According to http://en.wikipedia.org/wiki/Deterministic_system_(philosophy) : Deterministic system is "Some of these systems are complex, and events may be difficult to predict in practice, but if the starting conditions were known in enough detail, the outcomes of events in such systems could be predicted."
So maybe you can point me to a page that better explain Deterministic than that?
"Social determinism is the hypothesis that social interactions and constructs alone determine individual behavior (as opposed to biological or objective factors)." http://en.wikipedia.org/wiki/Social_determinism
>Are you suggesting that you can predict the behavior of an individual or group with 100% accuracy?
There exists studies that show that a predict the behavior in a group can be done, if the enviroment is controlled, I will try and remember what the name of the experiments are.
"Behaviorism, an approach to psychology based on the proposition that behavior can be researched scientifically without recourse to inner mental states, is usually considered to be deterministic and opposed to free will."
"One of the assumptions of many behaviorists is that free will is illusory, and that all behaviour is determined by a combination of forces comprising genetic factors and the environment, either through association or reinforcement."
And psychology has shown that a groups will is more powerful the the individuals will a documentay about the London subway-bombers suggestest just that, and the armies around the world uses the group will to form predictable elit units.
Flat View: This topic has 45 replies
on 4 pages
[
1234
|
»
]
Previous Topic
