Summary
Not by angry customers suing for damages after security breaches, or by governments breaking up monopolies, but by open source developers and security professionals accusing them of being obsessed by security.
Advertisement
Microsoft is a company we love to hate. In particular, the security of Microsoft products has been the target of fierce criticism. However, in the last few years, Microsoft has made a concerted effort to improve the security of their products. The Windows Security Push was launched in 2002 in the run up to the release of Windows Server 2003. At that time, the seeds of the Security Development Lifecycle (SDL) were sown. This process has since been refined by many more security pushes.
I had the pleasure to moderate a panel discussion "Should companies be emulating Microsoft’s Security Development Lifecycle?" on Tuesday at the OWASP Europe conference in Leuven.
The panelists were Pravir Chandra, chief security architect for Secure Software, Alex Lucas, a member of the Secure Windows Initiative team at Microsoft, Andrew van der Stock, OWASP Guide project lead and Dinis Cruz, OWASP .NET project lead.
Relatively little of the SDL is in the public domain at present. This is about to change with the publication of Michael Howard and Steve Lipner's book. Nonetheless, the SDL has apparently already gained some traction outside Microsoft, as Andrew van der Stock reported a successful implementation at the NAB where he is employed as a security engineer. Andrew also blew the trumpet for MSDN Security patterns and practices.
OWASP's motivation for organizing the panel discussion was the announcement of the release of an OWASP process guide: CLASP (Comprehensive Lightweight Application Security Process) was donated by Secure Software to OWASP for distribution and further elaboration. Pravir will lead the OWASP CLASP project.
CLASP addresses the same problem space as Microsoft's SDL.
Would it not have been simpler to just adopt the SDL?
According to Pravir, the distinguishing feature of CLASP is the opportunity to tailor the process to the needs of the organization. SDL is seen as too heavyweight. Alex Lucas points out the irony of the SDL description fitting into a book of fewer than 300 pages, while the supposedly more lightweight CLASP requires around 600. However, Pravir stresses that the SDL is too rigorous for small organizations who may not be able to afford to work to the same exacting security standards that Microsoft is currently setting. This statement seemed to be endorsed by a significant number of the audience.
So what, if anything, of the SDL is applicable to other companies developing web applications and should be in CLASP as well?
There was overwhelming consensus that threat modeling is correctly identified in the SDL as the single most important activity for improving security. However, some concern was raised whether Microsoft's techniques, as described in Writing Secure Code by Michael Howard and David LeBlanc and Threat Modeling by Frank Swiderski and Window Snyder, pays sufficient attention to rating the risks. Not all threats need to be addressed as some level of risk is acceptable. How much risk can be absorbed is specific to the organization.
Alex emphasizes the importance of security awareness, and beyond awareness, openness and willingness to discuss security problems.
The inevitable discussion about the relative merits of open and closed source for security follows. Dinis makes the classic case that many eyes make all bugs shallow. While this may sound attractive, for the overwhelming majority of open source projects, security reviews are not taking place. Moreover, many open source applications are being written by developers who are poorly security educated and unwilling to compromise on features for the sake of better security. Both Andrew and Alex recount reporting security bugs in open source software, only to be ignored.
Has Open Source lost the security edge and is it now being superseeded by the products and practices Microsoft is introducing?
I missed the panel at the OWASP conference, even though I live in Leuven and work at the university, so these points may or may not have been raised. I am also no expert on SDL or CLASP, but I don't think you can use only these to draw a conclusion on the security of Microsoft versus open source.
In reply to your comment about 'more eyes find more bugs and that many open source projects are written by poorly educated developers':
Is it fair to compare one company to every single open source programs out there? While it's true that some open source developers write poor software, there are also many people writing poor software for Windows (you can best compare most of the bad open source stuff to the bad Windows shareware). So comparing all of open source to just Microsoft is an unfair comparison (you can only compare all open source project to all of the closed source projects). In my opinion only the large projects should be compared to the large projects that Microsoft has (i.e. Linux kernel + KDE or Gnome + some other stuff versus Windows, Open Office versus M.S. Office, Firefox versus Internet Explorer). If you compare how these large projects have worked in the open source community versus the projects of the largest software company in the world, I think you will find that these projects do indeed get reviewed by more people and that they do take security seriously.
But you can only compare the number of security bugs found in one project versus another, it does not say much about the other security features or the presence or absence of any other bugs.
So, suggesting that open source has lost the security edge, is just plain wrong because it's hard to compare the platforms. One can add more security features to Linux than to Windows (SELinux, Apparmor, Propolice, Grsecurity, dnmalloc), but they are not necessarily there for all distributions and depend on the needs of the user. Although there are a lot more applications for Windows, the actual operating system is alot less diverse than what can be found in the open source world, and as such a one to one comparison is virtually impossible.
That said, I applaud the stance Microsoft has taken with respect to security, but I would like to point out that many of the security features that they are adding to Vista (http://blogs.msdn.com/michael_howard/archive/2006/05/26/608315.aspx) is a port from what was first released as open source: e.g. StackGuard -> /GS, PointGuard -> function pointer encoding, PaX ASLR -> Wehnus -> Vista ASLR, Openwall -> Non-executable memory regions. So even Microsoft has benefitted from the security added to the open source community.
There are areas in which Microsoft is probably ahead of many open source projects: SDL is interesting, they are also probably more ahead in using annotations for static analysis, but open source has it's own benefits (some of which I mentioned earlier).
So my conclusion to your post would have been: it's great to see that people are focussing on security both at Microsoft and in the open source world and it can only benefit us all if this trend continues.
> Can you give an example of where a MS product has > superceded a comparable open-source project in terms of > security?
The current IIS release is said to have had fewer security bugs reported than Apache2. Let me clarify: I am *not* voicing an opinion, just reporting. I have not looked at the numbers myself, nor have I compared the 2 products.
> The current IIS release is said to have had fewer security > bugs reported than Apache2. > Let me clarify: I am *not* voicing an opinion, just > reporting. I have not looked at the numbers myself, nor > have I compared the 2 products.
Understood. Actually there was a bit of a trap in my question. I'm getting really sick of the following fallacy: "we have this great process therefore we produce good results." It's something that is held as dogma at my (soon to be former) employer. The reality is that the process isn't great and the results are absymal.
The point is not that processes aren't good. It's that you can't say 'we have a good plan' and then conclude the result will be (and/or is) good. So the point is that the results tell you about the process and not the other way around. I didn't see anything about results in this blog. I just saw 'MS has a process' -> 'MS may be surpassing OS'.
As far as the 'reported incidents go' I don't know about that specific claim however, I do know that in the past, it has been shown that Microsoft commonly under-ranks their issues compared to Firefox, for example. Just because an issue is not reported doesn't mean it doesn't exist.
This brings up another annoyance. We need to be very careful about how we decide to quantify things like this. I once found a solution to serious issue that was costing my (soon to be former) employee a lot of money, every day. By a lot of money, I mean more that the two of us make in a year lost in a months time. I went through the effort to put the fix in as soon as possible. I was told by a higher up to delay so that I wouldn't have to put it in as an 'emergency'. Why? Because this person's performance is tied to the number of 'emergencies'. So instead of working to reduce the number issues, we just hid them, because it appears cheaper and we appear more effective on paper, even though we are really wasting time and money and pissing-off our customers. Here the point is that if you rate these products by reported issues, you are discouraging people from bringing them to light.
First, let me say that I agree with you in the the results should tell you something about the process and not the other way around.
My disclosure is that I work at MSFT as part of the security improvement efforts and that, in particular, I try to identify metrics and measure results. If you talk to the security folks in our group, most will tell you that we think we're only a few steps along on the road to improvement, acknowledging that we have further to go, but we believe objective metrics show that we *are* making progress.
Having said that, we can compare the number of vulnerabilities and security patches that were necessary to address publicly disclosed vulnerabilities between Windows Server 2003 (which went through most of our improved process) with Windows 2000 Server (which did not). We get about a 50%+ reduction in Critical and Important security patches and a similar reduction in vulnerabilities during the first 12 months of product availability. New process, better results - that's good.
Over to your question wrt Open Source, in my blog posting of http://blogs.technet.com/security/archive/2006/05/09/427849.aspx I took Mark Cox/Red Hat metric which they applied to Red Hat Enterprise Linux 4 and applied it to the first year of WS2003 and charted them together. This metric, adapted by Mark from NIST weights higher severity issues more than lower severity issues - which seems reasonable to me. The differences are fairly significant.
I'd go further and ask this - what trend do you find if you compare successive versions of RHEL 2.1, 3.0 and 4.0 in terms of security vulnerabilities? Normalize it to vulns/month or vulns/day. Filter it by severity if you wish. Is it decreasing? To remove the possibility that an increase might be to an increasing number of components, try filtering it to just the 250 more core/common components - what does one find?
> Over to your question wrt Open Source, in my blog posting > of > http://blogs.technet.com/security/archive/2006/05/09/427849 > .aspx I took Mark Cox/Red Hat metric which they applied to > Red Hat Enterprise Linux 4 and applied it to the first > year of WS2003 and charted them together. This metric, > adapted by Mark from NIST weights higher severity issues > more than lower severity issues - which seems reasonable > to me. The differences are fairly significant.
Again, the problem with this is that if you look at the issues called critical in linux or other OS product, they are the kinds of issues that Microsoft calls 'serious' or lower. By weighting, you've only exaggerated the problem.
Unless there is some objective criteria that we can agree on for evaluating how secure software is there really isn't any point in having this discussion.
> Again, the problem with this is that if you look at the > issues called critical in linux or other OS product, they > are the kinds of issues that Microsoft calls 'serious' or > lower. By weighting, you've only exaggerated the problem.
I don't see how you support your assertion of exaggeration, I certainly didn't define that metric, and I'm pretty sure Mark Cox didn't design it to favor Microsoft. One might argue he wanted to make sure that there was less weight given to the hundreds of lesser severity Red Hat OSS vulnerabilities, but that hardly supports an argument that OSS is less vulnerable.
What do you suggest as an objective alternative for measuring the security quality of the software? I'm happy to re-run the work using, for example NIST severity scores or CVSS scores.
You asked for an example and you've now had at least two. How about this? You define an objective methodology that measure security of software. Then we can discuss the pros can cons of your methodology and once that's ironed out, we can apply it to same test cases.
> I don't see how you support your assertion of > exaggeration, I certainly didn't define that metric, and > I'm pretty sure Mark Cox didn't design it to favor > Microsoft.
And as far as I know it was never meant to be applied to Microsoft products. The metric isn't the issue anyway. It's the uneven ratings between MS and OS products.
> One might argue he wanted to make sure that > there was less weight given to the hundreds of lesser > severity Red Hat OSS vulnerabilities, but that hardly > supports an argument that OSS is less vulnerable. > > What do you suggest as an objective alternative for > measuring the security quality of the software? I'm happy > to re-run the work using, for example NIST severity scores > or CVSS scores.
I don't have an alternative. There are lots of things that can not be measured objectively and accurately. What's coming to my mind is the Turing Halting Problem.
> You asked for an example and you've now had at least two.
Two examples that I don't find adequate.
> How about this? You define an objective methodology that > t measure security of software. Then we can discuss the > pros can cons of your methodology and once that's ironed > out, we can apply it to same test cases.
Do you know of a single <i>independent</i> (no Gartner-type shills) organization that ranks these products side by side for security? I think you'd have to start with something like that. Maybe the Us government does this? I know they fund research for a 'hard' version of Linux.
> Do you know of a single <i>independent</i> (no > Gartner-type shills) organization that ranks these > products side by side for security? I think you'd have to > start with something like that.
OK, you don't know of any independent study that has compared OSS and MS security and you think that's a necessary step. So if you are right and there is, in fact, no such study, we shouldn't draw any conclusions about the relative security of MS and OSS software.
I make this point because it would be a bit disengenous to claim that OSS software is more secure but when pressed to define an objective criteria to measure security, to say that it isn't possible or nobody has done an independent study.
> OK, you don't know of any independent study that has > compared OSS and MS security and you think that's a > necessary step. So if you are right and there is, in fact, > no such study, we shouldn't draw any conclusions about > the relative security of MS and OSS software. > > I make this point because it would be a bit disengenous to > claim that OSS software is more secure but when pressed to > define an objective criteria to measure security, to say > that it isn't possible or nobody has done an independent > study.
Perhaps not explicitly, which is why I used the words "would be". On the other hand you did say:
"As far as the 'reported incidents go' I don't know about that specific claim however, I do know that in the past, it has been shown that Microsoft commonly under-ranks their issues compared to Firefox, for example"
The phrase "commonly under-ranks" suggests to me that you believe that MS isn't following some sort of understood standard, yet in a later post you suggest that there is no standard and it's difficult to define one.
So rather than me speculating on your opinion I'll just ask: Do you believe that OSS software is more secure than MS's? If your answer is "I don't know", fine. If the answer is yes, what is your objective criteria to reach such a conclusion?
> "As far as the 'reported incidents go' I don't know about > that specific claim however, I do know that in the past, > it has been shown that Microsoft commonly under-ranks > their issues compared to Firefox, for example" > > The phrase "commonly under-ranks"
Out of context: the full phrase I used was "under-ranks their issues compared to Firefox"
> suggests to me that you > believe that MS isn't following some sort of understood > standard, yet in a later post you suggest that there is no > standard and it's difficult to define one.
That's a poor assesment. I didn't say it was difficult to define a standard just that it was hard to find data on this issue where a single standard was applied objectively.
> So rather than me speculating on your opinion I'll just > ask: Do you believe that OSS software is more secure than > MS's? If your answer is "I don't know", fine. If the > answer is yes, what is your objective criteria to reach > such a conclusion?
When you say OSS, that encompasses a lot of things. My personal gut feeling is that MS software is not very secure but I have no objective data to back that up, no.
I'm open to the possibility that MS is improving in this area and that this process is a major factor in that.
I find it frustrating that it seems impossible to discuss Microsoft without someone bringing a highly partisan stance (pro or con) into it.
> > "As far as the 'reported incidents go' I don't know > about > > that specific claim however, I do know that in the > past, > > it has been shown that Microsoft commonly under-ranks > > their issues compared to Firefox, for example" > > > > The phrase "commonly under-ranks" > > Out of context: the full phrase I used was "under-ranks > their issues compared to Firefox" >
Only you know if your intent was to be even-handed or not.
> > suggests to me that you > > believe that MS isn't following some sort of understood > > standard, yet in a later post you suggest that there is > no > > standard and it's difficult to define one. > > That's a poor assesment. I didn't say it was difficult to > define a standard just that it was hard to find data on > this issue where a single standard was applied > objectively.
Then what did you mean by the statement: "There are lots of things that can not be measured objectively and accurately" when asked to define an objective standard for software security.
> I find it frustrating that it seems impossible to discuss > Microsoft without someone bringing a highly partisan > stance (pro or con) into it.
I'm not sure what you mean by "highly partisan", but you just admitted that you believed that MS software wasn't very secure without proof. Isn't that a partisan stance?
Note that I haven't been partisan since I haven't expressed an opinion about the relative security of OSS and MS.
Flat View: This topic has 18 replies
on 2 pages
[
12
|
»
]
Previous Topic
