The Evolution of System Development Life Cycles

I recently attended a presentation titled “AGILE in a Waterfall World”.  The information on AGILE was informative, however the presentation made me realize there are still organizations using a System Development Life Cycle (SDLC) developed in the 60’s, Royce’s Waterfall model.  Maybe Waterfall has stood the test of time, but does a sequential methodology still work in today’s “technology as an enabler” environment?

Before answering that question, let’s look at the history of the SDLC.  The Waterfall Model was introduced by Dr Win Royce in 1969.  The goal of the model was to promote an orderly development of a large software system.  The large systems developed in the late 60’s early 70’s were systems that automated very labor intensive manual processes; for example: accounting systems, insurance rating systems, and banking systems.   For all of these systems the manual processes were well known, which allowed requirements to be easily documented.  Additionally there was usually a limited, controlled interface between the user and the “system”.   For example, in the mid 70’s  insurance industry there were just 2 interfaces between the mainframe system and the users.  The data entry staff “converted” policyholder request captured by an agent into machine readable code by typing special characters onto a piece of paper which was then scanned into the system (interface 1).  On the backend the system printed a piece of paper (interface 2).  The piece of paper was then manually compared to the changes documented by the agent.  The system had automated a manually intensive rating process.  Instead of a person manually looking up rates and calculating the price of the policy (a process which took several minutes), the system performed the same calculation in seconds.  The key in understanding the system development effort was that the computerized system was replacing a known manual process and therefor the requirements were well known and the project had minimal risk. In that environment the Waterfall Methodology worked extremely well. 

In the early 80’s (1983 to be precise) Dr Barry Boehm introduced the Spiral Model.  It is a two dimensional software development model where the final development phase looks very similar to a waterfall.  The difference between Waterfall and Spiral Models is that the Spiral uses several iterations to help develop the final set of requirements.  Between each iteration there are a series of evaluations where alternatives are compared against their risk in order to develop final product specifications.  Development is still sequential, however the concept of prototypes is introduced.  Why prototype?  That can be answered by looking at what is being developed in the 80’s.  Staying with the Insurance Industry, let’s look at the type of automation that was taking place in the late 80’s early 90’s.  Remember the old rating system depended on the data entry clerk memorizing hundreds of codes in order to create the typed input for the computerized system.  Acceptance testing of that input was completed on the back end by comparing the policy output to the agent request. It took months if not years to train a data entry clerk on the various codes and errors took days to correct.  The focus in the 60’s was automated the core process, now the focus of system development had shifted from the process to the interface.  The industry was taking advantage of system capabilities to change and or create new processes.  The insurance industry wanted to eliminated the data entry learning curve and after the fact acceptance testing with a system that allowed plain English data input.  The new system was the interface which converted plain English into input for the mainframe process (the core application did not change with the introduction of the new interface) and acceptance testing could now be completed before the data was processed.  There were new risks: the old system was as fast as the person could type and “load” paper into the typewriter.  The new system introduced the impact of screen navigation and response time.  The risk and the requirements were not known – we were no longer just automating a known manual process, we were changing the dynamics of the business and had to evaluate the impact of our decisions as we developed the alternatives. The Spiral model with its focus on risk and prototypes made perfect sense in an environment where technology was in itself the enabler and the new processes were unknown and not well defined.

We progress to the late 90’s and realize that systems are no longer stand alone processes, we now have the mainframes, midrange and personal computers and applications now have the potential for multiple interfaces based on the hardware around them.  The system development model has now evolved from a software development focus to an integrated focus of hardware and software. Figure 1.2 in CMMI version 2 (2007) illustrates the evolution and integration of the software development models with the system (hardware) development models.   In 1991 Forsberg and Mooz introduce the Vee development model at the first annual NCOSE Symposium.  (The Dual Vee version of the model was introduced in 2004 as a way to address the increasing complexity and size of the systems being developed.)  Even though I have been in presentations where the Vee was described as a “Broken Waterfall” it is not and never has been a linear model.  The Vee model starts by focusing on requirements, but only verifiable requirements, which means verification planning starts as soon as the initial requirements are defined. (Unlike the waterfall or spiral where verification planning only starts after development has been partially completed.) When the actual testing takes place, months or years later, the system requirements are still being validated.  Why?  Because given today’s rate of technology change, the chances the requirements have changed over time is extremely high.  Waterfall and Spiral do not adequately address those issues.  Let’s look one more time at the insurance industry.  Up until several years ago the only system a company within the insurance industry cared about was its own proprietary system.  Today, you don’t even need to talk to an agent, but you can access their rating system through multiple technologies (cell phone, PC, MAC) of your choice using the web interface (Internet Explorer, Firefox, Safari, Chrome, Android) of your choice.  Plus in order to make the process as error free as possible, the insurance company now interfaces with 50 different state auto registration systems enabling them to auto-populate your car’s VIN number based on the cars registered to that address.  Unlike the 80’s and 90’s the insurance company can no longer just focus on its own proprietary system, but all of the systems that interface with its system. 

The Waterfall and the Spiral SDLC’s do not address that level of complexity.  So why are they still being used?  I’m not sure, but my best guess has to do with the realization that most system development (maybe 70 - 80% of all system programming?) is still maintenance.  In a maintenance world Waterfall and Spiral still make sense.  The system is known and the maintenance is focused on a limited number of requirements. However Waterfall and Spiral development are not appropriate models in a new development mode. Neither do a sufficient job in addressing complexity.   

Now let’s get back to “AGILE in a Waterfall World”.  I agree with the speaker’s premise – AGILE is difficult to implement in a Waterfall World.  On the other hand – The Dual Vee model actually embraces AGILE as a means of implementing any of the appropriate subsystems being developed whether they are hardware, software or a combination of the two.  The challenge should not be to make AGILE work in a Waterfall World – the challenge should be to adopt an SDLC that works in today’s system of systems environment. 



Over the last year, I have been teaching a combination of SE and PM courses,often to the same organization, but with different student bases and different attendees in the classes.  One of the concepts I have become a firm beleiver in over the years is the use of a Product Breakdown Structure (PBS) as a starting point for the development of the Work Breakdown Structure (WBS).  Students who transition from a Requirements Concepts and Architecture course grasped the idea quickly and had no difficulty creating a PBS in a reinforcing exercise.  However students who started with the Technical Project Management Course had a difficult time performing the exercise.  As I worked through understanding the root cause of the difference, I realized that the Project Management literature was vague on the process. In fact the new ISO Project Management Standard (ISO 2150:2012) states that the primary inputs to the WBS are the Requirements and Approved changes.  The WBS is then the primary input to the development of activities.  The document goes on to state " The purpose of Create work breakdown structure is to provide a hierarchical decomposition framework for presenting the work that needs to be completed, in order to achieve the project objectives."  The process as documented ignores the point that there are multiple means to fulfilling the requirements and that the proposed solution is what is actually decomposed into the WBS.  System engineers are trained to look for alternative solutions and then pick the "optimum" solution given the project objectives.  Since they undrstand this tradeoff they grasp the concept of starting with a PBS.  On the other hand, project managers who do not have a background in system engineering actually try to dcompose the requirements, which is incorrect.  Requirements get allocated to solutions and solutions get decomposed. 

In addition to the standard project management certifications, there are related certifications in Business Analysis, Risk management, and Scheduling.  For the non-SE project, shouldn't there be a certification in Solution Development?

What are your thoughts?

Another word for estimate is guess.  It should be your best guess in relationship to task duration, task effort and task cost.  There are several difficulties associated with obtaining that best guess; is the work well defined; are completion criteria understaood by all; who will be performing the work; is there a basis for the estimate (past performance or expert opinion) or is the estimate being provided through the eyes of an optimist and or a pessimist?.  These are all variables that are difficult to control, however there is one variable we can manage, and that is when the estimate is developed.  We encourage all of our sudents to develop the work effort and duration estimates prior to sequencing, so that they are not unduly influenced in their estimating process by management desired date. Instead we ask students to estimate, thensequence and finally reconcile the differences between what management wanst and the schedule as developed.  Unfortunately, this is not the process docuemented in the new ISO Project Management Standard (ISO 21500; 2012), where Annex A represents the flow going from Define Activities, to Sequence Activities, to Estimate Activity Duration.

Personally, I think they have these backwords, what do you think?

Several years ago (early 90’s), I managed a benchmarking project for an insurance company that was trying to determine the best practices for managing all aspects of an investment risk.  I was working with investment officers and actuaries.  After performing research and due diligence, they identified one company that was the best at managing risk.  That one company was Bear Stearns, one of the key players during the financial crisis and a company that collapsed due to their inability to manage the risk within their portfolio.  I can only speculate on what happened to Bear Stearns; but I know that it is not unusual for projects to fail even when they have well defined risk management, because they lose sight of the corresponding opportunity they are pursuing with the risk.

For example, Iridium was, and is, a very capable satellite based telephone system.  The original business model, however, had it competing with cell phone technology.  They addressed the technology risk; but they failed to take into consideration the rapid adaptation of cell phone technology, which drove scale, which resulted in significantly lower costs.  In fact, cell technology scaled so quickly that the Iridium solution was no longer competitively priced, forcing the company into bankruptcy. 

Another company which I consulted with had the backing of several very smart Venture Capital firms.  The company was installing internet access into apartment complexes and hotels (multi-unit complexes).  They were leasing the equipment versus selling it, which kept cost of entry for the multi-unit complexes very low.  It was a sound model compared to companies that were selling the hardware to the multi-unit complexes, except they did not anticipate the rapid advance of wireless technology. A technology which had a very low cost of entry caused this company to go bankrupt.

There are a lot of companies that teach risk management, and even more that teach risk administration.  What makes PD Mentors unique is that we couple risk management with opportunity management.  A project takes on a risk to take advantage of an opportunity.  Your projects need to track and manage both.

Does your risk management process address the actions you need to take when the opportunity changes?   Don’t you think it should?