Summary
Back on Aug 8th, 1988, I started my first day at the AT&T Bell labs facilities in Naperville IL. When you first walk into a really large facility like that, you are overwhelmed by the size. There were so many people there, and so many things going on, like...

---

Advertisement

The Naperville Complex in 1988 was home to 5ESS hardware design, UNIX RTR development, the Teletype group, responsible for the DMD5620, 630 and later 730 and 730r2 terminals, Domestic Market (US) 5ESS software development and International Markets (China, Taiwan, UK and Netherlands in 1988 time frame). Across the street in Wheaton was the customer support center with the ODA group, domestic 5ESS labs, the commercial demonstration lab, etc. In total, there were about 11,000 people working in that complete complex. The 5ESS development buildings were located in the Indian Hill complex. There was IH-Indian Hill Main, IX-Indian Hill South, IW-Indian Hill West, and IHP-Indian Hill Park (about 1 mile away). The NSC-Network Software Center was the name given to the customer support center across the street in Wheaton (to the east across Naperville road). Today, the outside of these facilities resemble the pictures at http://www.theaustin.com/html/nr-rbi-lucent.html. Behind each of these modern structures, are the old structures which are much larger than these!

I spent 9 1/2 years at these facilities. I worked in the Software Release Retrofit group the whole time. I experienced ISO-9000 registration and validation exercises. We had extensive diversity training which was a great necessity in such a large place with such a diverse group of people. I learned things about myself and our societies that were quite alarming.

The Retrofit Software...Problems

When it came to software, I continued my facination with tools and productivity enhancements. The 5ESS Software Retrofit procedure was implemented using 4 layers of process structure. The RTR layer of C-Programs called the 5ESS layer of C-Programs which did some things to the 5ESS hardware. But, if we needed to do something to the RTR environment dealing with files or directories of files, we'd invoke shellscripts that would then run various UNIX flavor commands such as awk(1), sed(1), cp(1), rm(1) etc.

The problems with this environment were numerous. First, the shell script were notorious for containing

if grep pat file1 | awk '{print $1}' | sed 's/..//' | grep pat2 >/dev/null 2>&1; then
    ...do something...
fi

Now, when something went wrong, you had no idea what went wrong, and, if 'grep', or 'awk' or 'sed' complained about arguments or otherwise exited abnormally, the final grep would return an erroneous result, but noone new the better.

So, we developed logging facilities, change things to not redirect I/O etc. But still, the pipelines caused problems, as did other issues related to problems such as

val=`prog | grep foo | sed 's/..//'`

where if you don't check the exit codes of everything everywhere, you just can't guarentee anything is right.

Where Did These People Get Their Training?

As I was thinking about this issue, I came across some other related problems (training or lack thereof) while reviewing code for other projects (we had rigorous code reviews, and a diversity of inspectors was expected). I saw code such as

system("sleep 30");

in a C-language program. The person doing this had never been taught that the shell that they ran had to use 'sleep(3)' to do that sleep and that creating a process to put to sleep was a little bit of an overkill.

In another code review, I saw this code:

/* Store string into buffer */
strcpy( buf, "my string");
/* Terminate string */
strncat( buf, "\0" );

which was a completely amazing thing to see. There are two very basic mis-understandings of the core of the C-language string functions visible here. I just had to wonder how so many blatantly obvious, lack of training issues were getting by these inspections, that seemed to be attended by pretty knowledgable people.

Well, the next example was more elaborate. Part of the retrofit procedures was a program provided by the RTR group that was the menu control program that managed the steps that were supposed to be executed (5 total). This process used SIGCLD to catch process terminations, and show that the step had either completed, or failed. The problem was that it would register SIGCLD handlers, and then leave them in place. The SIGCLD handler used longjmp to break out of the loop that was being used to show progress updates. If the progress update found a problem, the code could return without canceling the handler. When the process finally completed, you got longjmp(2) calls back onto stack frames that were now invalid. The process would crash, and you had to restart it.

I have no idea how this problem made it through the code inspection phase, let alone the testing phase. I found it within about 1 hour, and fixed it quickly, and then spent the next 1 year trying to get it pushed back into the RTR load.

Finally, there were numerous examples of poor pointer handling in the C-language portions of the Retrofit code. There were complex arrays, linked lists, and numbers string buffer overflows that all made it clear to me that something quite different and quite dramatic had to happen.

Changing Everything Overnight Might Work?

From my days at OSU, I had the start of a language that was going to be TPU (the VMS text processing utility that I wrote a VI emulator for) for UNIX. The TPU language has no compile time type checking. Everything is done at runtime. This makes for a very easy to create and maintain compiler, but requires a more complex interpreter, since you have to check everything at runtime as many other more dynamic languages do.

I dusted this thing off, and decided that I wanted to create a new process control structure that would invert the control tree of the retrofit software environment. I wanted to have just two layers. The control layer, and the work layer. I wanted to run as few unix commands as possible because fork and exec was slow on the 3B20D (.75 mips! and you only get 1/3 of that max).

The language I started creating (deriving from some of TPU's control structures), would initially do two things well. It ran and managed UNIX processes well, with support for asynchronous process execution. It included support for creating recovery labels in the object stream of the compiled program. The language then supported recovering execution at one of these labels so that you could resume some complex script at the same place you stopped at last. This was very important to managing complex procedures.

The language state control was basically the following:

st = loadState("foo.st");
if( st != NOSTATE ) then
    resumeState( st );
endif;

...

st = saveState( "foo.st", "state1", var1, var2, var3 );
if( st == BACKOUT ) then
    ...undo things and exit(0)...
endif;

...

st = saveState( "foo.st", "state2", var2, var4 );
if( st == BACKOUT ) then
    ...undo things and exit(0)...
endif;

This let you BACKOUT of a procedure, knowing where you had gotten to and thus, what needed to be undone (if anything).

Late in 1993, I decided that I needed associative arrays to get some of the things that we did in awk(1) to be done in the language. As I was adding associative arrays, I was also exploring another language, Keynote (now Keykit) by Tim Thompson (www.nosuch.com). This is a midi event language that lets you develop code and UI that provide a midi composition environment that is very powerful. At that time, Tim was converting keynote over to objects, and I noticed how he had just reused associative arrays to provide classes.

I got the initial implementation of associative arrays done, and decided that I really should do objects too. So, I made class definitions just be associative arrays (hashtables) and suddenly, I could do a lot more with the language than previously. It was clear to me that the investment in Objects was a good one. My fellow group members were a little bit stressed to have to learn OO, but my supervisor liked the idea that our group was the only one in the 5ESS project using OO at that time :-)

The Web is Weaved by Objects and Keeps Growing More Trees

Earlier, I had been swept up with the concepts of web programming. I had created an TROFF document formatting CGI environment which had made about 10,000 5ESS documents accessible on the intranet inside AT&T (and later Lucent). I was intimately familar with the whole CGI programming business, and I was also frustrated by having limited control over the format of the documents on screen (we had no <table>, remember?).

My development environment, during this whole time had been a 630 and then a 730R2 terminal. I was very familiar with downloading code to run applications, and wrote numerous tools for the company employees and myself. These were distributed through the infamous exptools mechanisms that provided remote building and distribution services that made it easy to build and distribute tools for multiple platforms.

I had an interface to the MH mail system that was a 630 application. It used the USENIX faces facilities for displaying faces. I wrote a window manager for the terminal so that you could tile windows away and free their bitmap space for use by other applications. This was laborous programming, but it was necessary to get a good, productive environment.

Is There a Tree Growing Here, or Just an Flowery Shrub?

About the time I got the object stuff working in my language, and was starting the next phase of integration into the Retrofit processes, Java came on scene. Wow, I was shocked. Someone had actually had the same experiences, and came to the same sort of decisions that I had about fragile languages and overly complex APIs! I believe that would have been the fall of 1994.

I picked up Java at that point and just rejoiced! I had learned a lot of lessons about programming languages, operating systems. The investment I had made in my life (12 years or so) seemed huge. Now, I am comming up on the 10th anniversary of the release of Java (next year). 10 years of Java, and it is still expanding and maturing in all directions.

What have you invested in, and where has it taken you?

Talk Back!

Have an opinion? Be the first to post a comment about this weblog entry.

RSS Feed

If you'd like to be notified whenever Gregg Wonderly adds a new entry to his weblog, subscribe to his RSS feed.

Digg | del.icio.us | Reddit

About the Blogger

Gregg Wonderly graduated from Oklahoma State University in 1988 with an MS in COMSCI. His areas of concentration include Operating Systems and Languages. His first job was at the AT&T Bell Labs facilities in Naperville IL working on software retrofit for the 5ESS switch. He designed a procedure control language in the era of the development of Java with similar motivations that the Oak and then Java language development was driven by. Language design is still at the top of his list, but his focus tends to be on application languges layered on top of programming languages such as Java. Some just consider this API design, but there really is more to it! Gregg now works for Cyte Technologies Inc., where he does software engineering and design related to distributed systems in highly available environments.