Introduction.

On the Upper Nile River lies the Valley of the Kings, a place the ancient Egyptians buried their kings, queens and noble people. Many of the tombs were hidden, the pharaoh Tutankhamen’s was only discovered in the 1930’s. The ancient Egyptians went to great lengths, especially with later tombs, to ensure that the riches placed in the tombs for the dead to be happy in immortality with, were not spiritually impoverished a few days after the burial by local robbers. Various devices were placed in the passages to prevent penetration of the inner sanctum.

It has struck me on occasion that the very same goes on with our software, here in the 20^th Century. We seem to do everything in our power to make the understanding of our ideas as difficult as possible. Yet, as we proceed into a second half century with Computers, we are left with the task of determining, not just the actions of a certain program, but the reasons for its existence.

The inner sanctum of today’s software is the understanding of what the program stood for, what its reason for existence is or was. Without getting too philosophical about a program; after all it is just bits on a disk or black ink on paper, there are reasons we might wish to understand a recalcitrant program.

Software Archeology is the art of reviewing software and determining a number of facets about that code. We will attempt, most obviously, to determine the process that is bound into the lines of code. We will also frequently attempt to determine reasons for the actions taken in the program, and lastly we may wish to know something about the environment in which the code was created, or in which the code was worked on. Just as ancient earthenware pottery shows signs of use, so too does software. It is the reading of these that can help us understand better the software around us.

Real archeology is very much a science, there are clear principles. With old enough artifacts there are physical processes that may be applied that help display age and features. These details help an archeologist place an item into the established view of the world. No one item viewed in isolation is of much use but taking several items, find connections like age or style, and soon we can create a view of the world in which these items once existed. Math and hard science play a part in archeology, providing the field with a background that helps to solidify theoretical views.

Software Archeology is no more a science than writing software, but it does share some aspects of the real science: something exists that needs to be rendered into a view of a past epoch, something occurred and as a result was recorded in the software, someone or something once worked with the software. Software Archeology is a tool to help us peel back the mystery, help determine in a more logical manner the features of the software that are not necessarily obvious, outline the purpose of the software, and ultimately recreate the world the software inhabited.

Even though most archeology deals with artifacts that are hundreds or thousands of years old, Software Archeology is dealing with things that are only a few decades old, at most. Yet, such is the nature of software, and the pace of change in the computer industry, that features of the industry just a few years ago, are today just fanciful recollections. We are only just, in human terms, saying goodbye to the first generation of programmers, however the rate of change means that every 5 to 6 years a completely new generation of programmers has essentially washed away an older generation. This has lead to the quite rapid loss of knowledge about former programming generations, how they worked, and the tools that were available to them.

It is this knowledge that computer archeology can help restore. It is this knowledge that helps in creating a picture of the world that the software lived in, a picture that helps in placing the concepts that formed the software and possibly lead to its demise.

Computer History.

I’m not going to bore you with my version of computer history, your too bright and I’m too dull to subject either of us to that. However, we do need to know something of the environment that was prevalent at the time of our software’s inception. As we are not dealing with pre- history, its fair to say that the average second hand book store can be turned into a veritable Aladdin’s cave of knowledge. Here, we can find books on IBM System/360, or Tandy TRS-80, on ALGOL, Fortran-66, punch tape and punch cards, about core memory or PC- XT’s.

Some software you will deal with can be new enough that a look in yesterdays Times may be sufficient to tell you of the operating environment.

What do we want to do with the history? We use the history to establish a set of boundaries that the original programmers and designers had to work in. For instance, old programs may not refresh the screens, instead just scroll text uselessly off the top of the screen. Well, if you were not dealing with a modern GUI, but with a Teletype, you could just lean over the machine and pick up the heap of paper output and read that important text. Programs may be old enough that their original environment has completely vanished. Old computers had far less memory than we have now, so the programmers resorted to tricks of the trade to squeeze functionality into very tight spaces. Today the code looks a jumble, but once its was a miracle and a paragon of efficient coding.

To gain an insight onto the history surrounding the code, the best option is talk to people that were there. Talk to programmers that worked on the software, to managers that may have been programmers on it, or were just more junior managers at the time. Even, dare I say it, talk with the former users of the program. Be aware: computer people are likely to embellish the tales of the time painting pictures more reminiscent of the old Wild West, describing programming groups against the odds, where all around them were wild animals, crazed managers and insane timelines (What’s changed?)

Don’t forget that software written much before the mid 1960’s will be of a form we would today call unstructured. There is a good chance that if the code is new-ish, that it is also unstructured, why else would you be called into to dig through it? Software development methodologies, a mainstay of today’s software world, were not in vogue prior to the 70’s, this means that any software you get to look at may be quite ugly to your eyes. However, you are probably looking at the reason we now have those development methodologies!

The Software.

So you have a stack of listings a foot high, or is it just a single floppy disk? The first thing to do is look at it. If it’s a disk, always virus check it. They did have viruses when this disk copy was made… especially if the disk was copied just yesterday.

I find that first impressions are useful. Get a program editor, or print all the code out, look at it and try to record your thoughts. Does the code look homogenous? Is there obvious changes in style. Are there lots of GOTO statements? Does it have indentations? Is it neat or untidy? Are code comments immediately obvious (Comment Deserts are a frequent problem in this field).

Note; we are not trying to determine straight off the bat what the code is doing, just like a good hunter we are looking the place over, making sure that the country looks right. We are not going to catch a wild animal by just jumping in.

Frequently people will say that they can’t print out all the code, it covers too many pages. Don’t let this be an obstacle. If you think that you need to ruin a tree or two, do it. Just be aware that the original programmer probably did not have a 16 page per minute laser printer, more likely they had a big line printer or a dot matrix printer. The output will look different. They may have had 132 columns, you might only have 80 columns on a portrait oriented piece of 8x11. Also note that a listing made in mono-spaced type is more likely to retain position critical comments and code.

Make sure you have copies of all the software, you are going to totally alter at least one full set of the code, so its essential that you get photocopies of original printouts and make backups of disk based items.

Logical Process.

All good scientific processes have a logical, or planned, path. To be considered half way useful, I suppose we also have to get stuffy and come up with a process for our dig. We have a pile of software, some initial impressions, and an aim (or two). It is probably worth looking at what has to happen to get from A) "Some initial impressions" to B) "an aim (or two)."

I prefer to work with a word processor or programmers editor, it allows me to make notes that are independent of the software, this can be used as the basis for a subsequent report (doesn’t everyone love reports). If the software is on disk then a version of it can be displayed alongside the word processor taking the notes. This alone, in my mind, justifies that 20 inch monitor and multi-megapixel video card you have been hankering after. You will need lots of screen acreage, almost as much space as your real desktop is going to require if you have listings.

The note pad (the word processed notes) will contain the notes we make on the computer about what we find, and any random observations. Always start off by putting the program or module name and date at the top of the note, then save the note pad. We can say we have now done something and print it to prove the point.

Jot into the note pad the initial observations, list the following if possible:

• Form in which the software was delivered

• Condition of the code as it was first observed, include comments related to the initially observed code style.

The code will generally be attacked in a linear manner. Start at the beginning and work from there to the end. If you can find the main() function, or the program start then so much the better.

Try not to jump from calls in the code to functions, not just yet anyway, things can quickly get out of hand. This initial passthrough, which can be the most time consuming, is intended to allow you to see the lower levels of detail. I know this varies from person to person, but I tend to find it easier to get a bottom–up view of the code. I find this is more helpful in determining the structure than trying to find a structure then finding the detail. There are people I have met that have no trouble skimming code and finding the structure, prior to the in-depth attack. It’s all personal, but follow my line for a time.

Get a feel for the complexity of the code. If there are few lines of code in each procedure or paragraph, just lots of calls to other items, try to surf over the code more quickly. Make comments in the note pad about items of interest; look for variable and function names that look quirky, and jot them down. Also, as you are going through the code try and write comments into the body of that work, these should be your first stab at attempting to determine what small fractions of the code are trying to do. This will also help with taming the complexity.

Where possible try to establish the writer of the code. You can do this by looking at any comments and relating initials (some of us do put these in) to code modifications. Determining differences in style can give you a guide to the ‘layers’ in the software. These refer to the different periods the code was modified in. Just like real geology, one layer of code will become obscured by a later layer of code. This can date features in the program, and it can help in identifying the build up of the program.

It’s a truism of programming that code which gets the most fixing, continues to require the most fixing. If you come across a section of code that looks like Dresden c.1945, it’s a good candidate for something that provided the original programmers with hours of ‘fun’. This sort of find can be the most disheartening. If the original programmers had trouble understanding what they had created, you will have trouble understanding the mess they left behind. My suggestion is to skip it, unless this is the nub of your trouble. Make notes about pertinent details, then move on to the next section of program.

When a first pass has been complete, you should be able to identify where the program starts, possibly where it ends, and you may probably have identified the prime region(s) you are interested in. This is still not the time to dig into it.

Try to create a listing that shows the structural elements of the work, look at your notes about the functions you found, and use this along with the comments written in the code to help you. Essentially, you have to do a low grade reverse engineering of the code. This will help you identify to others where items of interest are in the program. You could even start to make a list that shows the linkage between functions, though this is probably a little excessive.

With a rough idea of the structure find the program start and proceed to bench run the application. The level of detail you do this to may depend on how lost you get, or how much detail is needed. I rarely (never) test the scope of every variable. Its simply too much work. On the first look through, you probably noticed variables that flowed through the code, test these. Give the variable realistic values and note what happens. These bench runs will become our theories as to how the internals of the program are functioning.

Dig deep into the areas you have identified, find the links between your specific area and other areas. Be careful to pull the plug on a search if you start to haul in the operating system API, know your limits. Annotate the real code with functional descriptions of what is there, and what sections are doing. Make note pad entries explaining what you are trying to find, and also what is coming out of the inspection.

Tools

If this software is in a form that can be run, and you have a machine capable of running it and you have debug or runtime tools… and you have the time, run the application and test your ideas about the scope of some of the values that you have an interest in. Note variances, and especially, theories that are proven.

There is a ton of old dBase code out there. Just about all of it can still be run in the development environment (the infamous DOT prompt). For older Cobol or Fortran code, there may be no way to easily execute the program and test its workings. I have been known to rewrite sections of code I have an interested in, in a language for which I do have debug tools (a favourite is BASIC, and more recently Visual Basic). If the application was written in C, there are any number of source level debuggers (XDB, CodeView, TDebug, etc). If the code is in something more esoteric (PL/1, PL/M, Snobol) you may either have to use the proprietary tools (Intel produced a thing called PMON for debugging PL/M 386 code), or forgo the luxury of testing your theories.

One of the most important tools you can use is a GREP. This is a UNIX inspired tool that allows you to search for text in files. If you have a collection of files this tool is indispensable, it allows you to hunt for words or phrases through all the files, or a subset of the files if you so choose. GREP can be a life saver. Where code becomes a contorted mess, GREP can be used to find the most fragmentary strings, this can help you zero in on what you are looking for.

GREP or the equivalent is available for most platforms, and if yours doesn’t have one, its not a tough program to create (we have to be masters of our tools).

Rounding Out.

The reason we look at old software is mainly to determine what a new application will have to do, and to determine why the old software got into the mess it did.

As you work through the code, continue to keep in mind that there are things here that are essentially company history. They may be artificial limits. I recently looked at code that the users said had a feature with a limit of 300 or so entries. As it turned out, the limit was not 300, but 32,000 (and change). A programmer had taken a number like 123.45 then scaled it to be an integer, giving 12345.0. Well, the environment only allowed 16 bit integers, so the most significant number that could be used was 327.67. The users had been backed into corner for 15 years by the actions of a programmer. And of course no one knew the reason why this 300 entry limit existed any more. It was painfully obvious as I looked at the code, that a hardware limit 15 years ago, had turned into a business practice horror today The understanding of issues like this can help to clear the air.

Gather your paper, the word processed notes, listings and other odds and ends and think about formatting this into something that can read by normal humans.

The Report.

I always attempt to produce a report, all this effort is useless unless the information is disseminated, after all, the reason you got this fine job was because knowledge and information had been lost in the first place.

I generally think the report should contain a discussion of the state of materials as they reached you, an outline of the functionality of the program, perhaps outlining why this dig has taken place. The document should contain a full description of the actual structure of the program, try and include block diagrams of the structural elements. If you managed to use a reverse engineering tool, some of these can puke out diagrams in staggering numbers, a few of these may be of some value assuming they are not too detailed.

Describe all the features that struck you as important, outline features that you found. Go into significant detail about the parts of the program that the dig was intended to unearth. Give detail about the logic you uncovered, about the variables used, any external factors (file formats), in fact anything that will help in describing the program facet.

I often also include the annotated program listings. Go through the listings, set the real code into some small, mono-spaced font, then use a larger bold proportional font for your annotations, this makes them stand out.

Assemble this wade of paper, put a cover and a back on it, melt on a spine, and prepared to argue points of view.

Conclusion.

Software Archeology is more than simple software maintenance, yet it is less radical than reverse engineering. Software Archeology serves a purpose in a world in which we have a huge number of unsupported legacy systems, systems that were written as Corporate throwaways, but ended up being cornerstones. We need to learn what the application does, but frequently only to do minor surgery, or encompass some of the old programs’ functionality in a much larger, more able program.

While reverse engineering produces a new specification for the software, this may be completely unnecessary, in fact it may even be counter productive. A specification will have to be checked against the old application by hand in all certainty, thus diminishing the worth of a machine generated output. The understanding of the program by a human is the prime objective. Software Archeology provides a frame work in which to attack the task of understanding that software.

Your software may not contain a Tutankhamen, but in all likelihood it does contain treasure. Treasure of significant worth to your company.