Software Development Overview

All software, outside of the most trivial examples, handles data (some form of digital representation). With that data, software basically does four things:

1. Capture
2. Process
3. Store
4. Output

In each case what the data is and how it is handled differs, but really that is the crux of software development. Leave out the hardware on which the software runs. These four items are it. That said, each one of them can be drilled down into very deeply, with each layer yielding more approaches, ideas, philosophies, etc.

Methodologies on how to create each of these pieces and make them work together compose volumes. Whole companies exist to serve small parts of each of the approaches.

That said, it is still just as simple as these four things.

Data has be be captured from the environment outside of the computer program, brought in from a keyboard being typed on by a human, from a file on a disk being read, from a camera storing an image or one of many other approaches. There are a multitude of ways for capture to occur. There are fierce debates about the best way to do so.

Processing and storage can be done in either order, but I am putting processing before storage because normally something happens to the data before it is stored for future use. All the clever algorithms for sorting or determining values that take up so much time on the discussion boards fall into this bin. Again, debates are fierce about approaches to processing.

Storage can be done in various ways, being slightly less contentious. Physically the data winds up in the memory of the computer, either volatile (in RAM) or non-volatile (a storage medium like a disc or flash memory). The way the data is stored, the format of it, can yield some debate but not nearly as much as where it should go and the method for retrieving it.

Output is communicating the processed and/or stored data to another system, be it a human or another computer. For humans this can be something like a monitor, speakers or a printer. Something for one of our senses to experience. For another computer (or the same computer, running a different program) it can be bits transmitted over a network or stored onto a disc (somewhat overlapping with the storage aspect).

As I said earlier, each of these items can be expanded on to very great extents but it is instructive to be able to come back to them whenever you start to get lost in the chaos of information that surrounds all human enterprise and by extension the computer software that supports it. Most often each of the pieces can be separated from each other and you can make decisions based on what works best for you for each one instead of having to put all of your eggs into one basket for all four items.

Philosophy surrounding the development of software to do these four things run the gamut from just sitting down and starting to write code to huge processes that involve many people who think for a very long time about what they want to accomplish and how they want to do it and how they will measure their efforts, etc. There are fervent proponents of each approach who tout the benefits of their favorite and will stick by it dogmatically without ever thinking that there might be another way to do things.

The techgnostic approach is to look at the problem you are trying to solve and then line it up with the four items that software development requires. Very often you can crank something out quickly that gets the immediate need addressed but will come back to haunt you later. You can also spend a huge amount of time trying to get everything perfect the first time. Try to shoot for something between the two, subject to constraints like safety.

If you are building software to control a medical device or drop a bomb you need to think long and hard about it and the solution will tend towards the more restrictive process. If it is a program to dump some data periodically for someone so they can run a trending analysis it requires very little oversight.

If you look around yourself, at the software that you use (like the browser you are reading this on) you can quickly discern where the four pieces are that make it work. Did the entity that produced the software make all the pieces depend tightly on each other, or are they interchangeable? There are probably other options for what you are trying to do that approach the problem domain from a different perspective. Do not get locked into only one way.

I will return to this subject over time because a lot of philosophy about how to get things done is buried in it. Cowboy programming versus gold plating. Architecture versus expedience. The topics are legion. To complicate things further there are significant issues associated with the process that transcend these four items to do with the management of people and the economics of the entire process.

The hard stuff is easy, the soft stuff is hard.

Technology, of whatever type, is a tool and that is all. Outside of enjoying a tool purely for its own sake (tool-o-phile?) there is no value to a tool except what you do with it. You don't buy a drill, you buy holes. A shovel might represent a crystallization of effort with a forged head and carefully turned handle, but it is just a shovel. Until someone picks it up and digs a hole with it there is nothing to see, move along.

"High" technology needs to be viewed the same way. It can become VERY complex and challenging to understand, but ultimately it is just a tool. A database without data in it is an empty shell. Even when full of data it means nothing until you DO something with it. Transform the pieces of data into information that you can use to make decisions. Again, the technology by itself is nothing. It takes people to use it to get value from it.

Technology is for most people the "hard" stuff. People skills like communication and collaboration are considered "soft" skills. Depending upon your personality type you might gravitate towards one or the other. In the end, though, the effectiveness of whatever your enterprise is will rest upon the behavior of the people using the technology. In most cases the hard stuff will come with a default setup that works in most cases, and if it doesn't you can find someone who can make it work for your situation. Really, in the grand scheme of things, that was the easy part.

The "soft stuff" is where the real challenge lies. Many technology people consider the soft stuff relatively worthless as it does not improve the speed of their hardware or software. It does not help directly with their up-time. It does not give them more disk space to store data. Very often they will gloss over the people-oriented items or leave them to others.

This is a mistake.

If you don't take into account who is going to be using your system you will fail. The end users of the system are most often not technologists. They are people who want to use a tool to get their job done. They want to get in the car, push the gas pedal and drive somewhere. They don't want to have to know any detail whatsoever about how the engine works, how the linkages between the control systems ensure that the vehicle goes in the correct direction, etc.

Gas, go, done, next.

You can build the most technically elegant system in the world but if you don't take into account the people who are going to use it you are dooming your efforts. This is true for systems that are internal to companies, or external to be sold to others. You can mandate usage internally (you WILL use the new system) but it will never be accepted and unless you have a monopoly you have no hope externally.

If you find yourself looking at a technology decision be sure to take into account the soft stuff. Who is going to use it? What are they going to do with it? What is the LEAST that you can do to make the system work? Systems do not exist for their own sake. They exist to serve people in some way. Keep that foremost in your mind and the decisions about what to do should be clearer.

Process

Process is kind of a loose word, where different people mean different things by it. When a process is broken, though, just stating that it is broken is useless. You have to describe WHAT about the process is broken, and why. Usually it is not very difficult to discover, but the piece that is broken is often there in order to make something easier for someone. It was not seen as a breakage, but just a short cut. Sometimes you can get away with one of these in a process, particularly when the process is long-running.

This is deceptive as people think "Well, that one little deviation did not hurt, one more won't hurt either." Soon the original deviation is one of many and the process breaks. Often there is a finger pointing exercise wherein everybody trots out their favorite theory of what went wrong, but the process remains broken. The deviations often have become the practice in the organization and people are invested in it.

Often an outsider is brought in to help "fix" the process.

Being an outsider, this person usually immediately sees where the problem is and tells everyone exactly what it is. Either the person is thanked and process is put back to a working state, or it is decided that the outsider does not "get it" and cannot understand the intricacies of how the organization works. Regardless of which result comes out of it, we can rest assured that the process will drift away from the ideal at some point. So, what can one do about it? Usually you have to have someone dedicated to a concept like "quality" where the process that is followed is periodically reviewed, and someone is held accountable for the worthiness of the process.

Many times this sort of function is seen as pure busywork of the worst kind, and a waste of effort. The time wasted every time the process breaks and profits or customers are lost more than make up for any investment in quality, but people will still make the short term decisions to save a few dollars.

In technology ventures, this whole "process" dance is even more vital as the processes are often automated and can occur thousands or millions of times before it becomes apparent that it is broken. Ironically, because of the fluid nature of software, process is given even shorter shrift than in real physical processes because you can often kludge something together to get to a short term goal. Inevitably you will pay for it later, with interest.

If you are involved in a technology project, regardless of vendor, platform, etc. be sure that you can describe your process for solving your problem in language that anybody could understand, and then be sure that you stick to it. I'm not saying never change the process, as they will evolve to meet changes, but if you cannot explain what you do, or how you accomplish your goals you are doomed to waste a lot of time and effort that could be used to help create more value from your technology investment.