Software Development Overview

All software, outside of the most trivial examples, handles data (some form of digital representation). With that data, software basically does four things:

1. Capture
2. Process
3. Store
4. Output

In each case what the data is and how it is handled differs, but really that is the crux of software development. Leave out the hardware on which the software runs. These four items are it. That said, each one of them can be drilled down into very deeply, with each layer yielding more approaches, ideas, philosophies, etc.

Methodologies on how to create each of these pieces and make them work together compose volumes. Whole companies exist to serve small parts of each of the approaches.

That said, it is still just as simple as these four things.

Data has be be captured from the environment outside of the computer program, brought in from a keyboard being typed on by a human, from a file on a disk being read, from a camera storing an image or one of many other approaches. There are a multitude of ways for capture to occur. There are fierce debates about the best way to do so.

Processing and storage can be done in either order, but I am putting processing before storage because normally something happens to the data before it is stored for future use. All the clever algorithms for sorting or determining values that take up so much time on the discussion boards fall into this bin. Again, debates are fierce about approaches to processing.

Storage can be done in various ways, being slightly less contentious. Physically the data winds up in the memory of the computer, either volatile (in RAM) or non-volatile (a storage medium like a disc or flash memory). The way the data is stored, the format of it, can yield some debate but not nearly as much as where it should go and the method for retrieving it.

Output is communicating the processed and/or stored data to another system, be it a human or another computer. For humans this can be something like a monitor, speakers or a printer. Something for one of our senses to experience. For another computer (or the same computer, running a different program) it can be bits transmitted over a network or stored onto a disc (somewhat overlapping with the storage aspect).

As I said earlier, each of these items can be expanded on to very great extents but it is instructive to be able to come back to them whenever you start to get lost in the chaos of information that surrounds all human enterprise and by extension the computer software that supports it. Most often each of the pieces can be separated from each other and you can make decisions based on what works best for you for each one instead of having to put all of your eggs into one basket for all four items.

Philosophy surrounding the development of software to do these four things run the gamut from just sitting down and starting to write code to huge processes that involve many people who think for a very long time about what they want to accomplish and how they want to do it and how they will measure their efforts, etc. There are fervent proponents of each approach who tout the benefits of their favorite and will stick by it dogmatically without ever thinking that there might be another way to do things.

The techgnostic approach is to look at the problem you are trying to solve and then line it up with the four items that software development requires. Very often you can crank something out quickly that gets the immediate need addressed but will come back to haunt you later. You can also spend a huge amount of time trying to get everything perfect the first time. Try to shoot for something between the two, subject to constraints like safety.

If you are building software to control a medical device or drop a bomb you need to think long and hard about it and the solution will tend towards the more restrictive process. If it is a program to dump some data periodically for someone so they can run a trending analysis it requires very little oversight.

If you look around yourself, at the software that you use (like the browser you are reading this on) you can quickly discern where the four pieces are that make it work. Did the entity that produced the software make all the pieces depend tightly on each other, or are they interchangeable? There are probably other options for what you are trying to do that approach the problem domain from a different perspective. Do not get locked into only one way.

I will return to this subject over time because a lot of philosophy about how to get things done is buried in it. Cowboy programming versus gold plating. Architecture versus expedience. The topics are legion. To complicate things further there are significant issues associated with the process that transcend these four items to do with the management of people and the economics of the entire process.

Software Code Re-use

One of the holy grails for software development is to build something that will be used again by somebody else, saving the development time it would take to write it over again. Instead of re-inventing the wheel you get the efficiency of just grabbing a wheel off the shelf and using it. Inside of a small project with a small number of developers (perhaps even just one) this is usually easy to achieve. Once the project grows or there is a different project with similar needs it gets more complicated.

Software is still much of a craft than it is an engineering effort. Developers identify very strongly with their work and often believe that even though there is an existing block of code that basically does what they need that they can do it better. A combination of "not invented here" and a need for a slight change often results in yet another implementation of code to do something with text values, for example.

Sometimes this occurs directly in the face of existing code, out of developer hubris, but often it comes from ignorance of the platform that is being worked on. Chances are very good that a modern framework like Java, PHP or .NET already has defined in their frameworks code that does most of the things that you need. In a worst case you can often use something that is "almost there" and just add the couple of things you need without having to rewrite it all.

Regardless, software re-use is an ongoing are of interest for many organizations, both as producers and consumers.

So far, the best use of "re-use" of code has been with services rather than libraries (above the very basic libraries built into languages). The re-use has been best with things like web servers, database servers, etc. and not necessarily specific blocks of code that make up parts of the services. This is because a service creates a clear boundary. It defines for you up front what it will accept and what it will return or otherwise cause to happen. It is of no importance to the user how the work is accomplished, it is accomplished.

The UNIX ideal of "small and sharp" tools also cleaves to this ideal, whereby small command line utilities that do only one thing but do it very well can be chained together to create useful applications.

In a wider sense this is called a "stack", where different services and glued together to provide an overall environment / platform. For example, the LAMP stack (Linux, Apache, MySQL and PHP) is credited with a large amount of the innovation that occured on the internet because all four components were FOSS and were in use by large groups of developers.

Additionally, if you lived in any one of the components you did not have to worry about any of the others.

A web developer in PHP does not need to know how to program the C code that makes Linux work, nor the low level routines of Apache of MySQL. They can concentrate on building web applications using PHP and just depend upon the other items to work together. The same kind of re-use of code through services is found in many other cases also.

The techgnostic approach to re-use is to stop before you write a large pile of code and determine if what you need already exists. Writing and maintaining code is very expensive. If you can leverage something that already exists and has been used by other and has been tested to be good you can deliver a lot more value much more quickly than rolling your own.

Free software equivalents

One of the blogs that I follow, The Simple Dollar, recently had a post about software best-sellers and their "free" equivalents. I read through the list and found some programs that I use on a regular basis.

In some cases the word "free" can mean one of two things, or both. There is "free" as in speech and "free" as in beer.

The first free would be more descriptive if you used the French word Liberté (as in Liberté, égalité, fraternité - Freedom, Equality, Brotherhood) or liberty, meaning the freedom to do as you will with the software. This is generally viewed to mean see how it works and adapt it to your own purposes. Another term for this is "open source", meaning that the code that is used to build the software is "open" and there for all to see.

The second free is the more common sense of the word, as in "without charge". You can use the software and not have to pay anyone for the right to do so. This is the part that most people are interested in. Unless you are a computer programmer the ability to see the code that built the program is not important to you.

Interestingly you can have software that is free as in no charge, but you cannot see the code (i.e., proprietary free programs) such as Internet Explorer (although whether that is truly free in that it only runs on MS Windows is another debate). You can also have code that a company maintains intellectual property rights over and charges for, but you can also see the source code, should you be so inclined (not as common). The free software that most often gets the headlines is free in both senses, often referred to as FOSS or F/OSS software (Free Open Source Software).

As usual there are two sides to every story. There are some people who say, essentially, that free software is only free if your time is not worth anything. If you have free software that you have to build and support by yourself maybe paying a few dollars for a pieced of software is not all that bad an idea. That said a lot of popular FOSS software like Firefox is available already built and is very easy to use.

Like most things to do with technology and software in particular there is not a clear cut black and white answer to any question to do with free software.

Of the software packages listed I have used the following:

• OpenOffice - Early versions of this very extremely slow, but it has gotten much better. Their stated target was to be equivalent to MS Office 97 (not a bad target, in my opinion, as that was somewhat the zenith for that package before it shot off into bloatware)
  
• ZoneAlarm - Nice firewall program, but they push hard to move you to the paid version
  
• AVG Free - Good anti-virus software, but again they push you to use the paid version, going so far as to present "warnings" about needing to upgrade to the paid version that are not true.
  
• Audacity - Good audio program with lots of community support. For most people this will fill their audio needs.
  
• Google Maps - Cool mapping application
  
• Ubuntu Linux - This is my desktop OS at home, in addition to a laptop running Windows XP Pro, plus my wife's Mac. I spend most of my time in Ubuntu, switching over to the Windows machine for very specific things that I do not have under Ubuntu (mainly other MS products like Office or Visual Studio)
  

In addition to QuickBooks Simple Start Free Edition 2009 there is another FOSS program called GnuCash that does well for book-keeping.

The thing to keep in mind when you look for software is that generally there are lots of people who want to do what you're doing or trying to do. Lots of people have to do the same thing you're trying to do and some of them know how to program. Over time software gets written to satisfy these needs. Some of it is commercialized and some people make simple knock-offs of it. Depending upon you comfort with computers you can get by with all kinds of free software. If you're not as comfortable with software you can purchase software that has commercial support and is perhaps more popular, increasing the number of people who can help you with it.

The techgnostic thing to do is to get a trial of the commercial software as well as copies of a couple different free alternatives. Give them all a try and go with the one that best balances off the satisfaction of your needs with the demands it makes on your resources, be they time or money. In the end you will probably wind up with a mix like me, and save a lot of money by not having to buy programs that have a thousand features when you only need two every now and again.

Do you have any favorite FOSS programs? How about proprietary programs that there are no alternatives for that you depend on?

What do I do with my data?

At this point in time there are two broad categories of approaches you can use to manage your data:

• Relational
• Non-relational

Since one is defined in terms of the other I'll start with relational.

Like the name suggests, relational data are items that can be related to one another through the use of a key value. For example, your government issued an identifier number to you that is used to identify you. This number is then linked to your name, where you were, born, etc. It is a key to reaching that information.

If you think of a table of information, with rows running horizontally across the page and columns intersecting the rows vertically, running down the page, the key value identifies the row. So, for information about person you might have something like this:

Key	FirstName	LastName
100	John	Smith
101	Fred	Brown
...	...	...
174	Zachary	Miller

Each key identifies a row, and each row has columns of information. Then, in another table where you store, say, order information, you do not have to enter John Smith's personal information every time you create an order for him. The data exists in one place. Then, if you find out that he spells his name "Jon" and not "John" you can change it in once place.

There are other advantages to the model, but this is the crux of it. Information only has to be entered once, and then related to other information.

A relational database management system (RDBMS) is software that allows you manipulate this type of information easily.

Non-relational is a grab bag of different approaches that do not include necessarily the idea of a key value (or at least not in a separate tables sort of way). Generally a non-relational data store will store all of the information for any one thing all together, all at once. For example, an order would be stored all on one line, with the customer information repeated over and over for each order they are linked to. You wind up storing the information more than once for certain items and you have a lot of work to do if you want to update that information or use it to link to information that is held elsewhere.

Storing information in a spreadsheet often winds up being non-relational but in more modern systems there are ways to make the data relate to other data. The lack of management tools, however, will generally drive you towards a management system.

If the data you are storing is VERY simple and small in size you can save time and money by just storing it into a text file that you update by hand. For anything beyond that you should consider more sophisticated software to help you with the process of storing and updating the information.

Weighing the pros and cons of the various different packages is a very involved process. Think through what your needs will be and then invest the time in some research. Make sure you base your decision on your needs and not on what some marketing tells you your needs are.

Make a techgnostic choice for your database needs and you will achieve your goals effectively and efficiently.

Do I need a database?

"Oh, obviously you need a database."

What?

What is a database, you ask? All you know is that it lives on the computer somewhere and keeps track of things. Why can't you just keep your paper and files? Or are you already using a computer? "Look, see - I have it all in a spreadsheet!".

The person who told you of your need for a database rolls his/her eyes and groans and says "THAT is not a database!"

Confusion reigns.

So, what is it?

Strictly speaking, ANY collection of data is a database. Your paper and files? That's a database. A bunch of index cards in a box? Database. That spreadsheet of yours. Also a database. Any one of many different software products with the word "database" in their name? Yeah, database. Really, a database is an organized body of related information. Period. The exact mechanics of it are where people get hung up.

Honestly, the person who recommended a database to you most likely meant that you would benefit from a "database management system" (DBMS) such as has been around for many years, specifically designed to manage your information. Even though your stack of paper is technically a database how long does it take you to find a very specific piece of data out of it, say, an invoice from a particular vendor? When the pile is small it is trivial. As the pile grows it becomes non-trivial.

You've moved from a small pile / number of cards / number of spreadsheets. What happens when someone else needs to get at the same information? Any more than a couple of people cannot effectively search through your pile. And when the information must be updated what do you do? Mark up the original? Attach sticky notes? Put the file on a shared drive and hope for the best? And if you want to know some kind of aggregate information like "how much money do I owe all people with outstanding invoices?" you now have a much larger task on your hands. That is where a DBMS really comes into its own.

A DBMS has been designed, from the beginning, to efficiently store your information, with methods to capture, store, analyze and create outputs based on the information stored within. They are at the core of much of the efficiency that is attributed to computer technology.

For anybody who is doing anything in their lives that requires capturing information, storing it for future use, sifting and analyzing it and creating output such as reports or simply answers to questions about the data the need for a computerized database is clear.

However, just like the question of "do I need a computer" (most often "yes") is followed by "what kind of computer do I need?" there are some questions you have to ask yourself before you start down the path of using a DBMS. As the blog title implies we will cover this in a techgnostic fashion in the next post.

For a sneak peek, the questions will be along the lines of:

• What kind of data am I storing?
• How much data am I storing?
• How many people need to get at it?
• What other requirements do I have?

That will be followed by another post wherein some of the more popular alternatives are considered, as well as some less popular choices. Come to think of it, this might become a series of posts. Stay tuned.

The rise of the Netbook

Making computers smaller was a goal from the very earliest incarnations of the technology. Down from the size of a room to the size of a few fridges. Down from that to a largish box that could sit on the top of a desk. Soon someone asked "How about one I can take with me?" Portable (in the sense that they COULD be moved, not necessarily EASY to move) computers themselves iterated through various forms. Of course someone said "What if it ran on batteries?" and computers became truly portable, freed from the tether of the electrical grid (or a generator). The same evolution has continued over the years with portables becoming smaller and more powerful with each cycle.

Until very recently these smaller and more powerful machines commanded premium prices. You could expect to pay much more than you'd pay for a desktop computer with similar power. The intent was that the portable computer should have the same power (or close to it) as a desktop. This meant all kinds of trade-offs had to be made and components had to be engineered to be even smaller. All this engineering did not come cheap and having a laptop computer was a badge of honor and the province of the technological (or just wealthy) elite.

Eventually a kind of tipping point was found, not unlike with cell phones. In the same way that the huge cell phones of the 1980's become smaller and smaller into the 1990's so that design decisions started to rotate around making the keys LARGE enough for human fingers to operate as the phones had become so small laptops started to change also. Two kinds of consumers were looking for portable machines, with two sets of criteria: as SMALL as possible (most often frequent travelers and/or gadget freaks) versus as POWERFUL as possible (the so called "desktop replacement" crowd). Each of these consumer types was provided with what they wanted, but in all cases you continued to pay dearly for the machines.

As time and Moore's Law marched on an intersection occurred. This was the intersection between relatively powerful laptops and the ubiquity of wireless internet. Again something that was once the province of the technical elite became very common, with wireless access available virtually anywhere for a very small fee, if at all. Instead of existing as an alternative to a desktop computer the laptop became basically a terminal to connect into the content of the internet. The need for the terminal hardware (i.e., the laptop) to be powerful was lessened. Beyond a certain amount of power was a waste. A simple machine could be used to read e-mail, surf the web, watch videos, etc. The hardware of a new type of laptop was ready. The other part necessary was the software.

Open source software, which languished for some years in obscurity from the public, leapt to the forefront of these new machines, dubbed "netbooks" as they are basically used to access networks and not much else. Rather than having to pay the "Microsoft tax" for every machine netbook manufacturers started to load open source operating systems like Linux onto their machines, erasing the extra payment. This dropped the price a little bit more, along with the smaller screen, small hard drive and lack of optical drive to produce a netbook for a few hundred dollars instead of almost a thousand dollars.

The effect on the market was slow to build but once it did it ran away, catching the large manufacturers unawares. Coinciding with some economic hard times netbooks flew off the shelves and accounted for most of the growth in the sales of portable computers. Now all the major manufacturers have their own netbooks out and Microsoft is scrambling to try and find a way to ensure that they are a presence in the market, albeit at a lower per unit cost as in the past.

For many people a netbook is all the computer that they need. They can use a web based provider of e-mail, store their pictures on a photo sharing site, use online applications for word processing and generally do what they need to do without very much regard to the computer that they do it on. If their inexpensive netbook bites the dust the only thing you might need to do is rediscover your bookmarks (although even those can be cached away by one of many online utilities).

There will always be a market for the more powerful and expensive laptops for people who need the power of local processing for tasks like video processing or engineering that burns up a lot of CPU cycles, in the same way that there will still be people who will spend thousands of dollars on desktop machines to do very specific tasks. In both cases, though, there are now inexpensive alternatives, often using open source software, that give people just what they need for a small amount of money.

Keeping in mind what you're trying to get done before you choose your technology is a techgnostic thing to do. Think it through the next time you advise somebody on what kind of computer to buy or examine your own needs the next time you put your money down.

Once more unto the breach

I guess a year is an unreasonable gap. Lots has gone on in my life. Many events that kept me from thinking about an online journal. Most significantly was the passing of my father, who influenced me very strongly. Few days pass by that I do not think of him and wish I could talk to him.

2008 was a year of transitions for me on many levels both personally and professionally. Now I think it's time to re-engage a few things I'd shelved during the chaos. One of those things is this blog. I will commit myself to a personal goal of posting to this blog once per week, come Hell or high water. I've already queued up a few weeks worth of topics (not yet written).

Going back and reading over the older posts I realize that my interest are more or less the same as they were, and I will be continuing in the same vein. That means more posts about the various manifestations of information and computer technology, their use in business process management and trying to make all the parts of the process dance together: people, technology and business.

And not only that, but also from the point of view that what's important at the end of the day are the results and not sticking to any one religion (e.g., Microsoft vs. Mac vs. Linux). Sometimes a post will be just because it's interesting.

Once more unto the breach, dear friends, once more.

Process

Process is kind of a loose word, where different people mean different things by it. When a process is broken, though, just stating that it is broken is useless. You have to describe WHAT about the process is broken, and why. Usually it is not very difficult to discover, but the piece that is broken is often there in order to make something easier for someone. It was not seen as a breakage, but just a short cut. Sometimes you can get away with one of these in a process, particularly when the process is long-running.

This is deceptive as people think "Well, that one little deviation did not hurt, one more won't hurt either." Soon the original deviation is one of many and the process breaks. Often there is a finger pointing exercise wherein everybody trots out their favorite theory of what went wrong, but the process remains broken. The deviations often have become the practice in the organization and people are invested in it.

Often an outsider is brought in to help "fix" the process.

Being an outsider, this person usually immediately sees where the problem is and tells everyone exactly what it is. Either the person is thanked and process is put back to a working state, or it is decided that the outsider does not "get it" and cannot understand the intricacies of how the organization works. Regardless of which result comes out of it, we can rest assured that the process will drift away from the ideal at some point. So, what can one do about it? Usually you have to have someone dedicated to a concept like "quality" where the process that is followed is periodically reviewed, and someone is held accountable for the worthiness of the process.

Many times this sort of function is seen as pure busywork of the worst kind, and a waste of effort. The time wasted every time the process breaks and profits or customers are lost more than make up for any investment in quality, but people will still make the short term decisions to save a few dollars.

In technology ventures, this whole "process" dance is even more vital as the processes are often automated and can occur thousands or millions of times before it becomes apparent that it is broken. Ironically, because of the fluid nature of software, process is given even shorter shrift than in real physical processes because you can often kludge something together to get to a short term goal. Inevitably you will pay for it later, with interest.

If you are involved in a technology project, regardless of vendor, platform, etc. be sure that you can describe your process for solving your problem in language that anybody could understand, and then be sure that you stick to it. I'm not saying never change the process, as they will evolve to meet changes, but if you cannot explain what you do, or how you accomplish your goals you are doomed to waste a lot of time and effort that could be used to help create more value from your technology investment.

Good Enough Hardware

Used to be, a "cheap" desktop computer hovered around a thousand dollars in price, without a monitor. Those days are now far behind us. Cheap desktop systems with bundled LCD monitors are easily found for less than five hundred dollars. Laptops have similarly crashed in price, with a race for the bottom still underway. There is still a high end to the market, with multi-thousand dollar machines, but only people with very specific requirements should buy them.

If you only drive a few nails a year to hang pictures, you can buy a five dollar hammer and you're set. If you take that five dollar hammer onto a job site and drive a few thousand nails with it you will quickly destroy it, rounding the head of it, shredding the handle, etc. In the same way if you are a hard core video game player you would not buy a five hundred dollar computer and expect to be happy.

All that said, most people can get by with the modern "low end" computer, that has more processing power, memory, disk space, etc. than machines they purchased a few years ago. And that computer will be sufficient for a fairly long time. In the 90's software could get ahead of your computer very quickly, and your hardware quickly became obsolete. With so much time between major updates of software now, plus the power available at the low end of the market, breaks this cycle.

The one place that this is somewhat not true is with the Microsoft Vista operating system, if you want to run it with all the bells and whistles. If you want that then the low end machines might not be exactly what you want. If you are only wanting to surf the web and do e-mail you should consider alternative operating systems that run on the same hardware, such as a Linux distribution like Ubuntu.

The Macintosh computers are more expensive, but come loaded with a lot more bells and whistles that narrow the gap between the low end PC and the low end Mac. They offer a good level of stability and if you are looking for easy they are hard to beat.

When you go to purchase your next computer, consider all the factors, including what you are actually SPECIFICALLY going to use the machine for, and do not fall for the sales pitch of the latest, greatest and fastest. A "good enough" computer is going to be sufficient for 95% of people, so don't turn up your nose at a computer because of its low price. You'll be pleasantly surprised by your new machine, regardless.

Make your choice techgnostically, and don't be swayed by hype (I'm looking at YOU, Microsoft).

So where does "techgnostic" come into it?

From my title block:

• gno·sis (no-sis) n. Intuitive apprehension of truths, an esoteric form of knowledge.


• ag·nos·tic (ag-nos-tik) n. One who is doubtful or noncommittal about something. A skeptic.


• tech-gno·stic (tek-nos-tik) n. Intuitive apprehension of technology truths, a skeptical take on technology knowledge.

Why append my name with such a title?

I have worked over the years on many flavors of Unix (Sun, HP, SG, BSD and Linux), different incarnations of the Mac OS (from when the whole OS ran off a single floppy in my Mac 512e) and DOS/Windows since the XT. There are zealots in every camp, and strong reasons for their biases. Untold billions of characters have been typed into computers during flame wars about the "best" OS, programming language, editor, database, you-name-it. There is only one thing that can be said for all of this.

All computers suck, equally. I'm not the only person who thinks so.

A bit glib, I admit, but nice and pithy and with enough truth in it to allow it to stand up. Each "platform" of hardware / software / applications was created to solve a particular kind of problem and usually accomplished its mission. When an attempt is made to repurpose it to another task it often fails. From a certain perspective, any other platform equates to "sucks".

So, beyond appearing to be a technological dilettante, so what?

My own technological belief is to use whatever tools, platform, etc. make sense for your own situation. Again this seems like just giving in, but it is not. There are still fundamentals to keep in mind in terms of getting support, safeguarding your data, etc. but they can be enacted on virtually any platform. For all the flak that Microsoft Windows takes it can be locked down quite well. For all the raves that the Mac gets for style there is a closely controlled platform that stifles some kinds of development. For all the intellectual purity of most Linux versions there is still often a steep curve for the computing novice. For all the vaunted stability, etc. of the mainframe there is a prohibitive cost and lack of common support in the workforce.

If you or your company already employ a bunch of Windows admins and everybody is used to Windows there is not much point in having a Mac or Linux jihad because of some perceived superiority. It just does not make sense in that situation. Similarly, if you have a Unix setup and all is well, why upset the system to introduce a single Microsoft application when there are ways to integrate it into your existing setup?

All that said, strong arguments can be made for using standardized PCs, running open source software in "clean sheet" situations where low cost is an overriding concern, such as with the One Laptop Per Child project, or indeed for any other organization who only needs certain kinds of generic functionality, not tied to a particular software package.

The techgnostic part of all this is that in an ideal situation where the platform has not been dictated there are many options to solve your technology problems and we should not shut ourselves off into any one camp. The personalities and business practices of the players should not play a major role in our decisions - what works for us in our situations should be front and center.

That is being techgnostic.