Software Development Overview

All software, outside of the most trivial examples, handles data (some form of digital representation). With that data, software basically does four things:

1. Capture
2. Process
3. Store
4. Output

In each case what the data is and how it is handled differs, but really that is the crux of software development. Leave out the hardware on which the software runs. These four items are it. That said, each one of them can be drilled down into very deeply, with each layer yielding more approaches, ideas, philosophies, etc.

Methodologies on how to create each of these pieces and make them work together compose volumes. Whole companies exist to serve small parts of each of the approaches.

That said, it is still just as simple as these four things.

Data has be be captured from the environment outside of the computer program, brought in from a keyboard being typed on by a human, from a file on a disk being read, from a camera storing an image or one of many other approaches. There are a multitude of ways for capture to occur. There are fierce debates about the best way to do so.

Processing and storage can be done in either order, but I am putting processing before storage because normally something happens to the data before it is stored for future use. All the clever algorithms for sorting or determining values that take up so much time on the discussion boards fall into this bin. Again, debates are fierce about approaches to processing.

Storage can be done in various ways, being slightly less contentious. Physically the data winds up in the memory of the computer, either volatile (in RAM) or non-volatile (a storage medium like a disc or flash memory). The way the data is stored, the format of it, can yield some debate but not nearly as much as where it should go and the method for retrieving it.

Output is communicating the processed and/or stored data to another system, be it a human or another computer. For humans this can be something like a monitor, speakers or a printer. Something for one of our senses to experience. For another computer (or the same computer, running a different program) it can be bits transmitted over a network or stored onto a disc (somewhat overlapping with the storage aspect).

As I said earlier, each of these items can be expanded on to very great extents but it is instructive to be able to come back to them whenever you start to get lost in the chaos of information that surrounds all human enterprise and by extension the computer software that supports it. Most often each of the pieces can be separated from each other and you can make decisions based on what works best for you for each one instead of having to put all of your eggs into one basket for all four items.

Philosophy surrounding the development of software to do these four things run the gamut from just sitting down and starting to write code to huge processes that involve many people who think for a very long time about what they want to accomplish and how they want to do it and how they will measure their efforts, etc. There are fervent proponents of each approach who tout the benefits of their favorite and will stick by it dogmatically without ever thinking that there might be another way to do things.

The techgnostic approach is to look at the problem you are trying to solve and then line it up with the four items that software development requires. Very often you can crank something out quickly that gets the immediate need addressed but will come back to haunt you later. You can also spend a huge amount of time trying to get everything perfect the first time. Try to shoot for something between the two, subject to constraints like safety.

If you are building software to control a medical device or drop a bomb you need to think long and hard about it and the solution will tend towards the more restrictive process. If it is a program to dump some data periodically for someone so they can run a trending analysis it requires very little oversight.

If you look around yourself, at the software that you use (like the browser you are reading this on) you can quickly discern where the four pieces are that make it work. Did the entity that produced the software make all the pieces depend tightly on each other, or are they interchangeable? There are probably other options for what you are trying to do that approach the problem domain from a different perspective. Do not get locked into only one way.

I will return to this subject over time because a lot of philosophy about how to get things done is buried in it. Cowboy programming versus gold plating. Architecture versus expedience. The topics are legion. To complicate things further there are significant issues associated with the process that transcend these four items to do with the management of people and the economics of the entire process.

Digital music players

I overheard a conversation the other day that baffled me. The bafflement comes from my own egocentric reality where I assume everybody has a certain level of knowledge. As I no doubt have missed the point on this sort of thing before I thought I should write a little bit on it.

The confusion was two people talking about portable music players. One person was showing off their new player, a Microsoft Zune. The other person said "Ugh. You should have bought an iPod instead. They're much cooler", to which the reply was "I wanted something that works with Windows - I don't have a Mac."

Apparently the Apple marketing machine has not achieved full penetration.

This was followed on by "I think that you CAN use it with Windows, but you have to use iTunes for it to work", which elicited "I don't want to have to buy all my music through their store, and never be able to play the music anywhere else."

Again, maybe a couple of short commercials from Apple to update the masses would help.

I won't go into intense detail, but basically any digital music player will play MP3 files, the general current standard for digital audio. This includes the iPod. Apple prefers the AAC format which up until recently was encumbered with digital rights management (DRM) that did in fact lock the song to your account. Now, however, most of the music in the iTunes store is DRM-free. That said, you can get music from elsewhere into your iPod. The most important one which seemed to escape the people having the conversation is that you can rip the audio from your regular CD and put it onto your iPod.

Also, you can purchase or acquire MP3 files from any other source and put them into your iTunes library.

Disclosure: I myself have an iPod, which was about the fourth digital music player that I bought. I liked the other players just as much when they were playing music (I am not an audiophile and found the quality to be more or less equal) but where the iPod was clearly superior was the user interface and the client software. I use my iPod for also listening to podcasts (essentially recorded radio shows) and being able to easily organize all my audio content was worth it for me to move to an iPod. Your mileage may vary.

So be sure to be clear on what it is you want from a digital audio player, given your preferred operating system and method of acquiring music. You can then make a clear decision based on your own needs instead of being herded into one camp or another. Marketing seeks to push you into a decision based on irrational feelings. See past them and decide based on your actual needs.

Pareto Principle in Technology

I first heard of the Pareto Principle when I was at university studying economics. It was used in reference to wealth distribution (which was what Pareto was studying) where it had been observed that 20% of the people controlled 80% of the wealth. Over time more studies showed the same sort of distribution.

With respect to technology it is a handy rule of thumb that while not exact gives you something to go from when thinking about issues. A good recent example are analytical reports such as from AdMob of the applications on the Apple online software store (where Apple boasts of the tens of thousands of applications) wherein it was found that a small number of the applications account for most of the traffic and most of the profit. Various limitations (such as a set top 10 list) feed into some of this behavior but largely it boils down to most of the applications having very little value, leading to few people recommending or using them.

In software development the Pareto Principle also rears its head. In iterative approaches like Agile you work on a feature list from a priority list, with the highest value items first, in small batches. Rather than trying to deliver all 100 pieces of functionality that you would like to see you deliver them in batches of five or so and because the first ones being delivered have the highest value you might get 20% of them done (20 features) and have most of the benefits that you want, rather than taking a lot more time and maybe missing the mark with a lot of the work. Delivering sooner rather than later, even if it is a small amount of functionality, also plays into this approach.

Studies by large companies like Microsoft find results like fixing the top 20% of the bugs (by frequency of complaint) results in 80% percent of the errors and crashes going away.

Many examples can be found all around us of the Pareto Principle at work. As far as technology goes it is a good idea to keep it in the front of your mind when you are deciding how you should approach your projects. What is the smallest amount of features / product that you can deliver that provide the biggest bang? Grab this low hanging fruit and get something out rather than trying to design the ultimate application that does everything. You may find that after a few iterations that what you have is good enough and you can then move onto another project and find more success rather than continuing to try and reach a shifting and unattainable goal.

Disappearing Data

A software development blog that I read had an entry on alternative data storage. It referred to a tongue in cheek project called PaperBack. The idea is that your data is printed out to paper so that it will not degrade like magnetic or optical storage. There are serious people thinking on this subject. All in all I cannot tell if PaperBack is a joke or serious. Regardless, it highlights something that I often think about.

I look at all the data that I and my family have acquired over time and wonder about it. Documents, audio files, image files, videos and others. I have managed over time to collapse them down to a single large collection, which I have on multiple different hard drives. The collection represents all the floppy and zip disks I'd hauled around for 20+ years plus CD and DVD archives. Luckily external hard drives have plunged in price.

I should have started all this sooner than I did and count myself lucky that catastrophe did not strike. I have personally encountered tape that could not be used for anything or there was no software to read it. Hundreds of thousands of dollars worth of data inaccessible. Myself, I have CD's from the late 80's that are no longer readable. I also have a graveyard of dead hard drives.

It makes me think about how fleeting all this is. The fact that hard drives even work is something of a miracle with all the things that have to work JUST so between the hardware and software, at many levels. Every couple of years I have a fit of getting things together and saved. After I spend all the time doing it and making sure I've got it all I worry about the drives that I am backed up to failing. Or that a tornado will come in and annihilate it all regardless, making my efforts futile.

Then I get very philosophical and wonder what is so important with what I have. One time when I thought I had lost about six months worth of photos I felt physically ill and dreaded telling my wife. Luckily I found the files on another backup and all was saved. Regardless, if a tornado came along and annihilated the house and both backups we would be glad just to be alive.

I guess sometimes you have to balance off covering yourself against absolutely everything and what really matters.

Regardless, if you have not already done it, go out and buy an external hard drive and back up all your data. It is only a matter of time before your main hard drive fails. It is the cheapest insurance you will ever buy.