Software Development Insanity

Speaking of software development, let me detour into a grumpy old developer diatribe.

First, go read this.

Done? OK. For me, the choice quote:

I may be getting old, but I don’t think I am out of touch. My sanity of being a software development professional seems to be tested daily by our industries predilection for Silver Bullets. The latest it seems is Scrumban. My wife can’t stop laughing when I say it to her.

Building software is an expensive business and there are always people trying to sell the latest and greatest improvements. The purported improvement is sometimes a product (a new IDE or language) and sometimes a process (XP, Agile, etc.). There is always a certain segment of the developer community that scrambles after anything that is new and shiny and with the advent of the web the frenzy of fandom can create more buzz than is deserved.

In an attempt to be "with it" the new improvement is tossed around as proof of how up-to-date one is. Previous approaches are dismissed and people who use them are dinosaurs.

Regardless, as the blog author points out (like I sort of did in my last post) software development at its root is fairly basic. In this case "Requirements, Design, Code and Test."

A book that influenced me strongly was The Pragmatic Programmer and its approach of "tracer bullet" development. The idea of roughing out a whole system with stubs where real code would eventually go was appealing. You could quickly get a rough idea of how the whole thing would hang together and could immediately see what pieces would be more effort than others. Not surprisingly the authors of that book became influential in the whole Agile movement.

The marketing and buzz that sprang up around Agile now means that the term is tossed around without much care as to what it means in practice, as long as you sign up for the seminar or pay to be certified as an Agile coach. All that management hopes for is that they can start to get more software faster than they did before. Unfortunately for them all of this glosses over many other items of importance like the huge differences between the abilities of individual developers.

I think that there is room for discussion on craft vs. engineering in software development. People who write code for a living, day in and day out, are close to the workings of the technology and have an intuitive grasp of what they are doing. Attempting to impose engineering rigor on the process gives the illusion of control with boxes and lines and probably some kind of metrics. At the end of the day the develop has to know what is supposed to be built and have a clear idea how they are going to do it. The rubber hits the road when the developer hits the keyboard.

I am not sure if there will be some kind of a backlash an a "back to basics" push anytime soon but I for one start to tire of the relentless onslaught of new approaches that do not offer significant benefit over known and proven approaches. There is an awful lot of code out there that would be considered obsolete by the current cutting-edge developers that nevertheless is important and valuable to businesses.

Being techgnostic means keeping all of this in mind before embarking down the path of new software development. What you already have might basically do what you need (Pareto principle) or what you need might be a one-off fix that does not warrant a full blown process. Step back, take a deep breath and ask yourself what you are trying to accomplish and produce just enough to make it work, getting something out sooner rather than later.

Software Development Overview

All software, outside of the most trivial examples, handles data (some form of digital representation). With that data, software basically does four things:

1. Capture
2. Process
3. Store
4. Output

In each case what the data is and how it is handled differs, but really that is the crux of software development. Leave out the hardware on which the software runs. These four items are it. That said, each one of them can be drilled down into very deeply, with each layer yielding more approaches, ideas, philosophies, etc.

Methodologies on how to create each of these pieces and make them work together compose volumes. Whole companies exist to serve small parts of each of the approaches.

That said, it is still just as simple as these four things.

Data has be be captured from the environment outside of the computer program, brought in from a keyboard being typed on by a human, from a file on a disk being read, from a camera storing an image or one of many other approaches. There are a multitude of ways for capture to occur. There are fierce debates about the best way to do so.

Processing and storage can be done in either order, but I am putting processing before storage because normally something happens to the data before it is stored for future use. All the clever algorithms for sorting or determining values that take up so much time on the discussion boards fall into this bin. Again, debates are fierce about approaches to processing.

Storage can be done in various ways, being slightly less contentious. Physically the data winds up in the memory of the computer, either volatile (in RAM) or non-volatile (a storage medium like a disc or flash memory). The way the data is stored, the format of it, can yield some debate but not nearly as much as where it should go and the method for retrieving it.

Output is communicating the processed and/or stored data to another system, be it a human or another computer. For humans this can be something like a monitor, speakers or a printer. Something for one of our senses to experience. For another computer (or the same computer, running a different program) it can be bits transmitted over a network or stored onto a disc (somewhat overlapping with the storage aspect).

As I said earlier, each of these items can be expanded on to very great extents but it is instructive to be able to come back to them whenever you start to get lost in the chaos of information that surrounds all human enterprise and by extension the computer software that supports it. Most often each of the pieces can be separated from each other and you can make decisions based on what works best for you for each one instead of having to put all of your eggs into one basket for all four items.

Philosophy surrounding the development of software to do these four things run the gamut from just sitting down and starting to write code to huge processes that involve many people who think for a very long time about what they want to accomplish and how they want to do it and how they will measure their efforts, etc. There are fervent proponents of each approach who tout the benefits of their favorite and will stick by it dogmatically without ever thinking that there might be another way to do things.

The techgnostic approach is to look at the problem you are trying to solve and then line it up with the four items that software development requires. Very often you can crank something out quickly that gets the immediate need addressed but will come back to haunt you later. You can also spend a huge amount of time trying to get everything perfect the first time. Try to shoot for something between the two, subject to constraints like safety.

If you are building software to control a medical device or drop a bomb you need to think long and hard about it and the solution will tend towards the more restrictive process. If it is a program to dump some data periodically for someone so they can run a trending analysis it requires very little oversight.

If you look around yourself, at the software that you use (like the browser you are reading this on) you can quickly discern where the four pieces are that make it work. Did the entity that produced the software make all the pieces depend tightly on each other, or are they interchangeable? There are probably other options for what you are trying to do that approach the problem domain from a different perspective. Do not get locked into only one way.

I will return to this subject over time because a lot of philosophy about how to get things done is buried in it. Cowboy programming versus gold plating. Architecture versus expedience. The topics are legion. To complicate things further there are significant issues associated with the process that transcend these four items to do with the management of people and the economics of the entire process.

Digital music players

I overheard a conversation the other day that baffled me. The bafflement comes from my own egocentric reality where I assume everybody has a certain level of knowledge. As I no doubt have missed the point on this sort of thing before I thought I should write a little bit on it.

The confusion was two people talking about portable music players. One person was showing off their new player, a Microsoft Zune. The other person said "Ugh. You should have bought an iPod instead. They're much cooler", to which the reply was "I wanted something that works with Windows - I don't have a Mac."

Apparently the Apple marketing machine has not achieved full penetration.

This was followed on by "I think that you CAN use it with Windows, but you have to use iTunes for it to work", which elicited "I don't want to have to buy all my music through their store, and never be able to play the music anywhere else."

Again, maybe a couple of short commercials from Apple to update the masses would help.

I won't go into intense detail, but basically any digital music player will play MP3 files, the general current standard for digital audio. This includes the iPod. Apple prefers the AAC format which up until recently was encumbered with digital rights management (DRM) that did in fact lock the song to your account. Now, however, most of the music in the iTunes store is DRM-free. That said, you can get music from elsewhere into your iPod. The most important one which seemed to escape the people having the conversation is that you can rip the audio from your regular CD and put it onto your iPod.

Also, you can purchase or acquire MP3 files from any other source and put them into your iTunes library.

Disclosure: I myself have an iPod, which was about the fourth digital music player that I bought. I liked the other players just as much when they were playing music (I am not an audiophile and found the quality to be more or less equal) but where the iPod was clearly superior was the user interface and the client software. I use my iPod for also listening to podcasts (essentially recorded radio shows) and being able to easily organize all my audio content was worth it for me to move to an iPod. Your mileage may vary.

So be sure to be clear on what it is you want from a digital audio player, given your preferred operating system and method of acquiring music. You can then make a clear decision based on your own needs instead of being herded into one camp or another. Marketing seeks to push you into a decision based on irrational feelings. See past them and decide based on your actual needs.

Pareto Principle in Technology

I first heard of the Pareto Principle when I was at university studying economics. It was used in reference to wealth distribution (which was what Pareto was studying) where it had been observed that 20% of the people controlled 80% of the wealth. Over time more studies showed the same sort of distribution.

With respect to technology it is a handy rule of thumb that while not exact gives you something to go from when thinking about issues. A good recent example are analytical reports such as from AdMob of the applications on the Apple online software store (where Apple boasts of the tens of thousands of applications) wherein it was found that a small number of the applications account for most of the traffic and most of the profit. Various limitations (such as a set top 10 list) feed into some of this behavior but largely it boils down to most of the applications having very little value, leading to few people recommending or using them.

In software development the Pareto Principle also rears its head. In iterative approaches like Agile you work on a feature list from a priority list, with the highest value items first, in small batches. Rather than trying to deliver all 100 pieces of functionality that you would like to see you deliver them in batches of five or so and because the first ones being delivered have the highest value you might get 20% of them done (20 features) and have most of the benefits that you want, rather than taking a lot more time and maybe missing the mark with a lot of the work. Delivering sooner rather than later, even if it is a small amount of functionality, also plays into this approach.

Studies by large companies like Microsoft find results like fixing the top 20% of the bugs (by frequency of complaint) results in 80% percent of the errors and crashes going away.

Many examples can be found all around us of the Pareto Principle at work. As far as technology goes it is a good idea to keep it in the front of your mind when you are deciding how you should approach your projects. What is the smallest amount of features / product that you can deliver that provide the biggest bang? Grab this low hanging fruit and get something out rather than trying to design the ultimate application that does everything. You may find that after a few iterations that what you have is good enough and you can then move onto another project and find more success rather than continuing to try and reach a shifting and unattainable goal.

Disappearing Data

A software development blog that I read had an entry on alternative data storage. It referred to a tongue in cheek project called PaperBack. The idea is that your data is printed out to paper so that it will not degrade like magnetic or optical storage. There are serious people thinking on this subject. All in all I cannot tell if PaperBack is a joke or serious. Regardless, it highlights something that I often think about.

I look at all the data that I and my family have acquired over time and wonder about it. Documents, audio files, image files, videos and others. I have managed over time to collapse them down to a single large collection, which I have on multiple different hard drives. The collection represents all the floppy and zip disks I'd hauled around for 20+ years plus CD and DVD archives. Luckily external hard drives have plunged in price.

I should have started all this sooner than I did and count myself lucky that catastrophe did not strike. I have personally encountered tape that could not be used for anything or there was no software to read it. Hundreds of thousands of dollars worth of data inaccessible. Myself, I have CD's from the late 80's that are no longer readable. I also have a graveyard of dead hard drives.

It makes me think about how fleeting all this is. The fact that hard drives even work is something of a miracle with all the things that have to work JUST so between the hardware and software, at many levels. Every couple of years I have a fit of getting things together and saved. After I spend all the time doing it and making sure I've got it all I worry about the drives that I am backed up to failing. Or that a tornado will come in and annihilate it all regardless, making my efforts futile.

Then I get very philosophical and wonder what is so important with what I have. One time when I thought I had lost about six months worth of photos I felt physically ill and dreaded telling my wife. Luckily I found the files on another backup and all was saved. Regardless, if a tornado came along and annihilated the house and both backups we would be glad just to be alive.

I guess sometimes you have to balance off covering yourself against absolutely everything and what really matters.

Regardless, if you have not already done it, go out and buy an external hard drive and back up all your data. It is only a matter of time before your main hard drive fails. It is the cheapest insurance you will ever buy.

Google Voice

I lucked out and snagged one of the invitations for Google Voice. Everything you've read about it being cool is true. It is simple to enter however many phone numbers you want it to ring through to and the voice mail features are awesome. They are continuing to expand it and will be adding more features and integration going forward. For myself it is also ideal because it allows me to make long distance calls to Canada for a small fraction of what I normally pay, and I can have it ring through to whatever phone I am currently at. I'm normally not overly gushy about things but this one warrants it.

The service runs in the cloud and you access it with a web browser, wherever you are. There is some integration with GMail right now but it is limited. There is also a client side application for smartphones available that makes your use of the service transparent. This means that you can get a phone number that never changes (the Google Voice one) and hook it into whatever phone you want on the other end so you are no longer bound to a carrier to keep your number.

Apple's decision to block the application that allows tighter integration with the service is short-sighted and is causing bad press for them. Once Google closes the gap on a few more things like sending and receiving faxes it is going to be a de facto requirement for any small enterprise.

If you haven't yet got an account to to http://www.google.com/voice and request one. You will not be sorry.

The only caveat on this is going to be "how will they make money?" Right now it is free (except for international long distance, which is still very reasonable) but I predict it will move in the same direction as the Google Apps suite, with a limited free edition and a full feature paid version. Stay tuned as they continue to evolve it and competitors try to come up with something better (are you on it yet Microhoo?)

Software Code Re-use

One of the holy grails for software development is to build something that will be used again by somebody else, saving the development time it would take to write it over again. Instead of re-inventing the wheel you get the efficiency of just grabbing a wheel off the shelf and using it. Inside of a small project with a small number of developers (perhaps even just one) this is usually easy to achieve. Once the project grows or there is a different project with similar needs it gets more complicated.

Software is still much of a craft than it is an engineering effort. Developers identify very strongly with their work and often believe that even though there is an existing block of code that basically does what they need that they can do it better. A combination of "not invented here" and a need for a slight change often results in yet another implementation of code to do something with text values, for example.

Sometimes this occurs directly in the face of existing code, out of developer hubris, but often it comes from ignorance of the platform that is being worked on. Chances are very good that a modern framework like Java, PHP or .NET already has defined in their frameworks code that does most of the things that you need. In a worst case you can often use something that is "almost there" and just add the couple of things you need without having to rewrite it all.

Regardless, software re-use is an ongoing are of interest for many organizations, both as producers and consumers.

So far, the best use of "re-use" of code has been with services rather than libraries (above the very basic libraries built into languages). The re-use has been best with things like web servers, database servers, etc. and not necessarily specific blocks of code that make up parts of the services. This is because a service creates a clear boundary. It defines for you up front what it will accept and what it will return or otherwise cause to happen. It is of no importance to the user how the work is accomplished, it is accomplished.

The UNIX ideal of "small and sharp" tools also cleaves to this ideal, whereby small command line utilities that do only one thing but do it very well can be chained together to create useful applications.

In a wider sense this is called a "stack", where different services and glued together to provide an overall environment / platform. For example, the LAMP stack (Linux, Apache, MySQL and PHP) is credited with a large amount of the innovation that occured on the internet because all four components were FOSS and were in use by large groups of developers.

Additionally, if you lived in any one of the components you did not have to worry about any of the others.

A web developer in PHP does not need to know how to program the C code that makes Linux work, nor the low level routines of Apache of MySQL. They can concentrate on building web applications using PHP and just depend upon the other items to work together. The same kind of re-use of code through services is found in many other cases also.

The techgnostic approach to re-use is to stop before you write a large pile of code and determine if what you need already exists. Writing and maintaining code is very expensive. If you can leverage something that already exists and has been used by other and has been tested to be good you can deliver a lot more value much more quickly than rolling your own.

(Not) Getting Things Done

One of the core values of Getting Things Done (GTD) is to capture everything. You use "buckets" such as a pad of paper, a voice recorder, etc. to capture any and all things you need to get done. Ideally you should get by with as FEW buckets as you can, and you have to empty them periodically, ideally daily. Then all the items should be processed, put into appropriate slots for further action.

Some people start into GTD with guns blazing and eventually bail out, abandoning the system entirely as too much trouble and too rigid. Some others (like myself) continue to capture everything, but don't make the time to do the processing and wind up with information scattered hither and yon.

Myself, I have a digital voice recorder, a Palm PDA, my GMail account using a GTD Firefox extension, the GMail task list widget, a collection of Google documents and a couple of scattered OneNote files. I also recently loaded up EverNote to give it a swirl but have thus far put very little into it. Needless to say it is overwhelming when I look at all of them and try to wrap my head around it.

Like an addict of some kind, though, I continue to go through the motions even though I have no idea why I'm doing them. I guess I think I'll suddenly get a bunch of time to plough through and get my collective stuff in order.

I don't normally consider myself an overly hopeful individual but I must be. The MOST important items continue to bubble to the top but what I really need to do is clear the decks and do another reset. Hand-in-hand with that is a need to ruthlessly purge a lot of things that have been hanging around my lists for years now that I am just going to have to accept are not going to happen in my lifetime.

The geeky aspect of all this is that I'm saving up and waiting to get myself an iPod Touch to try and replace a bunch of these items. There are some people who follow GTD who have eschewed the technological approach and go with Hipster PDAs (collections of 3x5 cards with a binder clip).

I wonder what the techgnostic approach to all this should be? Generally it would mean an application of just enough technology to be effective without gold-plating for the purposes of geeky fulfillment or to fit into whatever current rhetoric my technological church of choice dictates. Maybe my technology needs are simpler than I think they are, and my fevered thoughts of creating or finding an XML format to allow for a vendor and platform neutral means to store all my GTD information are insanity?

Hmmm. Now that I start looking around apparently EverNote supports XML import / export. Here's someone else who's obviously put a lot of thought into this also. Plus there are many software packages for this.

The techgnostic thing to do, I think, would be to pick something that is just enough to help me integrate what I have and make it less painful to keep up. Another project to add my my list.

Doh!

Smart Phones

Ah - a couple weeks of downtime always refreshes.

Lots happened in the last couple of weeks(ish). One was the new iPhone came out. A friend of mine is agonizing over getting one, versus a Blackberry or something else. He believes that government and corporations are all sewn up for Blackberry.

For government - agreed (for now).

Corporate - it's shifting, and shifting rapidly.

With the iPhone OS 3.0 release there is now tighter integration with Exchange servers (really the thing that made corporate folks ga-ga for Blackberry in the first place) plus security items like the ability to remotely wipe a handset.

In many companies that I directly know of the upgrade cycle (starting at the top) is resulting in iPhones instead of Blackberries (execs first, of course). Governments will take longer to make such a change, if they do, and again it will start at the top, with the peons getting the goodies last (naturally).

Me, personally, I do not talk on the phone very much, and if I do it is from a land line. I use a pre-pay phone for the occasional voice call and use a PDA for other functions. When my current PDA (a Palm) dies I'll get an iPod touch (seeing as Palm got out of the PDA business).

If you're going to pay the monthly bill anyways go with the iPhone. Blackberry will probably be what you're provided with by your company.

That said, the market keeps shifting, with the Palm Pre now released and decent UI having been grafted onto Android by HTC.

As always, make a decision based on your own needs and not what the market blares at you.

Free software equivalents

One of the blogs that I follow, The Simple Dollar, recently had a post about software best-sellers and their "free" equivalents. I read through the list and found some programs that I use on a regular basis.

In some cases the word "free" can mean one of two things, or both. There is "free" as in speech and "free" as in beer.

The first free would be more descriptive if you used the French word Liberté (as in Liberté, égalité, fraternité - Freedom, Equality, Brotherhood) or liberty, meaning the freedom to do as you will with the software. This is generally viewed to mean see how it works and adapt it to your own purposes. Another term for this is "open source", meaning that the code that is used to build the software is "open" and there for all to see.

The second free is the more common sense of the word, as in "without charge". You can use the software and not have to pay anyone for the right to do so. This is the part that most people are interested in. Unless you are a computer programmer the ability to see the code that built the program is not important to you.

Interestingly you can have software that is free as in no charge, but you cannot see the code (i.e., proprietary free programs) such as Internet Explorer (although whether that is truly free in that it only runs on MS Windows is another debate). You can also have code that a company maintains intellectual property rights over and charges for, but you can also see the source code, should you be so inclined (not as common). The free software that most often gets the headlines is free in both senses, often referred to as FOSS or F/OSS software (Free Open Source Software).

As usual there are two sides to every story. There are some people who say, essentially, that free software is only free if your time is not worth anything. If you have free software that you have to build and support by yourself maybe paying a few dollars for a pieced of software is not all that bad an idea. That said a lot of popular FOSS software like Firefox is available already built and is very easy to use.

Like most things to do with technology and software in particular there is not a clear cut black and white answer to any question to do with free software.

Of the software packages listed I have used the following:

• OpenOffice - Early versions of this very extremely slow, but it has gotten much better. Their stated target was to be equivalent to MS Office 97 (not a bad target, in my opinion, as that was somewhat the zenith for that package before it shot off into bloatware)
  
• ZoneAlarm - Nice firewall program, but they push hard to move you to the paid version
  
• AVG Free - Good anti-virus software, but again they push you to use the paid version, going so far as to present "warnings" about needing to upgrade to the paid version that are not true.
  
• Audacity - Good audio program with lots of community support. For most people this will fill their audio needs.
  
• Google Maps - Cool mapping application
  
• Ubuntu Linux - This is my desktop OS at home, in addition to a laptop running Windows XP Pro, plus my wife's Mac. I spend most of my time in Ubuntu, switching over to the Windows machine for very specific things that I do not have under Ubuntu (mainly other MS products like Office or Visual Studio)
  

In addition to QuickBooks Simple Start Free Edition 2009 there is another FOSS program called GnuCash that does well for book-keeping.

The thing to keep in mind when you look for software is that generally there are lots of people who want to do what you're doing or trying to do. Lots of people have to do the same thing you're trying to do and some of them know how to program. Over time software gets written to satisfy these needs. Some of it is commercialized and some people make simple knock-offs of it. Depending upon you comfort with computers you can get by with all kinds of free software. If you're not as comfortable with software you can purchase software that has commercial support and is perhaps more popular, increasing the number of people who can help you with it.

The techgnostic thing to do is to get a trial of the commercial software as well as copies of a couple different free alternatives. Give them all a try and go with the one that best balances off the satisfaction of your needs with the demands it makes on your resources, be they time or money. In the end you will probably wind up with a mix like me, and save a lot of money by not having to buy programs that have a thousand features when you only need two every now and again.

Do you have any favorite FOSS programs? How about proprietary programs that there are no alternatives for that you depend on?

PDA Migration

Long ago my life started to get complicated enough that I started having to keep track of things on paper because my brain wasn't going to hold enough information anymore. I started with a simple paper organizer and then moved on to more involved ones as time went on like the Franklin.

In the early 1990's I purchased a Sharp Zaurus. Being a computer-guy it seemed appropriate for me to keep my data in an electronic format. I had to pay extra to get a serial cable interface so I could back up my data. It ran on AAA batteries and was super cool (to me, at least).

My the mid-late 90's, though still working, the Zaurus was getting passe so I moved onto a Palm Pilot. Again this was super cool. The device was designed for synchronization so it was very handy to move data to and fro, plus there was a programming kit so I could dabble with software for it.

This was followed by another Palm, then by a Sony Palm incarnation called the Clie (very nice). For a brief time I had a Palm VII, one of the first wireless PDAs. The data plan was way too expensive (data was charged by BYTE transferred). The latest (and probably last) Palm I got was the Zire, the low cost color version of the venerable Palm PDA. This mainly was my calendar, contact list and password storage device.

For a while, I had a laptop and largely used it for my information management needs. Then it died and I could not justify buying another one in light of my other responsibilities.

My work in the past has provided me with Blackberry or Windows Mobile devices. In all cases I have used them but never trust much to them, largely because they were provided by my employer and I keep my personal and professional items separate.

Now, my Zire is starting to fail. The digitizer is hit or miss and the screen is starting to go. Once a week or so I have to do a hard reset to make it go. Palm now no longer makes PDAs so I find myself at a crossroads. I want a small electronic device for handling my personal information, but the market only seems to be producing phones to do this, and I don't want a monthly bill.

I believe that now I want an iPod Touch. I had three other MP3 players before I got my iPod Nano a couple of years ago and I have to say hands down that the iPod is the superior experience. As a computer guy I was able to figure out my other devices but the iPod was just that much better. The iPod Touch is much larger and seems better suited to handling my needs for information management. There are a couple of sticking points.

One is price. At over two hundred dollars (as of this writing) it is not something that I can trivially purchase. It starts to get me into the neighborhood of buying a netbook instead. Admittedly the netbook would not have the portability of the iPod Touch, but it would have other strengths (like a keyboard) to make up for it.

Another is software. I have Palm OS software that I've used for years that will no longer be usable. I have to take that into account also.

For now I will limp the Zire along and see what Apple comes up with on Monday at the WWDC and go from there.

It's been interesting watching the rise and fall of each of the form factors and companies. From Sharp (still around but out of the PDA business) to Palm (with a history of mis-steps and frittering away their huge lead) to Apple (whose Newton venture paved the way for more functional PDAs). It will be interesting to see where things progress to. There are rumors that Apple will expand the iPod Touch into a small tablet and effectively re-launch the Newton. I'll be paying close attention.

What do I do with my data?

At this point in time there are two broad categories of approaches you can use to manage your data:

• Relational
• Non-relational

Since one is defined in terms of the other I'll start with relational.

Like the name suggests, relational data are items that can be related to one another through the use of a key value. For example, your government issued an identifier number to you that is used to identify you. This number is then linked to your name, where you were, born, etc. It is a key to reaching that information.

If you think of a table of information, with rows running horizontally across the page and columns intersecting the rows vertically, running down the page, the key value identifies the row. So, for information about person you might have something like this:

Key	FirstName	LastName
100	John	Smith
101	Fred	Brown
...	...	...
174	Zachary	Miller

Each key identifies a row, and each row has columns of information. Then, in another table where you store, say, order information, you do not have to enter John Smith's personal information every time you create an order for him. The data exists in one place. Then, if you find out that he spells his name "Jon" and not "John" you can change it in once place.

There are other advantages to the model, but this is the crux of it. Information only has to be entered once, and then related to other information.

A relational database management system (RDBMS) is software that allows you manipulate this type of information easily.

Non-relational is a grab bag of different approaches that do not include necessarily the idea of a key value (or at least not in a separate tables sort of way). Generally a non-relational data store will store all of the information for any one thing all together, all at once. For example, an order would be stored all on one line, with the customer information repeated over and over for each order they are linked to. You wind up storing the information more than once for certain items and you have a lot of work to do if you want to update that information or use it to link to information that is held elsewhere.

Storing information in a spreadsheet often winds up being non-relational but in more modern systems there are ways to make the data relate to other data. The lack of management tools, however, will generally drive you towards a management system.

If the data you are storing is VERY simple and small in size you can save time and money by just storing it into a text file that you update by hand. For anything beyond that you should consider more sophisticated software to help you with the process of storing and updating the information.

Weighing the pros and cons of the various different packages is a very involved process. Think through what your needs will be and then invest the time in some research. Make sure you base your decision on your needs and not on what some marketing tells you your needs are.

Make a techgnostic choice for your database needs and you will achieve your goals effectively and efficiently.

What kind of a database do I need?

So, we covered the need for one. Chances are you need one. But which one? First we have to think a bit.

There are some questions to consider the answers to:

• What kind of data am I storing? Is it just text and numbers, or do I have to store files or picture? Do I have to store documents?
  
• How much data am I storing? Do I have a few dozen customers and a couple of suppliers, or do I have hundreds of both? Do I have historical information I need to put in, and how much of it do I have?
  
• How many people need to get at it? Is this just for me, or do other people need to get at it also?
  
• What other requirements do I have? Speed, costs, etc. Does the system have to lightning quick or can you wait a second or two? Do you have a budget to spend on this, both for software and somebody to handle it?
  

There are several broad categories of alternatives.

• If the amount of data is small, simple and textual you could get by with a text file. An XML file can easily hold a fair amount of structured data and does not require very powerful software.
• If the amount of data is larger and maybe a bit harder to manage you might need a spreadsheet like Open Office Calc, Google Doc or Microsoft Excel.
• If your data needs are larger you start to get into actual "database" software, of which there are many different options from free to very expensive. The choices stretch out a great distance, but include smaller ones like Microsoft Access or FileMaker or ones that will scale out as needed like MySQL, PostreSQL or Microsoft SQL Server (various editions).
  

In each case the number of people need to access the data will drive what you wind up with. Think about your needs before buying what people are selling and come back to determine what kind you need. Answering these questions will help you be techgnostic in your choice.

Do I need a database?

"Oh, obviously you need a database."

What?

What is a database, you ask? All you know is that it lives on the computer somewhere and keeps track of things. Why can't you just keep your paper and files? Or are you already using a computer? "Look, see - I have it all in a spreadsheet!".

The person who told you of your need for a database rolls his/her eyes and groans and says "THAT is not a database!"

Confusion reigns.

So, what is it?

Strictly speaking, ANY collection of data is a database. Your paper and files? That's a database. A bunch of index cards in a box? Database. That spreadsheet of yours. Also a database. Any one of many different software products with the word "database" in their name? Yeah, database. Really, a database is an organized body of related information. Period. The exact mechanics of it are where people get hung up.

Honestly, the person who recommended a database to you most likely meant that you would benefit from a "database management system" (DBMS) such as has been around for many years, specifically designed to manage your information. Even though your stack of paper is technically a database how long does it take you to find a very specific piece of data out of it, say, an invoice from a particular vendor? When the pile is small it is trivial. As the pile grows it becomes non-trivial.

You've moved from a small pile / number of cards / number of spreadsheets. What happens when someone else needs to get at the same information? Any more than a couple of people cannot effectively search through your pile. And when the information must be updated what do you do? Mark up the original? Attach sticky notes? Put the file on a shared drive and hope for the best? And if you want to know some kind of aggregate information like "how much money do I owe all people with outstanding invoices?" you now have a much larger task on your hands. That is where a DBMS really comes into its own.

A DBMS has been designed, from the beginning, to efficiently store your information, with methods to capture, store, analyze and create outputs based on the information stored within. They are at the core of much of the efficiency that is attributed to computer technology.

For anybody who is doing anything in their lives that requires capturing information, storing it for future use, sifting and analyzing it and creating output such as reports or simply answers to questions about the data the need for a computerized database is clear.

However, just like the question of "do I need a computer" (most often "yes") is followed by "what kind of computer do I need?" there are some questions you have to ask yourself before you start down the path of using a DBMS. As the blog title implies we will cover this in a techgnostic fashion in the next post.

For a sneak peek, the questions will be along the lines of:

• What kind of data am I storing?
• How much data am I storing?
• How many people need to get at it?
• What other requirements do I have?

That will be followed by another post wherein some of the more popular alternatives are considered, as well as some less popular choices. Come to think of it, this might become a series of posts. Stay tuned.

The rise of the Netbook

Making computers smaller was a goal from the very earliest incarnations of the technology. Down from the size of a room to the size of a few fridges. Down from that to a largish box that could sit on the top of a desk. Soon someone asked "How about one I can take with me?" Portable (in the sense that they COULD be moved, not necessarily EASY to move) computers themselves iterated through various forms. Of course someone said "What if it ran on batteries?" and computers became truly portable, freed from the tether of the electrical grid (or a generator). The same evolution has continued over the years with portables becoming smaller and more powerful with each cycle.

Until very recently these smaller and more powerful machines commanded premium prices. You could expect to pay much more than you'd pay for a desktop computer with similar power. The intent was that the portable computer should have the same power (or close to it) as a desktop. This meant all kinds of trade-offs had to be made and components had to be engineered to be even smaller. All this engineering did not come cheap and having a laptop computer was a badge of honor and the province of the technological (or just wealthy) elite.

Eventually a kind of tipping point was found, not unlike with cell phones. In the same way that the huge cell phones of the 1980's become smaller and smaller into the 1990's so that design decisions started to rotate around making the keys LARGE enough for human fingers to operate as the phones had become so small laptops started to change also. Two kinds of consumers were looking for portable machines, with two sets of criteria: as SMALL as possible (most often frequent travelers and/or gadget freaks) versus as POWERFUL as possible (the so called "desktop replacement" crowd). Each of these consumer types was provided with what they wanted, but in all cases you continued to pay dearly for the machines.

As time and Moore's Law marched on an intersection occurred. This was the intersection between relatively powerful laptops and the ubiquity of wireless internet. Again something that was once the province of the technical elite became very common, with wireless access available virtually anywhere for a very small fee, if at all. Instead of existing as an alternative to a desktop computer the laptop became basically a terminal to connect into the content of the internet. The need for the terminal hardware (i.e., the laptop) to be powerful was lessened. Beyond a certain amount of power was a waste. A simple machine could be used to read e-mail, surf the web, watch videos, etc. The hardware of a new type of laptop was ready. The other part necessary was the software.

Open source software, which languished for some years in obscurity from the public, leapt to the forefront of these new machines, dubbed "netbooks" as they are basically used to access networks and not much else. Rather than having to pay the "Microsoft tax" for every machine netbook manufacturers started to load open source operating systems like Linux onto their machines, erasing the extra payment. This dropped the price a little bit more, along with the smaller screen, small hard drive and lack of optical drive to produce a netbook for a few hundred dollars instead of almost a thousand dollars.

The effect on the market was slow to build but once it did it ran away, catching the large manufacturers unawares. Coinciding with some economic hard times netbooks flew off the shelves and accounted for most of the growth in the sales of portable computers. Now all the major manufacturers have their own netbooks out and Microsoft is scrambling to try and find a way to ensure that they are a presence in the market, albeit at a lower per unit cost as in the past.

For many people a netbook is all the computer that they need. They can use a web based provider of e-mail, store their pictures on a photo sharing site, use online applications for word processing and generally do what they need to do without very much regard to the computer that they do it on. If their inexpensive netbook bites the dust the only thing you might need to do is rediscover your bookmarks (although even those can be cached away by one of many online utilities).

There will always be a market for the more powerful and expensive laptops for people who need the power of local processing for tasks like video processing or engineering that burns up a lot of CPU cycles, in the same way that there will still be people who will spend thousands of dollars on desktop machines to do very specific tasks. In both cases, though, there are now inexpensive alternatives, often using open source software, that give people just what they need for a small amount of money.

Keeping in mind what you're trying to get done before you choose your technology is a techgnostic thing to do. Think it through the next time you advise somebody on what kind of computer to buy or examine your own needs the next time you put your money down.

Reduce, reuse, recycle.

Being green is seen as increasingly important for companies. Sometimes it's a stretch to find something to trumpet about but you can always find something like using recycled paper in your printers or such. If the new measure actually saves the company some money (not often), all the better.

One place that companies could definitely get more bang for their green buck is with computer hardware. The hardware itself caused a lot of pollution during its manufacture and when it winds up in a landfill it leaches a lot of nasty chemicals. If you can avoid having to buy new computer hardware you can feel better for being green as well as save some money.

Most often the things that will fail on a computer are the parts that move: the hard drive, any fans in the case and the power supply fan. Otherwise, in the absence of a power surge, the rest of the solid state electronics will keep going for years (at least five). It used to be that software always outpaced hardware and you always had to buy new hardware to keep up. Hardware is now ahead of software in most cases (unless you're talking about niche needs like high-demand video games or such). With some more RAM (inexpensive) and maybe some new parts that you interact with (monitor, keyboard and mouse) you will feel like you have a new computer.

If your needs are not directly tied to software available on Macintosh or Windows you can load one of the freely available Linux distributions and get your computing tasks done very handily, while also saving some money and making your computer more secure. If you are not comfortable with such a move you could just back up your data and do a clean re-install of your operating system and again feel the joy of a new space. If you are 100% uncomfortable with either of these suggestions you can go and buy yourself a new computer but keep in mind that the prices are much lower than they used to be and you don't need a lot of power for most things you're going to do.

As always, you need to think about what it is you're trying to do when you are considering a hardware purchase. Do you want what you're getting just because you're "due" for some new hardware, or do you have a specific need for it? Are you tied to a particular software platform? If not, when you get new hardware is an ideal time to take look at alternatives. The computer market is in flux right now and there are lots of new approaches being tried.

Obviously everyone's situation is different, but what I am advocating is not necessarily just going on to the next revision of what you've always done, instead taking a look at alternatives, including keeping your existing hardware and just refurbishing it a bit. You'll save some money and help the earth by being green.

The hard stuff is easy, the soft stuff is hard.

Technology, of whatever type, is a tool and that is all. Outside of enjoying a tool purely for its own sake (tool-o-phile?) there is no value to a tool except what you do with it. You don't buy a drill, you buy holes. A shovel might represent a crystallization of effort with a forged head and carefully turned handle, but it is just a shovel. Until someone picks it up and digs a hole with it there is nothing to see, move along.

"High" technology needs to be viewed the same way. It can become VERY complex and challenging to understand, but ultimately it is just a tool. A database without data in it is an empty shell. Even when full of data it means nothing until you DO something with it. Transform the pieces of data into information that you can use to make decisions. Again, the technology by itself is nothing. It takes people to use it to get value from it.

Technology is for most people the "hard" stuff. People skills like communication and collaboration are considered "soft" skills. Depending upon your personality type you might gravitate towards one or the other. In the end, though, the effectiveness of whatever your enterprise is will rest upon the behavior of the people using the technology. In most cases the hard stuff will come with a default setup that works in most cases, and if it doesn't you can find someone who can make it work for your situation. Really, in the grand scheme of things, that was the easy part.

The "soft stuff" is where the real challenge lies. Many technology people consider the soft stuff relatively worthless as it does not improve the speed of their hardware or software. It does not help directly with their up-time. It does not give them more disk space to store data. Very often they will gloss over the people-oriented items or leave them to others.

This is a mistake.

If you don't take into account who is going to be using your system you will fail. The end users of the system are most often not technologists. They are people who want to use a tool to get their job done. They want to get in the car, push the gas pedal and drive somewhere. They don't want to have to know any detail whatsoever about how the engine works, how the linkages between the control systems ensure that the vehicle goes in the correct direction, etc.

Gas, go, done, next.

You can build the most technically elegant system in the world but if you don't take into account the people who are going to use it you are dooming your efforts. This is true for systems that are internal to companies, or external to be sold to others. You can mandate usage internally (you WILL use the new system) but it will never be accepted and unless you have a monopoly you have no hope externally.

If you find yourself looking at a technology decision be sure to take into account the soft stuff. Who is going to use it? What are they going to do with it? What is the LEAST that you can do to make the system work? Systems do not exist for their own sake. They exist to serve people in some way. Keep that foremost in your mind and the decisions about what to do should be clearer.

Once more unto the breach

I guess a year is an unreasonable gap. Lots has gone on in my life. Many events that kept me from thinking about an online journal. Most significantly was the passing of my father, who influenced me very strongly. Few days pass by that I do not think of him and wish I could talk to him.

2008 was a year of transitions for me on many levels both personally and professionally. Now I think it's time to re-engage a few things I'd shelved during the chaos. One of those things is this blog. I will commit myself to a personal goal of posting to this blog once per week, come Hell or high water. I've already queued up a few weeks worth of topics (not yet written).

Going back and reading over the older posts I realize that my interest are more or less the same as they were, and I will be continuing in the same vein. That means more posts about the various manifestations of information and computer technology, their use in business process management and trying to make all the parts of the process dance together: people, technology and business.

And not only that, but also from the point of view that what's important at the end of the day are the results and not sticking to any one religion (e.g., Microsoft vs. Mac vs. Linux). Sometimes a post will be just because it's interesting.

Once more unto the breach, dear friends, once more.