As i look around at some of the leading companies across the globe, I find a state of chaos across these companies, especially in information management. What is interesting to note is that the problem transcends Fortune 100 to Small Business and is very similar. Where did things go wrong? all these companies have spent valuable dollars to improve information management.

Deeper investigation reveals several gaps and loopholes. The vendors that came to provide strategy and technology services addressed "point in time" problems and developed focused solutions. With the craziness of technology improvements, market conditions (m&a), business changes all coming together, the entire situation is like playing "poker" at the high stakes tables in Vegas.

How do we unravel from the state of chaos to a state of order. Hiring more strategy consultants is not the only answer. We need to do a holistic overview of the business situation and find out where issues have come from and how the current solution process will address today and the future.

In order to recover from a state of chaos to clarity, you need to look at the strength of the organization and its people, assess its technological prowess, its competition, business value, market value, strategic maturity and adoption of processes in the organization and more.

However doing all this in a short span requires a great deal of focus and more maturity from the vendor that will be hired for this exercise. Size does not matter in this spectrum.

A vendor rating from the Analyst community is a starting edge of this process, but that alone is not enough, we need to ask for organization maturity, delivery capability and much more to not repeat the mistakes to get to a state of chaos. Sometimes a few "niche" vendors may be needed to solve the larger problem.

To be continued.....

We all know by now the word "columnar database" and some of the most popular providers - ParAccel, Vertica and Infobright. Though columnar databases have distinct performance advantages and applications that can be deployed, from a business perspective, there was a question mark and doubts of where to apply these solutions.

In my opinion, the unstructured data integration into the data warehouse is a very key area of applicability for the Columnar database. The reason for this statement, stems from the fact that the unstructured database is a very column oriented data store and if we are processing oodles of text, the reference of information will be more clustered in nature. This drives the need to store the data in a co-located manner, which then leads me to look at columnar databases more closely.

I do not imply that Oracle, SQL Server, DB2 or Teradata cannot support the unstructured database needs, but for those who want to adopt to the cloud, use more on demand scalability etc, columnar databases may be an in-house option. Additionally for those providers in cloud arena, columnar databases may be more optimal to adopt to.

I'm currently running tests on Infobright and will share the results in the next few days. I'm processing a large volume of semi-structured and unstructured data and will measure the throughput and performance against the standard RDBMS platforms.

In my technical opinion, choosing a columnar database for processing large textual data provides more business benefits, the final results can be migrated to a corporate environment, but operational aspects can be done in the columnar platform.

I have been researching on this subject for the past few months. It is becoming clear that to control errors in diagnosis and treatment of medical conditions, Evidence Based Medicine can be defined as "the conscientious, explicit and judicious use of current best evidence in making decisions about the care of the individual patient. It means integrating individual clinical expertise with the best available external clinical evidence from systematic research." (Sackett D, 1996).

This requires research and processing of clinical information that can be leveraged by a Doctor. While search from Google, Oracle and others are supporting the search portions, Clinical data is heavily semi-structured - a combination of structured and unstructured data. The unstructured portions is where you have all the content and context of diagnosis, treatments, prognosis, conclusions etc. This is where Textual ETL and the unstructured database build will help us. The entire DW2.0 methodology can be directly applied to this system. We can classify, categorize, contextualize and capture information from the clinical data into a database.

In conducting the research, I have found that we can process years of clinical data into an unstructured database with relative ease and when combined with the statistical research data, the results have been an eye-opener. The possibilities of reducing errors and offering the right treatment means the cost of healthcare can be managed well. The quality of the treatment will be more rewarding and welcome.

I know that many Doctors and other medical professionals have different opinions on this subject, but that was when no technology was available to support EBM, now that we do, we should start paying attention to this area.

I will be writing my first article for this year on this subject, stay tuned for the article.

I'm stumped by irrational behavior from large technology corporations, especially ones that feel that they have big muscles and have earned the right to such behavior. They go to any lengths to be winning, and do not realize that what they think is collateral damage from their perspective, can be a serious business factor in the future.

What do these companies think? do they even think before they get on an assault charade? do they care for ethics and values? if they do not, do we also start questioning their ethics and values?

The end result of such negative behavior is the loss of trust in the long course from their own customers and marketplace. Sadly their focus is on today's gain and not the overall future perspective probably, which to an extent explains such behaviors. One does not know how many times these companies have lost business due to these behaviors.

I hope that at some point in time, these corporations and their management do realize the value of earning the trust of their customers and the market is very important for their continued success and it is not just technology or financial glories that carry an impact.

In the past few days I have been seeing a flurry of corporate activity in Twitter, Facebook etc from large corporations like Schwab, Starbucks, Victoria's Secret etc. Not only have they got the corporate websites, catalog advertisements, direct mail, email, now they are connected in the "digital world".

The reason these companies and others have the "digital" presence, is because today you need to be connected to the customer to create a positive business impact for the business. How do you stay connected? only if you keep listening. With the advent of social media, the world has shifted from a passive to an active participant in providing businesses feedback. Not only is the feedback public, it is also creating disruptive impacts.

Then there is also the question of competitive threats and also collaborative opportunities. Due to all these business impacts, Social Media is becoming a very essential business requirement for businesses today

There is a TDWI night school that I'm holding on this topic next week for two days. If you are attending, we will see you there.

As you look at the Web2.0 and Web 3.0 world, it is becoming clear that information management on the internet with respect to data will be a catalyst to your success or failure in the new world order. Traditionally when you build applications in the Web, you do not look at it being very data savy and largely transactional in nature. But looking at the way Flickr, Facebook, Twitter and Social networks have changed the game, we understand that no longer are we looking at a transactional silo, but rather need to react to a "long tail".

In the new world, you will need to be "big" but "nimble", large and flexible. Wow that's a mouthful to keep saying. The reason for this thinking is we need to look at the structured and the unstructured data to understand the customer and their needs, and react quickly to address those needs. When you talk of unstructured data, in a Web world you cannot afford to load Gigabytes and Megabytes of data, most of which is noise. You need to get the intelligence extracted and linked, but leave the content and the context outside. How do you accomplish this? there are some companies addressing this need, but we need a nimble and strong ETL engine to do this process.

This is where you need to look at Textual ETL and understand how to build the unstructured database. The traditional vendors are doing their part, but the end result has left a lot to your imagination.

Textual ETL is complex and deals with data which has minimal structure and completely 180 degrees opposite of transactional data. As we move towards the Web 2.0 --> Web 3.0 world, we will encounter this hurdle and I hope there are tools that will handle this Large data management to integrate the transactional and textual data.

I have seen large organizations fail with providing successful BI services. Often the blame is on the quality of service that was provided in terms of people and technology. But dig deeper and you will see that a lack of cohesive focus is where that organization would have lost the impetus in providing the service.

One may ask what is cohesive focus. When you deal with a large organization, there are multiple processes and teams involved. There is bound to be different levels of maturity amongst the teams and this gap leads to lack of clarity and hence the lack of cohesive focus.

What are the signs of lack of focus? the warning signs will start when you see an alignment issue between the different managers in the team. The earliest warning will be when a single point of contact cannot be deployed to the project. When these signs manifest, I urge both the service provider and the customer organization to start working together at the earliest to mitigate downstream issues.  Often the damage is done by the time either organization realizes the situation.

When service providers decide to engage in program services, one word of advise, ensure that your organization is having checks and balances to avoid surprises. If you lack the desired level of maturity, ensure that you do not oversell the client expectations. Another food for thought is to develop a mutually agreed scorecard to monitor and correct course over the period of service.

No analyst organization will provide a course on how not to fail. But if you are too large or too small an organization, watch out for the lack of cohesive focus, which is your achilles heel.

Remember since this is a competitive domain, there will be no sharing of best practices or approaches. Learning from introspection is the best approach.

Companies today in all spheres from media to healthcare are struggling with the "Network Effect" phenomenon as I call it. The age of social networking has matured from zero to reality in a short span of five years, which has ushered in the "Network Effect".

What is the "Network Effect". Simply put it is the way we connect with each other as people today. One simple example is the movies, in the past few months movie producers have started looking at RSS feeds, Twitter and other social media avenues to advertise their movie and provide additional promotions to the target audience. Why?

The advent of text messages , Twitter etc have provided a channel for people to communicate their views and opinions on any subject. Such a trend has shown its impact on movies - for example last summer's release of "Transformers" did not succeed in generating interest in the young adult community as the movie did not receive rave reviews from first day or first week viewers and most had negative sentiments about the movie. On the same note, the current success of Avatar, is highly attributable to the genius of James Cameron, but the network effect of people recommending others to see the movie for its astounding technical effects and causing a mass attraction to the theaters shows the effect of "Network Effect".

We will continue to see the influence of "Network Effect" in the coming months and years. We will see in the coming days more discussions on this subject.

Year 2010 will be the year of Social Analytics in Business Intelligence. The prime reason for this thought stems from the fact that organizations have heavily invested in CRM, BI and all possible solutions to better manage themselves and their customers. But what is the use of all this data, if you cannot connect to the real world? i.e what do people talk about you, your products, services and your competition? how can you really understand the "network effect" that your customers will bring to your business?

As we move more closer to connecting the dots, the largest gap will lie in the social analytics arena. One can understand that the data volumes we are discussing is extremely large in nature, complex to process, more complex to decipher value from and extremely volatile in nature. This is exactly why I'm looking at Social Analytics being the "next BIg thing in BI".

Who are the companies that are investing into product development in this are, there are a few that come to mind, but I will hold off on revealing them in this blog.

Over the next few days, this blog will focus on this theme and expound this further.

The BI vendor market has become an imbalanced space. There is either consolidation leading to BIG behemoths or you have extremely niche startups or the last choice Open Source. What this leads to is confusion in user organizations.

With market consolidation, companies are left with a mixed bag of solutions and now need to reassess their investments, new market offerings have not reached enough maturity and open source is not accepted yet as enterprise capable in BI. Where we need to go with this situation is to setup an interoperable solution where the vendor consolidation will not impact current investments. There were third party companies that used to offer these kind of solutions and we need a new series of such technologies to be recreated.

I'm really wondering why we blame DW / BI architecture or approach as a failure, when a large portion of the silos are being built by vendor disruption in the market. I certainly hope the vendors will provide a scalable and interoperable answer in the near future.

Till we get some sensible set of solutions, we might have to live in a heterogenous world of solutins and thrive on chaos.