Jeff Jonas

Master Data Management (MDM) vs. Sensemaking

I get asked from time to time how Master Data Management (MDM) relates to my work in Sensemaking Systems.  There is sufficient confusion to warrant a blog post on this subject, so here it is.

Different missions, different tools.  Some organizations will use one or the other; most organizations will want both.

MDM in a nutshell

MDM is about helping companies gain control over business information by enabling them to manage and maintain a complete and accurate view of their master data.  MDM deals with a known problem.  Master data is “intentional” business data that is structured and flows from systems under one’s control.  Common master data domains include customers, products, and accounts.  The MDM outcome: organizations have higher quality, reliable and consistent master data records.

Sensemaking Systems in a nutshell

Sensemaking is about helping companies make sense of their diverse observational space, ranging from data they own and control (e.g., structured master data) to data they do not or cannot control (e.g., externally-generated and less structured social media).  Sensemaking deals with uncertainty around an unknown and every changing domain ? as if operating over an arbitrary number of puzzles, each unclear, incomplete and riddled with inconsistencies.  Common Sensemaking domains include customers, watch lists, social circles and investigations.  The Sensemaking outcome: organizations make better decisions, faster.

Both MDM and Sensemaking Systems make organizations more competitive.  Both can serve real time or batch missions.  And both play together quite nicely.

Now to highlight some stark differences between these two missions and hence the reason the algorithms used in each domain are fundamentally so different.

Rich Data vs. Poor Data 

Because master file data is owned and controlled by the organization, this data tends to be more complete (feature rich).  For example, customer records often contain a name, address, and some form of further identification like a date of birth or tax identification number.

While high fidelity data is better, Sensemaking Systems must routinely attempt to associate low fidelity observations e.g., connecting the dots between a third party watch list containing only name, nationality and date of birth fields and an internal cyber investigation database containing only email and IP addresses.

The algorithms required to deal with low fidelity data require union/set-based entity resolution algorithms which are fundamentally different than the record linking algorithms used when dealing with rich data.

Bad Data Bad vs. Bad Data Good

Organizations that deploy MDM seek to improve data quality at collection, e.g., detection and elimination of duplicate accounts during data entry ? the purpose being clean records.  This kind of control over internally generated master data means less bad data gets in the front door.  And in this pursuit of “golden” master data records … bad data is bad.

Contrast this with Sensemaking Systems.  In this class of system, errors such as misspellings and numeric transpositions are valuable regardless of whether these errors have been generated by accident or are professionally fabricated lies created by sophisticated criminals.  When a criminal uses different alias and date of birth combinations, Sensemaking Systems meticulously record this untrusted information, permit this ambiguity to fester, and monitor numerous versions of truth.  This “bad data” is sometimes the only source of an important clue. 

The algorithms used to harness an out of control observation space are quite different than those algorithms used to manage internally controlled, higher quality records.

Manageable Data vs. Unimaginable Big Data

For most organizations, master data will not exceed 500M records.  At these scales, the automated triage of changes to master data results in a small to modest number of records that must be arbitrated following human review.  Fortunately, as an organization’s master data is a closed domain … there is a real chance humans can keep up as “uncertainty” is minimal and manageable. 

Sensemaking Systems must be able to deal with extremely large data sets ? potentially involving tens to hundreds of billions of records ? being generated from an ever more diverse range of data sources (e.g., from Twitter to OpenStreetMap).  At this volume and diversity the “uncertainty” is beyond the capacity of virtually any human review.

The algorithms used to deal with the enormous ambiguity tend to be very different than algorithms that routinely face smaller, more controlled data sets that routinely benefit from human participation.  Sensemaking algorithms have also been seen to actually get smarter and faster over ever growing datasets ? an exciting feature for customers staring information overload in the face.

Firm Facts vs. Predictions

MDM is about being in control of the master data that an organization generates and about the truth of this data.  Master data becomes so reliable that its assertions are treated as fact ? and these facts are so firm they are rarely reversed or invalidated.

Because Sensemaking Systems are attempting to harness an out of control observation space, assertions about context are constantly postulated, reevaluated and retroactively readjusted.  This accumulating context is viewed more like a prediction that is expected to evolve as more observations are considered.

The algorithms required to constantly reassess and context correct the historical observations with every new observation, in real-time, are fundamentally different than algorithms that do not have this requirement.

Event Triggered vs. Everything a Trigger

MDM processes are triggered during master data events.  MDM is used to guide the collection and management of master data.  For example, real-time duplicate detection during:

• Customer on-boarding at a financial services company
• Account set-up at a telecommunications company
• New product creation at an insurance company
• Patient admission at an emergency room or doctor’s office

Sensemaking Systems are more akin to “intuition support systems” ? treating every piece of data that falls within the organization’s observational space as the potential to learn something new, detect something relevant, or as the basis for an alarm.  As each new observation arrives in the enterprise, the enterprise needs to know if it just learned something new, confirmed something it already knew, or has something wrong.  And with each new observation it must answer the hardest question of all (for a computer), “How does this relate to what I already know?  Does this matter and, if so, to whom?”  For example:

• As fast as the consumer joins the insurance company’s Facebook group, the insurance company recognizes the consumer’s significant social influence and is now instantly prepared to respond accordingly during the next business interaction
• The customer’s on-line retail purchase this morning is used to render a more relevant “next best action” recommendation later that morning when they visit the physical store front, e.g., the coupon on the back of the receipt no longer includes the item they already purchased
• While processing new deceased persons’ data from the Social Security Administration, notification is send to the marketing department suggesting they “inactivate” those persons in the bank’s prospect database
• During the creation of a casino arrest report, an alert is generated when the subject provides the same home address as that provided by the dealer on his employment application

The algorithms necessary to publish insight at ingestion based on incremental context accumulation and complex event processing are fundamentally different than algorithms that do not have this requirement.

MDM + Sensemaking

If an organization has a well constructed MDM system, it has a good handle on its own data and processes. This MDM data being a great starting point for Sensemaking Systems.  Nice, but not required.

Ironically, when Sensemaking Systems are used in weak signals domains e.g., the detection of criminal activity, irregularity and errors in the data being presented by adversaries happens to be extremely useful.  So while MDM systems (by design) clean up data entry, Sensemaking Systems should be provided the good, bad, and ugly data ? pre- and post- cleaning.  Point being: if you polish every piece of data to perfection, one may never find the weak signal.

Different technologies.  Different missions.   And a simple integration story.

RELATED POSTS:

 

Smart Sensemaking Systems, First and Foremost, Must be Expert Counting Systems

Sensemaking on Streams ? My G2 Skunk Works Project: Privacy by Design (PbD)

Entity Resolution Systems vs. Match Merge/Merge Purge/List De-duplication Systems

It Turns Out Both Bad Data and a Teaspoon of Dirt May Be Good For You

Smart Systems Flip-Flop

The Data is the Query

Data Finds Data

Big Data. New Physics

Data Beats Math

 

“It is a capital mistake to theorize before one has data”

~ Sir Arthur Conan Doyle

 

Over the years, folks have often asked me what kind of math am I using to create large scale, real-time, context accumulating systems (e.g., NORA).  Some fond of Bayesian speculate I am using Bayesian techniques.  Some ask if I am using neural networks or heuristics.  A math professor said I was doing advanced work in the field of Set Theory.

My answer is always, “I don’t know any math.  I didn’t finish high school.  But I can explain how it works, step-by-step, and it is really quite simple.”

That reminds me of a related funny story.  After IBM acquired my SRD company in 2005 I began touring IBM’s impressive research facilities around the world.  During a visit to IBM’s Almaden Research Center I explained my techniques to a room full of researchers.  A few months later, to my surprise, they sent me a technical paper to express my work … using math.  Fascinating I thought.  The idea that my algorithms are now expressed in math terms was really exciting.  Could it be?  I was so curious.  So I asked them to humor me and take me through the paper very slowly via a conference call.  It was actually a bit embarrassing.  I started out by asking the question what does an equal sign mean when a colon is in front of it?  Symbol by symbol I asked for an explanation.  Then I asked about this thing shaped like the letter “U” … what does that mean?  (Union as it turns out).  Anyway, I was able to follow the math and it all made sense until about halfway through the paper when I spotted an obvious error.  So I said, “um, the math here is inconsistent with my technique.”  I suggested a fix.  The phone went quiet for a minute and then about 45 days later they came back with a new and improved paper.  Continuing where we left off, I found a similar discrepancy further down the page and then provided some more specifics about my technique.  Unfortunately, I never received another draft.  Clearly, they could have.  But honestly, I suspect they simply lost interest in having to teach me math.

I wish we would have finished that paper, as then folks trained in formal methods would better understand what I am doing and seeing.

One of the things demonstrated by this mathy paper might have been the notion that “data beats math” ? at least when it comes to Assertion Algorithms.  Based on the available observation space, can an assertion be made?  Yes or no.  In short, there comes a point where sufficient evidence exists such that an assertion can be made as a “no-brainer” without feeling compelled to split hairs with probability math.

Here is practical example.  Imagine being presented with two identity records?

Record #1

Name: Mark Smith

Date of Birth: 05/12/1987

SSN: 555-00-1122

 

Record #2

Name: Mark Smith

Date of Birth: May 1987

D/L: 0099912334

Are they the same person?  It is certainly possible.  Using population statistics and some math someone could compute a reasonably accurate probability.  I say heck with using math to guess.  I’d say where can I find some glue around here?  For example, a record like this:

Record #3

Name: Mark K Smith

Date of Birth: May 12, 1987

D/L: 0099912334

SSN: 555-00-1122

So the point is: I’d rather look for corroborating and/or dissenting evidence than look to math for estimated probabilities.  And if a really important outcome might come from such an assertion, I would continue to seek observations until it was so obvious you could show the board of directors and they would say “duh.”  If you run out of available observations and you are still not sure … then you have a few choices: 1) locate and collect the kinds of observations you need, 2) wait until you luck into a future observation related to the assertion in question (letting the existing ambiguity fester), or 3) pound on it with math.  But I say only pound on it with math if it is going to be worth the additional effort/compute (e.g., you are playing high-stakes poker in Vegas).

My gripe, if any, is that way too many people are chipping away at hard problems and making no material gains in decades (e.g., entity extraction and classification) … when what they actually need is more data.  Not more of same data, by the way.  No, they more likely need orthogonal data ? data from a different sensor sharing some of the same domain, entities and features (e.g., name and driver’s license number).

When the quality of mathematical predictions start to flatten out, I recommend increasing your observation space.  Hence the above reference to this awesome quote:

 

“It is a capital mistake to theorize before one has data”

~ Sir Arthur Conan Doyle

 

RELATED POSTS:

Accumulating Context: Now or Never

Smart Systems Flip-Flop

Algorithms At Dead-End: Cannot Squeeze Knowledge Out Of A Pixel

How to Use a Glue Gun to Catch a Liar

It Turns Out Both Bad Data and a Teaspoon of Dirt May Be Good For You

Smart Sensemaking Systems, First and Foremost, Must be Expert Counting Systems

The Data is the Query

I have been talking about the notion that "the data must find the data and the relevance must find the user" for some time now.

Another way to think about this is "the data is the question."

As each new piece of data arrives in the enterprise, the enterprise just learned something.  And with each new observation one should ask, “How does this relate to what I already know?  Does this matter and, if so, to whom?"

This is the world of sense and respond, situational awareness, sensemaking or whatever you want to call it.

Example: An employee in the bank’s credit department changes his home address in the payroll system.  What if the employee's new address is the same address currently under investigation by the bank's own fraud department?  How would the bank know?

They wouldn't.

When the data is the query, a change to a home address in the payroll system is determined, at that split second, to be the same address under investigation by the fraud department.  And, at that split second this is determined to be relevant, so the fraud department is notified.

Real-time, sub-second.

When organizations can process arriving observations for relevance … organizations will be more competitive, or might even, for the first time, seem to be “awake.”

Note: The systems/technologies that are going to do this are very different than what organizations have in place today.  Existing operational systems cannot do this.  Neither can master data management systems, data warehouses, operational data stores, data mining engines … not even Hadoop Map/Reduce.

 

RELATED POSTS:

Data Finds Data

General Purpose Sensemaking Systems and Information Colocation

On A Smarter Planet … Some Organizations Will Be Smarter-er Than Others

 

Sensemaking on Streams ? My G2 Skunk Works Project: Privacy by Design (PbD)

Over the last twenty-eight months I have been quietly running a skunk works effort that I’ve code named “G2.”  To my delight, on January 28th, 2011 this system became officially viable and will be entering something akin to a “sea trial” phase through 2011.

I believe this system will prove to be my most innovative work to date.  I also believe it is the most responsible technology I have created to date.

This new technology, something that might be characterized as a “big data analytic sensemaking” engine, is designed to make sense of new observations as they happen, fast enough to do something about it, while the transaction is still happening.  This engine brings to life many of the principles I have been openly sharing on my blog, ranging from Sensemaking Systems Must be Expert Counting Systems, Data Finds Data, Context Accumulation, Sequence Neutrality and Information Colocation to new techniques to harness the Big Data/New Physics phenomenon.  That said, as this is version 1.1, there remain many things to do to realize my full vision.  It is a very ambitious effort, but more about that some other day. 

In terms of responsible innovation, I am even more proud to report that my team and I have baked in, from conception, more privacy and civil liberties enhancing technologies than any other product I am aware of to date. 

Friday, January 28^th, 2011 ? my official launch date ? also happened to be the international Data Privacy Day.  And on this day, internationally recognized privacy commissioner, Ann Cavoukian hosted a few hundred privacy executives and practitioners from around the world in Toronto Canada at her Privacy by Design: Time to Take Control conference.  During my keynote entitled “Confessions of an Architect” I highlighted seven (7) exciting features that have been baked into this new technology (Privacy by Design), specifically:

1. Full Attribution

2. Data Tethering

3. Analytics in the Anonymized Data Space

4. Tamper-Resistant Audit Logs

5. False Negative Favoring Methods

6. Self-Correcting False Positives

7. Information Transfer Accounting

The full presentation is here.

Here is a summary of the above seven PbD features:

1. FULL ATTRIBUTION: Every observation (record) needs to know from where it came and when.  There cannot be merge/purge data survivorship processing whereby some observations or fields are discarded.  Why is this so important?

A. If received data does not contain its data source and transaction pedigree, then system-to-system reconciliation and audit are virtually impossible, especially in large information sharing environments.

B. If the system merges and purges observations, only later to discover the wrong observations were merged or purged, then without full attribution correcting these earlier mistakes can be difficult if not impossible.  The typical alternative being periodic batch re-processing.

C. The Universal Declaration of Human Rights has four articles containing the word “arbitrary” e.g., Article 9 reads “No one shall be subjected to arbitrary arrest, detention or exile.”  If you don’t know where the data came from or when, how can any resulting action be anything but arbitrary?

2. DATA TETHERING: Adds, changes and deletes occurring in systems of record must be accounted for, in real-time, in sub-seconds.  Why is this so important?

A. Data currency in information sharing environments is important, especially if one is using data to make important, difficult to reverse decisions that affect people’s freedoms or privileges.

B. When derogatory data is removed or corrected in a system of record, it is vital to reflect such corrections immediately.  For example, if someone is removed from a watch list, how long should they have to wait before their name is cleared?

3. ANALYTICS ON ANONYMIZED DATA: The ability to perform advanced analytics (including some fuzzy matching) over cryptographically altered data means organizations can anonymize more data before information sharing.  Why is this so important?

A. With every copy of data, there is an increased risk of unintended disclosure.

B. Data anonymized before transfer and anonymized at rest reduces the risk of unintended disclosure.

C. If organizations can now share information in an anonymized form and still get a materially similar result, why would organizations want to share information any other way?

[Technical Note: As every anonymized value maintains full attribution, re-identification is by design to support Data Tethering as well reconciliation and audit.]

4. TAMPER-RESISTANT AUDIT LOGS: Each record of who searches for what should be logged in a tamper-resistant manner ? even the database administrator should not be able to alter the evidence contained in this audit log. Why is this so important?

A. Every now and then people with access and privilege take a look at records without a legitimate business purpose, e.g., should an employee at a financial services institution take a peek into their roommate’s file.

B. Tamper-resistant logs make it possible to audit user behavior.

C. And, when the word gets out to the work force that such accountability exists, this can cause a chilling effect on misuse.

5. FALSE NEGATIVE FAVORING METHODS: The ability to more strongly favor false negatives is of critical importance in systems that could be used to affect someone’s civil liberties.  Why is this so important?

A. In many business scenarios, it is better to miss a few things (false negatives) than inadvertently make claims that are not true (false positives).  False positives can feed into decisions that adversely affect people’s lives ? e.g., the police find themselves knocking down the wrong door or an innocent passenger is denied the ability to board a plane.

[Technical Note: Sometimes a new observation can lead to multiple conclusions.  Systems that are not false negative favoring may select the strongest conclusion and ignore the remaining conclusions.  But had the strongest candidate not existed, the second strongest conclusion would be asserted.  One false negative favoring method involves remedy such a condition, for example by reversing an earlier conclusion should a future observation bring to light that fact that multiple possible conclusions now exist.]

6. SELF-CORRECTING FALSE POSITIVES: With every new observation presented, prior assertions are re-evaluated to ensure they are still correct, and if no longer correct, these earlier assertions can often be repaired ? in real-time, not end of month.  Why is this so important?

A.  False positives occur when an assertion (claim) is made, but is not true.  If relied upon to make a decision, false positives can adversely affect people’s lives e.g., consider someone who cannot board a plane because he or she shares a similar name and date of birth as someone else on a watch list.

B. Without self-correcting false positives, databases start to drift from the truth and become provably wrong (even to the naked eye) ? necessitating periodic (batch) reloading to true-up the database.

C. Periodic monthly reloading to correct for false positives means wrong decisions are possible all month until the next reload, even though the system had everything it needed to know beforehand.

[Technical Note: Reversing earlier assertions in real-time at scale, as new observations present themselves, is computationally non-trivial.  Imagine making an assertion that two people are the same because they share exactly the same name, address and home phone number ? only later to learn through another series of observations that these are really two different people (a junior and a senior).  Our “self-correcting false positives” feature self-corrects for these rare cases, in real-time.  We consider our ability to perform sequence neutrality at scale one of several breakthrough aspects of our work.]

7. INFORMATION TRANSFER ACCOUNTING: Every secondary transfer of data, whether to human eyeball or tertiary system, can be recorded to allow stakeholders (e.g., data custodians or the consumers themselves) to determine how their data is flowing.  Why is this so important?

A. It is often cumbersome to learn who has seen what records, or what records have been shared with tertiary systems.

B. Much like a US credit report that contains an inquiries section exposing the list of recent inquiring parties, now so can your medical or financial file.

C. Users can now be easily provided with such disclosures, increasing transparency and control e.g. enabling a consumer in some cases to request an information recall.

D. When there is a series of leaks, information transfer accounting makes discovery of who accessed all records in the series quite trivial.  This can narrow an investigation when looking for criminals within.

What has me most excited is that where some features above would typically be an extra priced option in my new system so many are built in (e.g. this tamper-resistant audit logs).  And some of our privacy and civil liberties enhancing features cannot even be turned off!

Yes, there is an official name for my new technology.  And no, I’m not telling you, because this is not a sales pitch.  Rather, I am simply trying to inspire other technologists to consider Privacy by Design as they innovate.

I’ve had two most great days at IBM.  The first great day was in January 2005 when IBM bought my company, SRD.  And the second greatest day came six years later on January 28^th, 2011.

RELATED MATERIAL:

Privacy by Design (PbD)

 

RELATED POSTS:

Smart Sensemaking Systems, First and Foremost, Must be Expert Counting Systems

Data Finds Data

Accumulating Context: Now or Never

Sequence Neutrality

General Purpose Sensemaking Systems and Information Colocation

Big Data. New Physics.

Source Attribution, Don’t Leave Home Without It

Data Tethering: Managing the Echo

To Anonymize or Not Anonymize, That is the Question

Immutable Audit Logs (IAL’s)

Out-bound Record-level Accountability in Information Sharing Systems

Big Data Flows vs. Wicked Leaks

Decommissioning Data: Destruction of Accountability

Re-hosting Legacy Systems: Ground Hog Day

One of my favorite things to do is help organizations think about next generation architectures.

When talking to IT executives about them, one of the more common things I hear is “We need to re-host these damn legacy systems.”  This often makes me chuckle and then I share a story:

Back in say 1996, I was chatting with a Las Vegas CIO who was telling me how essential it was to replace the hotel’s damn legacy Property Management System (PMS).  The funny thing is: these were the same words he had used just three years prior when he upgraded to his current system.

Seen that movie called Ground Hog Day?  Yeah, like that.

All systems are legacy systems.  If not the moment they are implemented, then just wait a few minutes.

With this in mind, often the best end-state to wish for is simply one that is even better suited for change.

 

OTHER RELATED POSTS:

Scalability and Sustainability in Large Information Sharing Systems

Big Data Flows vs. Wicked Leaks

I was invited to deliver a short keynote about "big data" at today's OECD roundtable focused on the economics of personal data and privacy.  My presentation here.

Most big data flows by design.  But when big data leaks the consequences can be wicked. 

That said … protecting big data from wicked leaks is not going to be easy.  Defending against external cyber penetrations and insider threats are both hard problems.

Now with the Wikileaks disclosures it is clear the game has changed.  Historically, public disclosure of classified data has been limited and infrequent ? let’s even say, to a degree, tolerable.  Contrast that with the scope of the recently leaked cables.  At some point it may become intolerable.  In which case, I will not be surprised if a number of governments around the world attempt to enact new, wide-sweeping, anti-leak legislation directed at not only those engaged in the initial theft of the data, but the distribution points (e.g., Wikileaks) and the publishers (e.g., the media).  The principle being if one knowingly receives and benefits from stolen property, they are accomplices.  This pendulum could swing so far (backwards) the future will have far fewer media leaks than the historical (tolerable) volumes ? i.e., this whole fiasco resulting in less transparency and accountability.

BTW: I suppose it could have been worse.  What if the 250,000 classified cables where selectively and quietly passed around to various foreign intelligence services?  What if the US thought these were secrets?  Imagine believing one has a certain security posture … when one does not.  Would that be worse? 

Organizations with big data worth protecting must employ extraordinary controls to reduce the risk of unintended disclosure.  On that note, I closed with a few ideas related to protecting big data from wicked leaks, including:

Central indexes.  There are actually a number of scenarios where a single, central catalog of pointers is better than lots of copies of the same data scattered all over the place ? the advantage being fewer copies of the data and uniform access controls and audit logs.

Anonymization.  Despite the imperfections of data anonymization, when it comes to reducing the risk of unintended disclosure, most would agree that data anonymization is still better than clear text.

Immutable audit logs.  Tamper resistant audit logs can be used to help prove the users of the system are complying with law and policy.

Real-time active audits.  It is now going to be essential that user activity be more rigorously analyzed, in real-time, for inappropriate behavior.  Audit logs have actually been part of the problem ? just another big pile of data ? evidence of misuse hiding in plain sight against the backdrop of millions and millions of benign audit records.

 

RELATED POSTS:

Big Data. New Physics.

More Death Cheaper in Future

How Many Copies of Your Data? Is Somewhat Like Asking: How Many Licks to the Center of the Tootsie Pop?

It’s All About the Librarian! New Paradigms in Enterprise Discovery and Awareness

"Need to Know" vs. "Need to Share" ? A Very Fine Line Indeed

To Anonymize or Not Anonymize, That is the Question

Immutable Audit Logs (IAL’s)

Using Transparency As A Mask

General Purpose Sensemaking Systems and Information Colocation

General purpose sensemaking systems will colocate diverse data in the same data space.  Such an approach enables massively scalable, real-time, novel discovery over an ever changing observational space ? without re-engineering.

This of course should be no surprise.  Ever since Von Neumann’s assertion that computer memory should be used for operating instructions and data ? not two different memories for the different purposes ? general purpose computing became possible.

“Von Neumann machines differ in that they have a memory in which they store their operating instructions and data.  Such computers are more versatile in that they do not need to have their hardware reconfigured for each new program, but can simply be reprogrammed with new in-memory instructions; they also tend to be simpler to design, in that a relatively simple processor may keep state between successive computations to build up complex procedural results.  Most modern computers are von Neumann machines.”

~ Wikipedia: Computer data storage

Data structure governs function.  For example, a room full of DVD’s behaves one way and a SQL database behaves another.  Same holds true with enterprise operational systems: The human resources system uses one data model and the hotel reservation system another ? each underlying data structure designed for each specific mission ? and notably, of little use to anything else.

With information trapped in the tailored database schemas of systems of record, operational data stores, data warehouses and data marts, it is no wonder organizations continue to struggle to make sense of it all ? despite decades of effort and innovation.

Performing some kind of federated search over all these disparate data sets just has not ever delivered.  In fact, federated search bites when it comes to sensemaking because the diverse data structures are incapable of supporting a sensemaking function.

If you want to be smart, you will want to jam the available, diverse, observational space into the same data structure and in as close to the same physical space as possible. 

Data is data.

When reference data, transactional data, and even user queries are colocated in the same data structures and is the same indexes as the extracted features from text, video, biometrics, and so on … something very exciting happens: data naturally finds data and context can accumulate.

By way of background: I first stumbled into the importance of data colocation back in 1993 when designing a surveillance system for the casinos in Vegas ? a system that would help them keep the bad guys out.  After claiming I could build such a system in 90 days for $25k, I was forced to take some short-cuts.  Honestly, had the casino given me more time and been willing to pay more money I would have created a much more elaborate system containing a number of tailored database schemas (e.g., different structures for customers, employees, bad guys, vendors, stored user queries, etc.).  Given the time and money constraints, I had to make some compromises.  I decided to design one schema to support everything.  Each record in the system would then be designated with a role e.g., “Customer” or “Bad Guy.”  Long story short, when this general sensemaking system came on-line it started finding marketing hosts comping their roommates and lots of other unanticipated novel discovery.  So much novel discovery, it earned the name Non-Obvious Relationship Awareness or NORA, we got two rounds of funding, IBM bought my company to get its hands on the technology, and the rest is history.

Simply said, you have to have a brain (multi-purpose, general structure) to think (sense make).  Then with a brain, the smartest you are going to be is a function of what observations you have properly contextualized into that meat space between your ears.

 

OTHER RELATED POSTS:

Smart Sensemaking Systems, First and Foremost, Must be Expert Counting Systems

Asserting Context: A Prerequisite for Smart, Sensemaking Systems

Puzzling: How Observations Are Accumulated Into Context

When Federated Search Bites

Federated Discovery vs. Persistent Context ? Enterprise Intelligence Requires the Later

Accumulating Context: Now or Never 

What Came First, the Query or the Data?

Enterprise Intelligence ? My Presentation at the Third Annual Web 2.0 Summit

Big Data. New Physics.

I spoke at Defrag 2010 earlier today and introduced what I am describing as the new physics of big data.

Having designed and deployed a number of multi-billion row context accumulating systems over the last 14 years I cannot help but notice some very interesting, very exciting phenomenology.  Not research.  Not Theory.  Real.

1. Better Prediction.  Simultaneously lower false positives and lower false negatives.  A bit more about this here: Prediction: Channel Consolidation and Puzzling: How Observations Are Accumulated Into Context. 

2. Bad data good.  More specifically, natural variability in data including spelling errors, transposition errors, and even professionally fabricated lies ? all helpful.  A bit more about this here: It Turns Out Both Bad Data and a Teaspoon of Dirt May Be Good For You and There Is No Such Thing As A Single Version of Truth. 

3. More data faster.  Less compute effort as the database gets bigger.  A bit more about this most exciting phenomenon here: The Fast Last Puzzle Piece.

And more good news, as context accumulates, a better sense of when and where to place one’s attention (apply compute effort) develops -- including: (1) very smart observation filters and (2) fully automated ability to determine very specific, very relevant questions ? answers to which it may decide to fetch itself.  A system that Googles itself?  I have not blogged about this thinking as of yet, but hopefully will one of these days.

Anyway, imagine that: As the database grows, fewer CPU cycles are needed for better predictions and you never really wanted to clean all that data up in the first place.

I also took this keynote opportunity to share my latest skunk works project ? a project my team and I have been working on for almost two years now.  Yes, it’s true, I am building something ? a sensemaking engine, designed to fully harness this big data phenomenon.  Among other exciting properties, this system will also have an unprecedented number of privacy-enhancing features baked into it.  Internally I have been calling this little skunk works effort “G2.”  And when this little girl grows up I have big hopes for her.  For example, maybe she will help cancer researchers find a cure.

My Defrag 2010 MS PowerPoint presentation here.

 

RELATED POSTS:

Algorithms At Dead-End: Cannot Squeeze Knowledge Out Of A Pixel

Smart Sensemaking Systems, First and Foremost, Must be Expert Counting Systems

Data Finds Data

Puzzling: How Observations Are Accumulated Into Context

Your Movements Speak for Themselves: Space-Time Travel Data is Analytic Super-Food!

“Macro Trends: The Privacy and Civil Liberties Consequences … and Comments on Responsible Innovation” ? My DHS DPIAC Testimony, September 2008