SharePoint, SQL and Online Dating

I am often asked “What do I tell IT when they want to replace an existing Lucidea solution with one they promise to build in SharePoint?” Well, there are many advantages with our Lucidea solutions, but here I’d like to share one simple but very powerful differentiator: date handling and date searching. Once you’ve read this, you’ll have a thought-provoking response for IT!

Just so we are clear, I am not talking about dates in the Match.com sense! I mean dates such as the issue date of a serial publication, or the historical period of an artifact.

In SharePoint and SQL databases, dates are very precise. They use only two field types when dealing with dates:

• Date fields: Formatted as DD-MM-YYYY (e.g. 12-06-2009)
• Timestamp fields: formatted as or DD-MM-YYYY + a timestamp (e.g. 12-06-20091:36:58 PM EST)

This means that in SharePoint and SQL you can search for an exact date or time, and you can search for a range of dates or times.

Fuzzy logic

But for museum collections, KM initiatives and libraries, dates are often less precise. In fact, sometimes dates get downright “fuzzy.” For example, publishers often publish materials with imprecise dates, such as “Spring 2014” or “Winter 2002” or “2015 Edition.”Maybe you need to enter a date for an item in the collection, but it’s only an estimate so you need a text annotation. The problem with both SharePoint and SQL is that they are unable to deal with these kinds of situations. Hmm…is this reminiscent of a bad Match.com date?

Consider the options

At Lucidea, we understand dates, and we understand that they are often “fuzzy.” Below are some questions you can ask IT about how dates will be stored and how dates will be searched in SharePoint and SQL.

• Does the date query “2014” get automatically run as a date range query: Jan 1, 2014 to Dec 31, 2014?
• Does the date query “June 2014” get automatically run as a date range query: June 1, 2014 to June 30, 2014?
• When importing records with the value “Spring 2014” in the date field, will the records be imported? (Don’t be shocked when the IT answer is that these records will be rejected.)
• Does the date query “Winter 2014” find serials with a published date of “Invierno 2014” in the date field?
• Does the date query “Jurassic” have any meaning? (In Lucidea’s Museum application, Argus, it does!)
• Can you store a date such as “June 1, 2014 – estimate”? (In SharePoint and SQL, I don’t think so!)
• Can you easily (by “easily” I mean with no coding) set up relative date searches so that your users can answer questions such as “What new content has been added in the last week?” or “What articles have been published on topic X in the previous two months?”

You can tell your IT department that Lucidea develops KM, library automation and collections management solutions so that we can answer “yes” to all these questions, because we know that in the real world, dates are not always well-behaved (just like the “dates” from Match.com).

From data to meaning

How do we do this? Simply put, at Lucidea we add a layer of natural language date processing during data ingestion and during search. This ensures that the information exists in its raw and organic form, but we store more than the raw data: we store its meaning. We do the same during search, interpreting the query’s underlying meaning. So, for example, “Spring 2014” isn’t just a word search—when it’s entered in a Lucidea date field it has a very specific meaning—it’s a date range search, combining both precision and flexibility.