MySyndicaat Feedbot: sachamonotti_Data and Text Mining

Is Your City HyperLocal?

Matthew Hurst — Fri, 03 Jul 2009 21:51:17 GMT

Seattle resident’s are spoiled for choice when it comes to hyperlocal blogging. In addition to a couple of networks – Next Door Media {My Ballard, PhinneyWood, Magnolia Voice, Queen Anne View, Fremont Universe} and Neighborlogs {Capitol Hill Seattle, Central District News, The Southlake, etc.} there are a number of independents (e.g. West Seattle Blog). All told, there may be up to 100 blogs which focus on residential issues in and around Seattle.

Is Seattle unusual in its coverage, or are there other places that have a thriving hyperlocal blogging culture? Please comment if your neighborhood is covered.

...

Interactive Dashboards: US.Gov Aces, Wimbledon Double Faults

Thu, 02 Jul 2009 14:16:57 GMT

The US government has unveiled a dashboard to track IT spending, which looks at first sight a shining example of the power and value of business intelligence. The Wimbledon tennis dashboard, on the other hand, appears less than impressive

As reported in The New York Times, the so-called IT Dashboard was put together by the US government in a remarkably short six weeks, and will "track and analyze the more than $70 billion a year that the federal government spends on information technology."

Government Dashboard Scores a Bulls-Eye

Thu, 02 Jul 2009 13:23:56 GMT

The US government has unveiled a dashboard to track IT spending. Apparently put together in a remarkably short six weeks, it looks at first sight a shining example of the power and value of business intelligence. But don't take my word for it... browse the dashboard yourself!

As reported in The New York Times, the so-called IT Dashboard will "track and analyze the more than $70 billion a year that the federal government spends on information technology." Some features of the IT spending dashboard include:

Naughty Feeds

Matthew Hurst — Thu, 02 Jul 2009 02:26:21 GMT

Do you have a naughty feed? Come on, admit it. You deliberately left out the title, or did you put in an empty summary? Maybe you’re the one who doesn’t put in any dates, or perhaps you set the permalink to the home page of your blog. Well – you are a naughty blogger, shame on you!

Bing haz Twitter

Matthew Hurst — Thu, 02 Jul 2009 02:23:40 GMT

This is very cool. When Bing recognizes a search for a celeb (of the real space or other varieties), it will provide an answer composed of their Twitter identity and recent tweets. Not yet rolled out for me, but the Bing blog has a screen shot (repeated here).

CEP, Events, and Continuous {Transformation | Intelligence}

Wed, 01 Jul 2009 15:23:36 GMT

Given that BI thought leaders are wrestling with the notion of events, perhaps we will see a BI-mainstreaming of event processing in the not-too-distant future. Myself, I was way ahead of the game in my expectations of demand for BI access to stream sources. While a combination of legacy database and analytical technology has held BI back, lack of perception of need has been a far greater factor, especially given the under-utilization of conventional BI decades after the term first became popular.

Interest in streams and events has definitely picked up in the last few months -- I've reported on novel applications for "continuous transformation" and otherwise done a bit of writing to promote awareness -- and next year could very well be the break-out year for BI on data and event streams.

Mulling the Mystery of Microsoft's BI Market Share

Wed, 01 Jul 2009 13:24:05 GMT

How do you gauge Microsoft's business intelligence market share when it gives BI functionality away for free? That was a bit of a puzzle even before Microsoft's January announcement that it would end development of the PerformancePoint Server, the one and only entirely BI-focused product the company had. As part of that move, Microsoft now bundles what it calls "PerformancePoint Services" -- dashboarding, scorecarding and analytic capabilities -- into the enterprise edition of SharePoint. That's bundled as in free, just as Reporting Services and Analysis Services have long been bundled with Microsoft SQL Server. Microsoft's BI front end is Excel, the general-purpose spreadsheet tool that's part of the Office suite -- seldom purchased separately or used exclusively for BI.

So now if you own Microsoft SQL Server, SharePoint (Enterprise) and Office, you already own Microsoft BI, and these days, Microsoft executives take every opportunity to point that out. In an interview earlier this month, Kristina Kerr, group product manager of Microsoft BI told me, "the move we made in January has definitely spurred on a lot of growth and a lot of interest in BI among SharePoint Enterprise customers. These are tough economic times, so a lot of customers are looking internally to see what they already own and see how they can make the most of it."

Twisting Terms to Make BI Market Share Claims

Tue, 30 Jun 2009 16:18:25 GMT

I always look forward to IDC's annual BI market shares, waiting to see who comes out on top and who is losing ground. I'm sure everyone noticed that they were, in fact, a month early this year! Call me a bean counter at-heart, but I like the irrefutable, quantifiable comparison they bring that other evaluations (including my own BI Scorecard) lack. Or so one would think.

Most product evaluations involve a degree of subjectivity, with varying definitions and opinions of what capabilities and criteria matter more. The IDC market share figures, on the other hand, are cold-hard facts: what were the revenues, who's leading, who's growing. While such data may have been somewhat trackable on 10Ks when BI vendors were independent, it's now largely impossible as BI is often but a small part of a larger company. Software vendors don't have to report their revenues by market segment, and throughout the year, we only get vague, imprecise references about how the BI business is going. The IDC report lays out the breakdown by vendor.

Free, Blogs

Matthew Hurst — Tue, 30 Jun 2009 13:11:17 GMT

The Blog Herald comments on the Free skirmish between Gladwell and Anderson by pointing out

The blog is alive and kicking, if nothing else but because it is hard to pick critics and arguments to pieces in 140 characters or less.

I wrote quite a bit about the book when it was being formulated. Now, I’m looking for the (free) online version but haven’t found it yet. Chris/Wired is known for erring on the side of sensationalism to move units (as is appropriate in the mediate business), so I wonder what his investment in this new thesis is.

...

Bucking the Cloud Computing Hype

Mon, 29 Jun 2009 11:23:33 GMT

Years ago I was running a large software development shop when the "offshoring" movement became all that and a bag of chips. I felt extreme pressure to fire some of my staff in the US and cut a deal with the dozens of outsourcing firms that were calling me daily. I was asked about offshoring so many times that it was clear to me that the pressure was on.

I had to buck the offshoring hype at the time, but for good reasons. First, our systems had issues around quality and architecture. Until we corrected those issues, bundling the code up for development offshore would do little good, and could actually kill the software. Once we stabilized the code, then I could pick portions of it to be developed in outsourced development shops, but not until then. It was very tough explaining that to laymen who just saw the dollars and cents, along with the trend and hype that told them offshoring was the way to go.

Measure, Don’t Guess – Growth in the Blogosphere

Matthew Hurst — Sat, 27 Jun 2009 23:41:07 GMT

Charles Arthur writes a piece about the slow demise of the blogosphere. Arthur asserts that bloggers are a fading bread, and that

they've all gone to Facebook, and especially Twitter.

Arthur claims to have come to this conclusion via a mixture of anecdotal evidence, and data provided by Technorati. Let’s do our own experiment to see if the blogosphere is fading. Let’s take a very mundane search term – one that we expect to be a constant background in the sea of celebrity death buzz, hi-tech launches and liver transplants : ‘car repair’. As we can see from the Blogpulse graph, it is pretty stable with a few blips here and there:

Blogpulse plots the percentage of all blog posts on this topic. If the blogosphere were dying, the absolute counts would also be slowly reducing (even if the percentages were staying the same as the graph shows).

On Jan 4th, 0.026 of posts were on the term ‘car repair’. This translates to 142 posts (Blogpulse allows you to click through to see the number of hits). On June 21st, where there were 0.027 % of posts on the term, Blogpulse registers 144 hits. Ok, I don’t really see any slacking off there. What happens when we look at more data points? If we do this for ‘car repair’ and ‘birthday’ we get the results below. Here I’ve normalized the values by the percentage of posts (count/percent

...

The Long Tail of Text Mining

Matthew Hurst — Sat, 27 Jun 2009 23:10:30 GMT

In systems that execute inferences via a pipeline of steps, every step is an opportunity for failure. Therefore, it is imperative that implementers focus attention on the details of every step. For example, in text mining, systems have to

Import and parse documents – did you get the title? did you recognize the footers? did you strip out the page numbers?
Identify sentences and words – is the document in a latin alphabet language? are there word separations? are you dealing with acronyms? how is your unicode-fu?
Provide part of speech tags for the words – is the text an example of the type of data that the POS tagger trained on?
Identify entities – are you prepared to identify unusual names like Barack Obama?
etc.

I’ve seen a couple of attention to details bugs surface in the past few hours. The first was reported by Danny Sullivan, in which Google (and this is still the case at the time of writing) thinks that Michael Jackson the writer is the most salient person with that name.

The second is visible on WeSmirch, in which the system fails to identify the title of Lisa Marie Presley’s blog, naming it ‘Create Free Blogs & Online Journals on MySpace Blogs’:

...

Steve Irwin, Michael Jackson

Matthew Hurst — Fri, 26 Jun 2009 13:45:51 GMT

When Steve Irwin, the famous crocodile hunter, was killed by a sting ray, 5.5% of the posts on that day in the blogosphere (September 05, 2006) mentioned his name. Yesterday, mentions of ‘Michael Jackson’ topped out around 3%. Things are heading north from there today – currently around 8% – but note that the more immediate statistics come with a higher margin of error due to the lower sample size.

...

When Business Gets Too Personal

Fri, 26 Jun 2009 10:50:48 GMT

Visualization guru Stephen Few reminds us that analyst opinions, while offered by recognized experts, are inherently personal, and that on the other side of the table, there are real people behind products, marketing campaigns, and corporate decisions. I'll amplify that each of us does bring unique personal experience and even personality to bear when reviewing (analysts) or promoting (vendors) products, and I'll agree that we should each be accountable for what we write or claim.

It's an analyst's personal perspective, coupled with strong judgment, communications skills, and fairness, that creates a sense of authority and makes his or her views worth reading. Good analysts don't blindly accept vendor claims. We investigate, and sometimes we reject what we've been told. But I disagree with Steve that analysts should always name names. Some situations become simply too personal. I and others I know have even been the subject vindictive behavior, which unhelpfully diverts attention from products to people. In the worst cases I've seen, the vendor can even exploit personal conflict to dismiss or attempt to denigrate the analyst. Among cases I've encountered recently, involving other analysts and myself.

Twitter Trending Terms – Could Do Better

Matthew Hurst — Fri, 26 Jun 2009 05:59:14 GMT

Twitter has oodles of data – millions of tweets a day. They have smart people working on this data, and they make all the right noises about social search. However, I’m looking at the trending topics on the site just now and I see these: MJ’s, Rip MJ, RIP Michael Jackson, Farrah Fawcett, #iranelection, Pop, Thriller, MTV, Iran, #michaeljackson.

I don’t get it. There are a number of problems here:

These aren’t topics, they are words or phrases. There are only 4 topics present here (Michael Jackson’s death, Farrah Fawcett’s death, the Iranian Election and MTV).
The ‘phrases’ present in the terms are pretty lame: ‘MJ’s’? Earlier today the phrase ‘Did Michael Jackson’ was a trending topic.
There is no attempt at normalization (RIP MJ == RIP Michael Jackson)
They actually are not at all interesting – anyone out there not know about Michael Jackson, Farrah Fawcett or Iran? Perhaps the MTV thing is a little more obscure.
They aren’t trending – ok, the RIP stories are, but the Iranian election? that’s been top of mind for many days now.

Twitter’s featuring of these topics and the immaturity of the technology in spite of the promise of the data does not paint a good picture for their prospects.

...