collective unconscious of the smarter planet

Alternative database technologies - including NoSQL - are beginning to cut into Oracle’s revenues.

Something is seriously wrong in Larry Land. Oracle does not command absolute control like it once did. You can see this clearly with the earnings the company posted last week and the growth that startups like Datastax are witnessing as more customers seek alternative databases for online applications.

Until this past week, the extent of Oracle’s problems were not known. But there is a cut, a slight bleeding that’s now visible. But how deep is the cut? How much is Oracle really bleeding? That’s exactly the question analysts asked in a Reuters story after the earnings results:

“Data base revenue, which has been the cash machine of the company, has changed. There are now alternative databases, as well as the cloud,” said Mark Moerdler, an analyst at Bernstein Research. “That pressure is still a tiny bleed, but it is out there and the question is – is it bigger than we think it is?”

We know this much. Oracle reported this week that new software licenses are down two percent. And that decline is in part reflected by the adoption of NoSQL databases offered by Datastax and a variety of other services that use in-memory technology and new SQL offerings at the database layer. Update: Of course, there are competitive forces at play with enterprise giants such as SAP, IBM and Microsoft playing their part. But it’s the startups that represent the core of the innovation.

The reason for the drop has more to do with the enterprise acceptance of online applications more than anything else, said Datatastax CEO Billy Bosworth in an interview last week.

That’s the truth. NEA Ventures Scott Sandell said to me at SXSW that CIOs are convinced to move their workloads but cloud security is still an issue.

That’s where companies like Datastax enter the picture. Datastax is built on Cassandra, a high performance, fault tolerant Apache open-source database technology.

Datastax, founded in April 2010, finished its first year with 26 employees. It ended 2012 with 100 employees. Bosworth expects to have 160 people on staff by the end of this year.

Customer growth has increased significantly. By the end of 2011, Datstax had 27 customers. One year later it had 270, with 20 from the Fortune 100.

Several dozen of those customers have moved either all or parts of their application off relational technology such as what Oracle provides.

When companies come to Datastax, they say the number one thing they need is security, Bosworth said. They are building from day one to avoid disaster scenarios.

Datastax, like other NoSQL providers, spans its database technology in a fully distributed way, across private data centers and the cloud.

Datastax differentiates by offering high performance at scale but without complexity.

How customers use Cassandra reflects on why Oracle growth has begun to stall. Often, customers will continue to use Oracle databases but will put it deeper in the backend. They will take another piece of the app and put it on Datastax.

Customers will build in a middle layer of services components that allows the app to decide which database to use for which workload.

Lighter Oracle workloads means less revenues, which we see reflected in the company’s earnings.

To counter this swarming hive of distributed systems, Oracle has taken the opposite approach, building out engineered solutions with their software running on big, new age mainframes. That strategy does not seem to be working very well. Oracle bought Sun Microsystems with plans to sell the hardware with its software.

Analysts tend to agree:

“The problem is, the growth of SaaS (software as a service) applications is undermining that strategy. When you subscribe to salesforce.com, you don’t need to buy a database, middleware or hardware,” said Patrick Walravens, an analyst at JMP Securities in a Reuters story last week.

Oracle has lost money every quarter since it acquired Sun for $5.6 billion. And there is little proof that companies are going to start using one company like Oracle for all their hardware and software needs. Instead, they will mix Oracle software on commodity systems. Or they may even go with the new open-source server technology coming out of Open Compute. They have plenty of other options, too. OpenStack, the open cloud effort, is growing fast, as is Cloudstack, the open-source cloud service now part of the Apache Foundation.

Datastax has its own challenges. It competes with Amazon Web Services and all the other NoSQL providers such as 10gen. The ecosystem is still quite young. Finding qualified people is a challenge. Developers need more education, a change in thinking for the new cloud approach.

But overall, it’s clear that Oracle really is starting to show the pains of being an aging innovator. The earnings show a slight cut. The question is how deep the cut is and how Oracle will respond to challengers like Datastax.

An interesting corrective to the idea that pure, black-and-white analytics is a panacea. Analytics isn’t a replacement for vision and creativity. The most successful companies have a firm view of a possible future and they align their priorities and decisions around it.

Thiel described the argument Zuckerberg finally came down on like this: ”[Yahoo] had no definitive idea about the future. They did not properly value things that did not yet exist so they were therefore undervaluing the business.” 

Thiel told this story to make a larger point about how the most successful entrepreneurs operate. He said that the best entrepreneurs, like Zuckerberg, have a definitive view about the future and plan for it; they don’t willy-nilly chase luck—using statistics, probability, and iterative processes—to stumble upon something, anything that flies.

“All of us have to work toward a definite future…that can motivate and inspire people to change the world,” he said. In this scenario, “luck is something for us to overcome as we go along the way, but not something that becomes this absolute dominating force that stops all thought.”

Thiel doesn’t subscribe to what he calls the start-up “religion” of a-b testing every tweak (until you run out of money) or incrementally-iterating at every step—to be so systematically chasing some random success that it strips out all conviction and creative ideas about the future.

A really interesting article by Intel for IEEE Xplore on the future of big data in the entertainment industry. Despite the entertainment focus, there are some good points about big data in general.

Could Watson kill cultural creativity? Slate explores a potential downside of big data -

Watson—IBM’s supercomputer—is about to start wading through thousands of legal and medical documents to make assessments that no lawyers or academics can (not with so many data to look through, anyway). If the goal is to analyze what has sold in the past and try to predict what, based on all these data points, is likely to sell in the future, Watson could easily expand into music, film, and books.

Alas, such expansion, while benefiting sales, might stall cultural innovation. Would Watson be able to predict—if it were around back then—the rise of the impressionist painting or of the futurist poetry or the new wave cinema? Would it have approved of Stravinsky? Big Data would have probably missed the Dada.

To understand the limits and opportunities of algorithms in the context of artistic creation, we need to understand that the latter usually consists of three elements: discovery, production, and recommendation. Startups like Fuzz target the last element—recommendation—hoping that some would rather be guided by humans than algorithms.

RedMonk argues that R is rapidly becoming a go-tool analytics tool thanks to bottom-up advocacy by statisticians who used it extensively in school -

The ongoing promotion of developers from serf to kingmakers has many implications, but perhaps none so important as technology adoption. For decades, we’ve been seeing the manifestation of this trend, as the growth in market share of technologies from Chrome to Linux to Mac to MySQL have been driven at least in part by developer populations that preferred them. This was in evidence yet again last week on a Google Hangout James and I participated in organized by IBM.

In discussing the market for analytics, one of the topics of discussion was people, or more specifically the lackthereof. One of the least controversial statements one can make in the technology industry today is that demand for talent is outstripping the supply, at least in most markets. Analytics are no exception. As a result of this shortage, many organizations are the proverbial beggars now unable to be choosers. Where they may previously have hired for analytical roles only those trained on sanctioned analytical tools, businesses are now compelled to hire outside of these comfort zones. And in the analytical world, this most often benefits the statistical language R.

When developers – or in this case, statisticians – are permitted to choose their own tools, an increasing number turn to R if for no other reason than the fact that it was the basis for their academic training. So prevalent is R in academic environments, in fact, that the datasets associated with the text taught in my second semester of statistics were available as a downloadable package via R’s repository, CRAN.

What this means, then, is that like Linux, MySQL, PHP, AWS and other technologies before it, R is being annointed in bottom up fashion as a fundamentally important technology moving forward – it’s not often we get to watch this in real time. Whether enterprises approve of that or not. Even those conservative enterprises who wish to dictate technology choices to their rank and file will find that increasingly difficult in a tight labor market. Fragmentation is the new reality, and as developers make more choices – if only by default – the number of technologies employed within enterprises will inevitably rise. Those businesses that understand this at a fundamental level and do not seek to oppose it will have significant advantages in both hiring and productivity. Those vendors, meanwhile, that appreciate that hetereogeneity is the new norm and optimize for interoperability will, likewise, benefit.

As for the developers? With their newfound influence comes greater responsibility; just as enterprises need to be more flexible in technology adoption, so too should developers be more careful in the selection process. Heterogeneity is fine, even beneficial. Chaos, less so.

Some more interesting thinking on the “end of insight” from Steven Strogatz at the Edge.org -

I worry that insight is becoming impossible, at least at the frontiers of mathematics. Even when we’re able to figure out what’s true or false, we’re less and less able to understand why.

An argument along these lines was recently given by Brian Davies in the “Notices of the American Mathematical Society”. He mentions, for example, that the four-color map theorem in topology was proven in 1976 with the help of computers, which exhaustively checked a huge but finite number of possibilities. No human mathematician could ever verify all the intermediate steps in this brutal proof, and even if someone claimed to, should we trust them? To this day, no one has come up with a more elegant, insightful proof. So we’re left in the unsettling position of knowing that the four-color theorem is true but still not knowing why.

Similarly important but unsatisfying proofs have appeared in group theory (in the classification of finite simple groups, roughly akin to the periodic table for chemical elements) and in geometry (in the problem of how to pack spheres so that they fill space most efficiently, a puzzle that goes back to Kepler in the 1500’s and that arises today in coding theory for telecommunications).

In my own field of complex systems theory, Stephen Wolfram has emphasized that there are simple computer programs, known as cellular automata, whose dynamics can be so inscrutable that there’s no way to predict how they’ll behave; the best you can do is simulate them on the computer, sit back, and watch how they unfold. Observation replaces insight. Mathematics becomes a spectator sport.

If this is happening in mathematics, the supposed pinnacle of human reasoning, it seems likely to afflict us in science too, first in physics and later in biology and the social sciences (where we’re not even sure what’s true, let alone why).

When the End of Insight comes, the nature of explanation in science will change forever. We’ll be stuck in an age of authoritarianism, except it’ll no longer be coming from politics or religious dogma, but from science itself.

Computers may be extending our knowledge - but might they extend it further than we can truly comprehend?

But what if it were possible to create discoveries that no human being can ever understand? For example, if I were to give you a set of differential equations, while we have numerical and computational methods of handling these equations, not only could it be difficult to solve them mathematically, but there is a decent chance that no analytical solution even exists.

So what of this? Does such a hint of non-understandable pieces of reasoning and thought mean that eventually there will be answers to the riddle of the universe that are going to be too complicated for us to understand, answers that machines can spit out but we cannot grasp? Quite possibly. We’ve already come close. A computer program known as Eureqa that was designed to find patterns and meaning in large datasets not only has recapitulated fundamental laws of physics but has also found explanatory equations that no one really understands. And certain mathematical theorems have been proven by computers, and no one person actually understands the complete proofs, though we know that they are correct. As the mathematician Steven Strogatz has argued, these could be harbingers of an “end of insight.” We had a wonderful several-hundred-year run of explanatory insight, beginning with the dawn of the Scientific Revolution, but maybe that period is drawing to a close.

So what does this all mean for the future of truth? Is it possible for something to be true but not understandable? I think so, but I don’t think that that is a bad thing. Just as certain mathematical theorems have been proven by computers, and we can trust them, we can also at the same time endeavor to try to create more elegantly constructed, human-understandable, versions of these proofs. Just because something is true, doesn’t mean that we can’t continue to explore it, even if we don’t understand every aspect.

But even if we can’t do this—and we have truly bumped up against our constraints—our limits shouldn’t worry us too much. The non-understandability of science is coming, in certain places and small bits at a time. We’ve grasped the low-hanging fruit of understandability and explanatory elegance, and what’s left might be possible to be exploited, but not necessarily completely understood. That’s going to be tough to stomach, but the sooner we accept this the better we have a chance of allowing society to appreciate how far we’ve come and apply non-understandable truths to our technologies and creations.

As I’ve argued, if it’s our machines doing the discovering, we can still havenaches—we can take an often vicarious pride and joy in the success of our progeny. We made these machines, so their discoveries are at least partly due to humanity. And that’s exciting, as these programs of the future begin to uncover new truths about the universe.

They may just inject a bit more mystery into the world than we might have bargained for.

Really good long article on Acxiom in the NY Times. Here are some excerpts:

IT knows who you are. It knows where you live. It knows what you do.

It peers deeper into American life than the F.B.I. or the I.R.S., or those prying digital eyes at Facebook and Google. If you are an American adult, the odds are that it knows things like your age, race, sex, weight, height, marital status, education level, politics, buying habits, household health worries, vacation dreams — and on and on.

Right now in Conway, Ark., north of Little Rock, more than 23,000 computer servers are collecting, collating and analyzing consumer data for a company that, unlike Silicon Valley’s marquee names, rarely makes headlines. It’s called the Acxiom Corporation, and it’s the quiet giant of a multibillion-dollar industry known as database marketing.

Few consumers have ever heard of Acxiom. But analysts say it has amassed the world’s largest commercial database on consumers — and that it wants to know much, much more. Its servers process more than 50 trillion data “transactions” a year. Company executives have said its database contains information about 500 million active consumers worldwide, with about 1,500 data points per person. That includes a majority of adults in the United States.

Such large-scale data mining and analytics — based on information available in public records, consumer surveys and the like — are perfectly legal. Acxiom’s customers have included big banks like Wells Fargo and HSBC, investment services like E*Trade, automakers like Toyota and Ford, department stores like Macy’s — just about any major company looking for insight into its customers.

For Acxiom, based in Little Rock, the setup is lucrative. It posted profit of $77.26 million in its latest fiscal year, on sales of $1.13 billion.

But such profits carry a cost for consumers. Federal authorities say current laws may not be equipped to handle the rapid expansion of an industry whose players often collect and sell sensitive financial and health information yet are nearly invisible to the public. In essence, it’s as if the ore of our data-driven lives were being mined, refined and sold to the highest bidder, usually without our knowledge — by companies that most people rarely even know exist.

…

In a fast-changing digital economy, Acxiom is developing even more advanced techniques to mine and refine data. It has recruited talent from Microsoft, Google, Amazon.com and Myspace and is using a powerful, multiplatform approach to predicting consumer behavior that could raise its standing among investors and clients.

Of course, digital marketers already customize pitches to users, based on their past activities. Just think of “cookies,” bits of computer code placed on browsers to keep track of online activity. But Acxiom, analysts say, is pursuing far more comprehensive techniques in an effort to influence consumer decisions. It is integrating what it knows about our offline, online and even mobile selves, creating in-depth behavior portraits in pixilated detail. Its executives have called this approach a “360-degree view” on consumers.

“There’s a lot of players in the digital space trying the same thing,” says Mark Zgutowicz, a Piper Jaffray analyst. “But Acxiom’s advantage is they have a database of offline information that they have been collecting for 40 years and can leverage that expertise in the digital world.”

Yet some prominent privacy advocates worry that such techniques could lead to a new era of consumer profiling.

Jeffrey Chester, executive director of the Center for Digital Democracy, a nonprofit group in Washington, says: “It is Big Brother in Arkansas.”

SCOTT HUGHES, an up-and-coming small-business owner and Facebook denizen, is Acxiom’s ideal consumer. Indeed, it created him.

Mr. Hughes is a fictional character who appeared in an Acxiom investor presentation in 2010. A frequent shopper, he was designed to show the power of Acxiom’s multichannel approach.

In the presentation, he logs on to Facebook and sees that his friend Ella has just become a fan of Bryce Computers, an imaginary electronics retailer and Acxiom client. Ella’s update prompts Mr. Hughes to check out Bryce’s fan page and do some digital window-shopping for a fast inkjet printer.

Such browsing seems innocuous — hardly data mining. But it cues an Acxiom system designed to recognize consumers, remember their actions, classify their behaviors and influence them with tailored marketing.

When Mr. Hughes follows a link to Bryce’s retail site, for example, the system recognizes him from his Facebook activity and shows him a printer to match his interest. He registers on the site, but doesn’t buy the printer right away, so the system tracks him online. Lo and behold, the next morning, while he scans baseball news on ESPN.com, an ad for the printer pops up again.

That evening, he returns to the Bryce site where, the presentation says, “he is instantly recognized” as having registered. It then offers a sweeter deal: a $10 rebate and free shipping.

It’s not a random offer. Acxiom has its own classification system, PersonicX, which assigns consumers to one of 70 detailed socioeconomic clusters and markets to them accordingly. In this situation, it pegs Mr. Hughes as a “savvy single” — meaning he’s in a cluster of mobile, upper-middle-class people who do their banking online, attend pro sports events, are sensitive to prices — and respond to free-shipping offers.

Correctly typecast, Mr. Hughes buys the printer.

But the multichannel system of Acxiom and its online partners is just revving up. Later, it sends him coupons for ink and paper, to be redeemed via his cellphone, and a personalized snail-mail postcard suggesting that he donate his old printer to a nearby school.

Analysts say companies design these sophisticated ecosystems to prompt consumers to volunteer enough personal data — like their names, e-mail addresses and mobile numbers — so that marketers can offer them customized appeals any time, anywhere.

Still, there is a fine line between customization and stalking. While many people welcome the convenience of personalized offers, others may see the surveillance engines behind them as intrusive or even manipulative.

…

Today, Acxiom maintains its own database on about 190 million individuals and 126 million households in the United States. Separately, it manages customer databases for or works with 47 of the Fortune 100 companies. It also worked with the government after the September 2001 terrorist attacks, providing information about 11 of the 19 hijackers.

…

ACXIOM’S Consumer Data Products Catalog offers hundreds of details — called “elements” — that corporate clients can buy about individuals or households, to augment their own marketing databases. Companies can buy data to pinpoint households that are concerned, say, about allergies, diabetes or “senior needs.” Also for sale is information on sizes of home loans and household incomes.

Clients generally buy this data because they want to hold on to their best customers or find new ones — or both.

A bank that wants to sell its best customers additional services, for example, might buy details about those customers’ social media, Web and mobile habits to identify more efficient ways to market to them. Or, says Mr. Frankland at Forrester, a sporting goods chain whose best customers are 25- to 34-year-old men living near mountains or beaches could buy a list of a million other people with the same characteristics. The retailer could hire Acxiom, he says, to manage a campaign aimed at that new group, testing how factors like consumers’ locations or sports preferences affect responses.

But the catalog also offers delicate information that has set off alarm bells among some privacy advocates, who worry about the potential for misuse by third parties that could take aim at vulnerable groups. Such information includes consumers’ interests — derived, the catalog says, “from actual purchases and self-reported surveys” — like “Christian families,” “Dieting/Weight Loss,” “Gaming-Casino,” “Money Seekers” and “Smoking/Tobacco.” Acxiom also sells data about an individual’s race, ethnicity and country of origin. “Our Race model,” the catalog says, “provides information on the major racial category: Caucasians, Hispanics, African-Americans, or Asians.” Competing companies sell similar data.

Acxiom’s data about race or ethnicity is “used for engaging those communities for marketing purposes,” said Ms. Barrett Glasgow, the privacy officer, in an e-mail response to questions.

There may be a legitimate commercial need for some businesses, like ethnic restaurants, to know the race or ethnicity of consumers, says Joel R. Reidenberg, a privacy expert and a professor at the Fordham Law School.

“At the same time, this is ethnic profiling,” he says. “The people on this list, they are being sold based on their ethnic stereotypes. There is a very strong citizen’s right to have a veto over the commodification of their profile.”

He says the sale of such data is troubling because race coding may be incorrect. And even if a data broker has correct information, a person may not want to be marketed to based on race.

“DO you really know your customers?” Acxiom asks in marketing materials for its shopper recognition system, a program that uses ZIP codes to help retailers confirm consumers’ identities — without asking their permission.

“Simply asking for name and address information poses many challenges: transcription errors, increased checkout time and, worse yet, losing customers who feel that you’re invading their privacy,” Acxiom’s fact sheet explains. In its system, a store clerk need only “capture the shopper’s name from a check or third-party credit card at the point of sale and then ask for the shopper’s ZIP code or telephone number.” With that data Acxiom can identify shoppers within a 10 percent margin of error, it says, enabling stores to reward their best customers with special offers. Other companies offer similar services…

Will the quants leave Wall Street for big data opportunities?

The Internet’s been around since 1969, the web since 1990. I was one of the first 5,000 people on the Internet, back in 1971, when they were giving accounts to students at schools (like UCLA, Harvard, MIT, CMU, Stanford) involved in developing the network that linked the 69 (yep, count ‘em, 69) sites on the Net. ARPA, the DoD Advanced Projects Research Agency, which was paying for the whole thing, also had a few sites in that bunch of 69.  Those mostly had design specs for the Net, which everyone on it already had. There just wasn’t much traffic.

The reason they gave accounts to lowly undergrads was to generate some traffic to see if the damned thing worked. There wasn’t much to do besides pass around the UNIX joke file and peruse the generally sparse digital archives offered. But there were lots of undergrads with email, and the UNIX jokes never die, so it worked.

But nobody, anywhere, ever seemed to care even a fempto-iota what was in your emails or which sites you visited. This lack of interest continued during the early days of the consumer Internet. One of 1993’s most popular cartoons, by the New Yorker’s Peter Steiner (which would be called a meme now), showed two mutts in front of an old style CRT. One says to the other “On the Internet, nobody knows you’re a dog.”

Now, not only do they know you’re a dog, they know your size, preference in kibble, chew toys, and if you have fleas. It may sound small, but it’s in the Big Data-verse. For a partial list of the Big Data they have about you, click here.

There are so many kinds of “Big Data” people talk about that it’s hard to tell the players without a program. Lots of us, including me, are trying to figure out the “Big Data Landscape.” My fellow Forbes contributor Dave Feinleib has published a handy map. I visited many of the websites, and this is a vast technological territory.

My last post (“Speed Is Making The Market Dangerous. Nascar Shows How We Can Make It Safer.“) was about the kind of big data produced in financial markets. People are interested in that because of the perceived value in it. But if we measure by entrepreneurial energy, especially in Silicon Valley, there is more perceived value in the new flavor of Big Data than the old.

This idea was particularly vividly stated to me by George John, the CEO of Rocket Fuel, a fast-growing Big Data science firm in the Valley. George, and his lead quant jock, Mike Benisch, both have quant investing cred, having worked on Wall Street, and read “Nerds on Wall Street,” and they invited me over for a visit.

George’s insight is that “Big Data is the New Quant Investing,” and the more I think about it, as a longtime quant investor, the more I think George is right. Many of the ideas from quant investing make sense in this context; histories are huge, and experimentation is easy. There’s an underlying behavioral model, plus, you know your counter-parties.  The large volume and variety of data allows use of new “data voracious” statistical and machine learning methods that, in finance, are useful for high-frequency trading, but are worthless on daily or monthly market data. It’s words as well as numbers, so natural language tools can work along with numerical calculation.

A jumbo bowl of quant chow for sure.