collective unconscious of the smarter planet

A really interesting take on technology and the complexity that characterizes our modern predicament.

You just went to the Google home page.

Simple, isn’t it?

What just actually happened?

Well, when you know a bit of about how browsers work, it’s not quite that simple. You’ve just put into play HTTP, HTML, CSS, ECMAscript, and more. Those are actually such incredibly complex technologies that they’ll make any engineer dizzy if they think about them too much, and such that no single company can deal with that entire complexity.

Let’s simplify.

You just connected your computer to www.google.com.

Simple, isn’t it?

What just actually happened?

Well, when you know a bit about how networks work, it’s not quite that simple. You’ve just put into play DNS, TCP, UDP, IP, Wifi, Ethernet, DOCSIS, OC, SONET, and more. Those are actually such incredibly complex technologies that they’ll make any engineer dizzy if they think about them too much, and such that no single company can deal with that entire complexity.

Let’s simplify.

You just typed www.google.com in the location bar of your browser.

Simple, isn’t it?

What just actually happened?

Well, when you know a bit about how operating systems work, it’s not quite that simple. You’ve just put into play a kernel, a USB host stack, an input dispatcher, an event handler, a font hinter, a sub-pixel rasterizer, a windowing system, a graphics driver, and more, all of those written in high-level languages that get processed by compilers, linkers, optimizers, interpreters, and more. Those are actually such incredibly complex technologies that they’ll make any engineer dizzy if they think about them too much, and such that no single company can deal with that entire complexity.

Let’s simplify.

You just pressed a key on your keyboard.

Simple, isn’t it?

What just actually happened?

Well, when you know about bit about how input peripherals work, it’s not quite that simple. You’ve just put into play a power regulator, a debouncer, an input multiplexer, a USB device stack, a USB hub stack, all of that implemented in a single chip. That chip is built around thinly sliced wafers of highly purified single-crystal silicon ingot, doped with minute quantities of other atoms that are blasted into the crystal structure, interconnected with multiple layers of aluminum or copper, that are deposited according to patterns of high-energy ultraviolet light that are focused to a precision of a fraction of a micron, connected to the outside world via thin gold wires, all inside a packaging made of a dimensionally and thermally stable resin. The doping patterns and the interconnects implement transistors, which are grouped together to create logic gates. In some parts of the chip, logic gates are combined to create arithmetic and bitwise functions, which are combined to create an ALU. In another part of the chip, logic gates are combined into bistable loops, which are lined up into rows, which are combined with selectors to create a register bank. In another part of the chip, logic gates are combined into bus controllers and instruction decoders and microcode to create an execution scheduler. In another part of the chip, they’re combined into address and data multiplexers and timing circuitry to create a memory controller. There’s even more. Those are actually such incredibly complex technologies that they’ll make any engineer dizzy if they think about them too much, and such that no single company can deal with that entire complexity.

Can we simplify further?

In fact, very scarily, no, we can’t. We can barely comprehend the complexity of a single chip in a computer keyboard, and yet there’s no simpler level. The next step takes us to the software that is used to design the chip’s logic, and that software itself has a level of complexity that requires to go back to the top of the loop.

Today’s computers are so complex that they can only be designed and manufactured with slightly less complex computers. In turn the computers used for the design and manufacture are so complex that they themselves can only be designed and manufactured with slightly less complex computers. You’d have to go through many such loops to get back to a level that could possibly be re-built from scratch.

Once you start to understand how our modern devices work and how they’re created, it’s impossible to not be dizzy about the depth of everything that’s involved, and to not be in awe about the fact that they work at all, when Murphy’s law says that they simply shouldn’t possibly work.

For non-technologists, this is all a black box. That is a great success of technology: all those layers of complexity are entirely hidden and people can use them without even knowing that they exist at all. That is the reason why many people can find computers so frustrating to use: there are so many things that can possibly go wrong that some of them inevitably will, but the complexity goes so deep that it’s impossible for most users to be able to do anything about any error.

That is also why it’s so hard for technologists and non-technologists to communicate together: technologists know too much about too many layers and non-technologists know too little about too few layers to be able to establish effective direct communication. The gap is so large that it’s not even possible any more to have a single person be an intermediate between those two groups, and that’s why e.g. we end up with those convoluted technical support call centers and their multiple tiers. Without such deep support structures, you end up with the frustrating situation that we see when end users have access to a bug database that is directly used by engineers: neither the end users nor the engineers get the information that they need to accomplish their goals.

That is why the mainstream press and the general population has talked so much about Steve Jobs’ death and comparatively so little about Dennis Ritchie’s: Steve’s influence was at a layer that most people could see, while Dennis’ was much deeper. On the one hand, I can imagine where the computing world would be without the work that Jobs did and the people he inspired: probably a bit less shiny, a bit more beige, a bit more square. Deep inside, though, our devices would still work the same way and do the same things. On the other hand, I literally can’t imagine where the computing world would be without the work that Ritchie did and the people he inspired. By the mid 80s, Ritchie’s influence had taken over, and even back then very little remained of the pre-Ritchie world.

Finally, last but not least, that is why our patent system is broken: technology has done such an amazing job at hiding its complexity that the people regulating and running the patent system are barely even aware of the complexity of what they’re regulating and running. That’s the ultimate bikeshedding: just like the proverbial discussions in the town hall about a nuclear power plant end up being about the paint color for the plant’s bike shed, the patent discussions about modern computing systems end up being about screen sizes and icon ordering, because in both cases those are the only aspect that the people involved in the discussion are capable of discussing, even though they are irrelevant to the actual function of the overall system being discussed.

Some more interesting thinking on the “end of insight” from Steven Strogatz at the Edge.org -

I worry that insight is becoming impossible, at least at the frontiers of mathematics. Even when we’re able to figure out what’s true or false, we’re less and less able to understand why.

An argument along these lines was recently given by Brian Davies in the “Notices of the American Mathematical Society”. He mentions, for example, that the four-color map theorem in topology was proven in 1976 with the help of computers, which exhaustively checked a huge but finite number of possibilities. No human mathematician could ever verify all the intermediate steps in this brutal proof, and even if someone claimed to, should we trust them? To this day, no one has come up with a more elegant, insightful proof. So we’re left in the unsettling position of knowing that the four-color theorem is true but still not knowing why.

Similarly important but unsatisfying proofs have appeared in group theory (in the classification of finite simple groups, roughly akin to the periodic table for chemical elements) and in geometry (in the problem of how to pack spheres so that they fill space most efficiently, a puzzle that goes back to Kepler in the 1500’s and that arises today in coding theory for telecommunications).

In my own field of complex systems theory, Stephen Wolfram has emphasized that there are simple computer programs, known as cellular automata, whose dynamics can be so inscrutable that there’s no way to predict how they’ll behave; the best you can do is simulate them on the computer, sit back, and watch how they unfold. Observation replaces insight. Mathematics becomes a spectator sport.

If this is happening in mathematics, the supposed pinnacle of human reasoning, it seems likely to afflict us in science too, first in physics and later in biology and the social sciences (where we’re not even sure what’s true, let alone why).

When the End of Insight comes, the nature of explanation in science will change forever. We’ll be stuck in an age of authoritarianism, except it’ll no longer be coming from politics or religious dogma, but from science itself.

Will the quants leave Wall Street for big data opportunities?

The Internet’s been around since 1969, the web since 1990. I was one of the first 5,000 people on the Internet, back in 1971, when they were giving accounts to students at schools (like UCLA, Harvard, MIT, CMU, Stanford) involved in developing the network that linked the 69 (yep, count ‘em, 69) sites on the Net. ARPA, the DoD Advanced Projects Research Agency, which was paying for the whole thing, also had a few sites in that bunch of 69.  Those mostly had design specs for the Net, which everyone on it already had. There just wasn’t much traffic.

The reason they gave accounts to lowly undergrads was to generate some traffic to see if the damned thing worked. There wasn’t much to do besides pass around the UNIX joke file and peruse the generally sparse digital archives offered. But there were lots of undergrads with email, and the UNIX jokes never die, so it worked.

But nobody, anywhere, ever seemed to care even a fempto-iota what was in your emails or which sites you visited. This lack of interest continued during the early days of the consumer Internet. One of 1993’s most popular cartoons, by the New Yorker’s Peter Steiner (which would be called a meme now), showed two mutts in front of an old style CRT. One says to the other “On the Internet, nobody knows you’re a dog.”

Now, not only do they know you’re a dog, they know your size, preference in kibble, chew toys, and if you have fleas. It may sound small, but it’s in the Big Data-verse. For a partial list of the Big Data they have about you, click here.

There are so many kinds of “Big Data” people talk about that it’s hard to tell the players without a program. Lots of us, including me, are trying to figure out the “Big Data Landscape.” My fellow Forbes contributor Dave Feinleib has published a handy map. I visited many of the websites, and this is a vast technological territory.

My last post (“Speed Is Making The Market Dangerous. Nascar Shows How We Can Make It Safer.“) was about the kind of big data produced in financial markets. People are interested in that because of the perceived value in it. But if we measure by entrepreneurial energy, especially in Silicon Valley, there is more perceived value in the new flavor of Big Data than the old.

This idea was particularly vividly stated to me by George John, the CEO of Rocket Fuel, a fast-growing Big Data science firm in the Valley. George, and his lead quant jock, Mike Benisch, both have quant investing cred, having worked on Wall Street, and read “Nerds on Wall Street,” and they invited me over for a visit.

George’s insight is that “Big Data is the New Quant Investing,” and the more I think about it, as a longtime quant investor, the more I think George is right. Many of the ideas from quant investing make sense in this context; histories are huge, and experimentation is easy. There’s an underlying behavioral model, plus, you know your counter-parties.  The large volume and variety of data allows use of new “data voracious” statistical and machine learning methods that, in finance, are useful for high-frequency trading, but are worthless on daily or monthly market data. It’s words as well as numbers, so natural language tools can work along with numerical calculation.

A jumbo bowl of quant chow for sure.

Some IoT musings from the Guardian.

Sometimes trying to predict the digital revolution seems very much like that old cliche of waiting for a bus. You know, you spend 12 years waiting for mobile to come along and then social and cloud arrive at the same time. That perfect, ahem, storm of connectivity – combined with the rapid adoption of smartphones – is changing society and shaking up business faster than any of us can imagine.

So imagine something even more disruptive: the social, cloud and mobile connected identity of everyday objects, or the “internet of things” as it is often called. Just two years ago, the internet of things was widely framed by examples such as a washing machine telling its owner and the manufacturer when it needed a service. Or the fridge having a chat with Waitrose (other fine supermarkets are available) when you’re running low on milk.

Fast forward to today and most new cars can’t function without the help of embedded, connected computers. Take the S-class Mercedes. It has nearly as many embedded computers as an Airbus A380. But as connected and functional as these everyday objects may be, they are still inanimate, lacking a bit of identity you might say.

That is about to change – and quite rapidly if the tech lessons of the past few years are any indication. Take apps as an example, a multibillion dollar industry that didn’t exist four years ago. But more important than business turnover is the transformative effect apps have had on our lives and on the object – mainly the smartphone – they dwell on. Simply put, apps have supercharged phones by allowing them to offer social, personalised and relevant services to their owners. When we use apps on smartphones we’re telling telco and the app providers key details about us and our purchasing preferences, and those providers are able to shape products and offerings to match our individual needs.

But what about objects that aren’t smartphones? Say you’re a keen musician. What if your favourite electric guitar had an unique online identity that allowed you to connect to it using your smartphone? Not only could it tell you what type of strings to use and how to tune them but it could also recommend songs that match your playing tastes, local and global musicians online to jam with and, of course, product updates and add-ons from the guitar manufacturer.

That’s just one of the examples a new startup, Evrythng, cites when describing its vision of creating a digital identity for everyday objects and how it might transform our relationship with the products we buy. “We’re looking to solve a problem that manufacturers didn’t think there was a solution to. How do I get closer to my customers when I don’t know who most of them are?” says Andy Hobsbawm, Evrythng’s co-founder and chief marketing officer.

The social media revolution has opened new avenues for customers who crave information about the products they buy while providing manufacturers with direct relationships to those customers. Evrythng believes the product can be a key digital conduit between the customer and the company and so has started a “software as a service” business that creates what they call an active digital identity for any product or object. Each product, be it a can of beans or a power drill, can have a unique identifying tag like simple QR codes or NFC (near field communications – what your Oyster card uses, and in all smartphones made this year) tags digitally printed on to the label or packaging. When scanned or swiped the product uses your smartphone to instantly connect to the net, creating a unique profile just for you serving up personalised digital content and services to help you get the most out of it.”

Already, Evrythng is working with companies such as Diageo to create digital identities and, hence personalised online services, for customers.

At this point, it’s easy to drift into your worst Hal 2001: A Space Odyssey nightmare scenarios and envisage the creepy data map companies will soon be compiling once, say, our digitally enabled toothbrush is snooping on our daily brushing habits. But just like any technology, this retrofitted internet of things can only be as good or bad as the way we choose to use it.

So let’s imagine how objects with connected online identities can actually drive sustainability. Imagine a portfolio of household good products – your laundry detergent and your dishwater – communicating with you to give a personal record that can help reduce water and energy use. Or imagine medical devices like glucose monitors that come with dietary advice and medicines that provide online side-effect alerts and tests. Or wine and spirits bottles that provide not just terroir history and cocktail tips but also personalised healthy drinking advice.

Established peer-to-peer services like AirBnB and the US private car sharing/rental company Relay Rides already point to how connected objects can promote sustainability. In the case of Relay Rides, subscribers who need access to cars but don’t want to own a vehicle rent other people’s private cars on a journey-by-journey basis. Now spin that model forward to multiple shared ownership of a single vehicle equipped with a digital identity connected to all the owners. The vehicle becomes the hub of an online network that allows, for example, four different owners, to effectively share that one car. They can plan and monitor usage schedules, parking, service history as well as individually tailor radio stations, seat positions etc to their individual needs.

“The web of things has been coming for a long time and people have been talking about products having a presence online,” notes Andy Hobsbawm. “Now you’re entering a zone where the cost per unit of tagging a unit is becoming affordable for scale and where bandwidth is continuing to fall.” That zone will likely see this newest online “revolution” take its place alongside mobile, social and the cloud as a driver of technological change.

How we take advantage of that change is up to us.

Ford has a very interesting vision of the future. Very smarter planet.

The auto industry has long talked about “the connected car.” But Ford Motor chairman Bill Ford’s vision stretches far beyond simply allowing your vehicle to connect to the Internet for better directions or groovy Pandora tunes.

Instead, he sees the car of the future as a node on a giant network that helps optimize the driving experience for everyone.

For example, let’s say you’re driving home late at night. “Why should that traffic light be red, and you have to stop and burn fuel when nobody’s coming?” Ford told Fast Company in an exclusive interview at an event in Mountain View, Calif., Monday night to formally open the company’s new Silicon Valley research lab. “Why shouldn’t [the light] sense you coming, go to green, and let you go through?”

…

Ford envisions a proliferation of sensors within the vehicle itself and an open-source approach to software and hardware that allows third parties to use data generated by Ford vehicles to build new services that were never before possible. For example, a San Francisco company, Weather Underground, is already exploring how to use data collected from windshield wiper activity to create more fine-grained weather predictions. This vision, Bill Ford says, requires a “melding of the auto industry with the tech industry.” Hence the new lab. Ford Motor will use it as a home base to reach out to the tech community to find partners and collaborators. 

But the hub will also help smaller, cutting edge companies connect to the automotive behemoth. “If you’re a startup out here and you have a great idea, you [probably] don’t have a clue how to get a hold of someone in Dearborn,” Ford says.

Ford Motor already collaborates with Microsoft, Apple, and eBay on some of the company’s more near-term initiatives, especially infotainment ones, like making sure the company’s cars and Apple’s devices are compatible.

As for which partners will help enact the meta-congestion-battling strategy, Ford says the company doesn’t know who those will be yet. “We’re starting that conversation with a number of companies,” he says. “There’s not going to be one solution.”

Ford isn’t the only automotive company setting up shop in Silicon Valley. A GM lab moved in six years ago and has been working on projects like the Cadillac CUE infotainment interface. VW also has an electronics research lab here, exploring systems to start and stop cars automatically in traffic jams and monitor driver stress levels. 

For its part, Ford Motor has created OpenXC, a platform that allows third-party developers to access certain types of vehicle data. The first beta developer kits were shipped out to universities like Stanford, MIT, and the University of Michigan earlier this year.

Bill Ford says he can’t pinpoint a date when his congestion-battling vision will become a reality, but he suspects it will be sooner rather than later. “When we first started talking about this, we thought it was going to happen in 30 years,” he says. “Then it was 20. But now, it’s [happening at] the rate of technology. You see it out here. It’s very fast.”

Andrew Curry is a forecaster at the Futures Company. Here he writes about how the shape of today’s technology will reshape business - through the lens of Carlota Perez. How will organizations maintain their “edge” (or shape) in an increasingly permeable, fluid world?

New business models emerge late on

Looking at these two diagrams together, one of the striking features of Carlota Perez’ technology surges model is that new forms of business organisation and new business models tend to emerge clearly only in the second half of the surge, and later rather than earlier, although some pioneers are seen before this. Codification of these emerging ideas as management practices isn’t seen until the final decades of the surge. Looking back, for example, Alfred Sloan pioneered some of the main features of the 20th century assembly line business at General Motors in the 1930s, after the crash, but the theory that supported his practices wasn’t developed until the 1960s by writers and academics such as Alfred Chandler and Peter Drucker. In other words, we are only now beginning to see the development of the innovative business models which will characterise the digital technology age.

One of the important aspects of any new technology platform is that the shape of businesses adapt to the shape of the technology, and not the other way around. The steam engine emphasised the concentration of production, and the assembly line the importance of throughput. The successful businesses of those times absorbed these structural lessons, and adapted themselves to fit.

A radical challenge to the idea of the organisation

The core ideas at the heart of the next phase of digital technology are about:

• Pervasiveness
• Unbundling
• Inter-connection
• Reconfigurability

These four characteristics will define the shape of business over the next decade and beyond. Different businesses, and different sectors, will respond differently to these factors, depending, for example, on their value networks, the present mix of data and tangible output, and their wider external constraints. Strategy is always distinctive; every organisation has to find its own route through to a strategic future which is right for it, which matches its capabilities against a landscape shaped by changing social values, by evolving technologies, and by infrastructure, systems, and regulation.

Nonetheless, in all sectors, these characteristics represent a radical challenge to the idea of the organisation, which is at heart defined by boundaries (if it has no boundaries, it ceases to be an organisation). The big question for any organisation, commercial, public, or non-profit, is how it maintains its edges in a world where technology renders so much fluid; relations with customers, suppliers, stakeholders, and employees all start to melt into bytes. Indeed, the distinctions between the organisation and its customers and its suppliers start to blur. These are big strategic shifts which represent a deep challenge to prevailing business structures.

Scott Smith is a forecaster whom I’ve been bumping into again and again over the past couple of weeks’ web wanderings. Here’s a piece he semi-recently posted in response to McKinsey’s “second economy” report. Very interesting.

As a part of ongoing scanning, among the many sources I read at least weekly are job ads (there’s a foresight joke in here somewhere, but I’ll leave it for now). To me, the changing nature of work as described in the evolution of required duties and experience levels are an interesting leading indicator, at least of perception. What do companies think they will need, in what quantity, and based on what skills?

Granted, I’m looking at certain sectors that are the apparent leading edge of services, manufacturing and media, but several big shifts seem to be taking place in the way roles are described, and what skills and experience are required by employers to fill them. Some examples:

• Required education levels are getting higher. A BA or BS is becoming the equivalent of finishing secondary school. Master’s degrees are sought after more and more. Leading to…
• A jump in demand for specialized education. A general master’s level degree doesn’t seem to suffice—it has to be specialized in a niche area. Urban planning, design research, astrobiology, early childhood education. 
• Transdisciplinary capabilities. It’s not enough to have had a breadth of experience, but one needs to be able to call on both sides of the brain, and all areas of practice, all of the time. Graphic design? Check. Interactive design? Check. Project management? Check. Marketing and communication? Check. Research and insights? Check. On and on, a single department or division, rolled into one person. I suspect this is a sort of risk averse trend whereby many potential needs are crammed into one job, requiring not so much as a “T-shaped” individual as a fantasy firefighter/ninja who will cover the inadequacies of others when faced with an increasingly complex and chaotic environment.
• Analytical capabilities. This is the one that jumps out more and more. Problem solving isn’t sufficient. Being able to assess and analyze “insights,” what industry generally calls half-digested data today, is de rigueur. Increasingly, jobs appear to come with a data set, or a petabyte of data, attached. Be prepared to receive a data dump, and capable of telling a coherent story from it. Which means…
• The ability to weave a narrative. One must be able to always tell a story, craft a use case, layer up a persona, or imagine a segmentation. With increasing levels of disruption around them, organizations of all kinds seem to need to tell themselves a story. Who are we? Who do we serve? What is our backstory? What are our customers’/competitors’ motivations? 

The availability of data seems to be driving these latter two elements. As a recent McKinsey Quarterly article called “The Second Economy,” posits, our global economy is in the midst of a sort of molting—the physical economy that grew from the Industrial Revolution has spawned a second economy that is largely invisible, and a product of IT, a data economy. “This vast global digital network that is sensing, ‘computing,’ and reacting appropriately—is starting to constitute a neural layer for the economy.” writes W. Brian Arthur in this piece. “The second economy constitutes a neural layer for the physical economy.” 

Arthur goes on to say that this second economy doesn’t require the jobs the first, physical economy did. In his view, the intelligence we build into the system—the algorithms—do a lot of the lifting individuals used to. A clerk isn’t required to draft a report, move a document along, and make a decision. If you apply for a credit card today, or try to book a one-way flight with cash, somewhere an algorithm is sniffing you, matching your behavior to a set pattern, and rendering judgement. Others are coming to a similar conclusion, with much handwringing about how technology is displacing jobs. 

But the anecdotal evidence of scanning the job postings suggest otherwise. Our data sets are growing larger as the “aspen root system,” as Arthur calls the sensing tendrils of this second economy, grows more extensive. We can collect more, so we have more not to simply analyze, but contextualize and convert to meaningful narrative. This need to see the patterns in the noise itself may be the catalyst for an even higher (or deeper) level economy—a third economy of sensemaking. Collecting and warehousing massive amounts of data is simply an exercise in hording if we can’t see, contextualize, and use the patterns in the noise. (This article on the growing genetic data glut is a great example.)

This evolution will requre even greater, and more meaningful, analytical ability from workers of the near future. If sensing and collection is ubiquitous, then our ability to be analytical polyglots must be as well. Arthur is correct in that fewer people will be required to push a button or shift a document, but we are in serious trouble if we don’t grow the capability to make sense of what we “know” at a pace relative to the speed with which we can collect. Algorithms can only take us so far, which we are seeing even now.