IFTF's Future Now

Deconstructing turbulence

via FutureFeeder-

Richard Feynman once called turbulence the most important unsolved problem in classical physics - so it's notable that MIT researchers are making some headway in understanding its underlying structure.  They tracked the motion of minuscule particles in a turbulent fluid using lasers, high definition cameras, and software to follow the particles over small increments of time.

MIT researchers report that they have visualized for the first time a convoluted tangle underlying turbulence. This work may ultimately help engineers design better planes, cars, submarines and engines.
...
The researchers identified a complex network of two types of curves formed by two distinct groups of particles. The first type of curve, which the researchers colored red, attracts other fluid particles. At the same time, the second type, colored blue, repels other fluid particles. Both sets of curves evolve with the flow.
...
The MIT researchers call their discovery the "Lagrangian skeleton" of turbulence because their particle-based approach is motivated by the work of 19th-century mathematician Joseph-Louis Lagrange. "Lagrange developed mathematical tools still used today for calculating mechanical and fluid motion," said Peacock.

Swarm Agent Modeling Wiki

Swarm is an agent modeling system originally developed at the Santa Fe Institute. Its now run by a public consortium, the not-for-profit Swarm Development Group. They have just re-designed their wiki site, which is rich in content about Agent Modeling and the Swarm system.

Simulating Realities

All business is a forecast, and all forecasts are a kind of simulation of the future. In recent years there has been increasing interest in how to do simulation and how simulation can get you a leg up on what the future will be. James Swain, a former professor and advisor of mine, publishes an overview article on simulation capabilities in ORMS/Today. Included is a survey of dozens of packages, their capabilities and uses. This is more about packages than it is about vendors, but many package makers will do work for you or point you to those that can.

Cellular Phones and Cellular Automata

A colleague recently sent me a pointer to WolframTones, a clever site developed by Wolfram Research that uses cellular automata concepts to generate musical compositions that can be used as cellphone ringtones as a new kind of music. We have followed Wolfram's work for some time. Most notable is his book A New Kind of Science, which seeks to replace standard science with lots of cellular automata. Cellular automata are totally unrelated to cellular phones. The invention of CA's are often attributed to Manhattan Project physicist Stanislaw Ulam. In the 80s they were reapplied by Los Alamos Labs scientist Christopher Langton, to create a discipline called artificial life. We met Langton in the 90s as he pushed the modeling capability of AL, which then morphed into the modeling called Agent-Based Modeling, or ABM. That technology is being used to build corporate simulation models. The kind of models that Wolfram is proposing to replace conventional science are very different from those used for ABMs, but the general concept is the same: individual agents interacting in relatively simple ways generate complex behavior. Amazing that Ulam's phsyics would end up in cell phone tones and corporate models.

Making Stick Figures Walk

A clever animation of a Genetic Algorithm process (GA), instructive for understanding how difficult problems can be solved with this method. A GA is designed to solve problems that are too difficult to solve using mathematical optimization .. they search through lots of options by using a method inspired by genetics: using population, mutation and crossover. In this simple example a stick figure is 'taught' to walk using a genetic method. Via the Illinois GA Lab blog. Also, GAs and their application.

Considering the Nature of Chatter

I recently reviewed Patrick Keefe’s new book Chatter .... I won’t describe that book here, but I did post a critical review of it elsewhere. The book did make me think about the term and how it has come to be used to describe the activity of terror groups. How do we think about the changes in such signal intelligence over time, determine when changes are significant, and consider the causal effects that influence the changes?

Security agencies think about chatter in terms of terror level and how we should react, but many other industries are starting to have access to data streams that can be analyzed in terms of signals and their changes over time. Point-of-sale data sources, call centers, increasing use of sensors, video-image analysis and RFID tagging are producing an avalanche of rich data sources, which can be studied in real time. Even the notion of ‘real-time’ has started to evolve. Where in the past analysis of commercial data may have been performed monthly or quarterly, now transactions can be tapped into to provide data directly from the cash register or RFID scanner to use much smaller time slices.

So how do we know when an observed measure has changed? In the 1920’s Walter Shewhart of Bell Telephone Labs developed the Control Chart, a statistical means of observing such streams and determining the significance of changes. Later Deming was to lionize them for understanding the operation of processes. These methods are starting to re-emerge as a means for understanding data streams. These are powerful techniques, but also rely on statistical assumptions that may not hold for new forms of data.

How do we determine what has caused changes in a data stream? Intelligence communities have to worry that terrorists may be manipulating the chatter level. In the same way we need to understand what causal elements can be influencing any set of signals we are observing.

Also important is the idea that processes are not just all about a single measure, they are often dependent on multiple streams of information, with varying time dimensions, accuracy and availability. Newly purposed Bayesian techniques are now being used to model and analyze such multiple data-stream systems.

Companies are emerging to address these problems. Here is a recent article on some vendor activity in this area. Notable is the University of Illinois startup Riverglass, which "... has developed software that merges data from multiple, disparate data streams--including unstructured text and numeric data--and applies real-time data modeling and analysis techniques to those streams. The goal is to detect patterns in the data to identify potential investment risks and opportunities ..."

Chatter is now all around us, and as we continue to develop new means to generate data though sensor networks, we will need to understand what it all means, and link it to decisions we can ultimately make. There is much work to be done.

Social Modeling: JASSS Journal

I neglected to mention it in my previous note on agent-based simulation, but an excellent source of information on social simulation, agent-based and otherwise, is the online Journal of Artificial Societies and Social Simulation. I have followed it for years. To get a good feeling for how this area is progressing, its useful to browse some of their recent titles. It shows current progress and also the challenge of using social simulation to understand systems of people.

Agents of Change: Social Modeling

A good overview article on Agent-Based Modeling (ABM) philosophy, techniques and applications, from the February ORMS Today, by Douglas Samuelson. He makes the case that this technique will transform social science because of the natural linkage between social and agent entities and their inclusion of behavioral changes. He also links ABMs to classical simulation techniques and Systems Dynamics. Some useful further links.

A Wisdom of Models?

I recently presented at the Informs Roundtable Meeting on Data Mining. I will post several short individual items on useful learnings.

Data mining is about finding useful patterns in data, frequently from data that was not gathered explicitly to solve a current problem at hand. Modeling methods for data mining include neural nets, regression, genetic algorithms and classification approaches. Dr John Elder of Elder Research promotes the idea of trying a number of models and then combining the results in an ensemble answer, the simplest approach would be to average the results ... a kind of Wisdom of Models, see his paper on why an ensemble of models can do better.

Genetic Blogging

The comments on my recent post on the Value of Genetic Design pointed me to an interesting blog on Genetic Methods, see: IlliGAL Blogging: Life, Liberty, and the Pursuit of Genetic Algorithms. David Goldberg and others are contributing, contains useful technical details and excellent application examples. I see also that they are now pointing back to us as well, welcome.

Physics Model for Predicting Sales

We were pointed to this MIT Tech Review article on predicting book sales, using physics models. It uses what they call an "Epidemics-Type Aftershock Sequence model" to track how information about a book travels through social networks. Here is the UCLA Press release on the same topic with more technical detail. The researcher is Didier Sornette of UCLA. Could be used for prediction outside of books ... the basic premise is intriguing:

...Information about a book travels through the network of potential buyers in two possible fashions: exogenous and endogenous. Exogenous shocks come from sources outside the system they affect, like billboards or newspaper articles; endogenous shocks are made up of very small exogenous shocks that happen in a coordinated fashion, like word-of-mouth recommendations. The model predicts how sales will decline after they peak according to how the peak occurred. The decline after an exogenous shock is fairly steep, while the decay after an endogenous shock is more gradual. The model was 84 percent correct in the researchers tests ...

Cleansing Data and Benford's Law

Cleansing data is the process by which datasets are examined, usually by statistical means, to determine if they contain errors, in preparation for data mining or analysis. Lots of things can cause errors, even scanning items on a market register is less than perfect. The classic case is when a cashier rings up multiple flavors of an item by scanning one item, then using the register to mutiply the single items by the count. The count turns out to be correct, but the inventory by type is incorrect.

One simple means of data cleansing is to look at some value, then see if they are more than some range from the mean of all values. In other words, it is classifed as an 'outlier', whose cause we do not know. A simple technique, though sometimes also dangerous, since some outliers can contain useful information about a dataset.

Cleansing then is all about pattern. But the pattern has to be chose carefully.

Back in school we learned about one such pattern analysis, called Benford's Law. A recent article in Intelligent Enterprise reminded me of the technique, which looks at the pattern of digits 0-9 in numbers that are in observations to determine if they are natural or artificial. That pattern of occurrance of digits is different from what what you might guess. This technique has been used indicate fraud, or systematic errors in sensor data.

The Wikipedia also has a well-done article on Benford's Law.

Firing Your Customer

This has been in the news of late, with the buying season at hand ... The books/articles on classifying your customers via data mining make mention of the fact that you might want to 'fire' your lowest profit (or net loss generating) customers ... but this is the first case I have seen of a retail enterprise actually doing it. Almost all of the loyalty/club cards attempt to reward on the high end in some way, and just ignore the bottom third. I have to applaud Best Buy for trying this, they are the obvious ones to do it since there are certain consumer behaviors, especially with regard to returns, that are quite costly to them. Though you have to wonder a bit if on a purely psych basis, isnt rewarding good behavior still much better than punishing bad behavior? But maybe its not the same as training a puppy?

Previous Slashdot article brought to my attention (excerpt):

"Best Buy is one of the retailers that has now decided that the customer is not always right. Best Buy consultant Larry Selden has identified "demon customers" like those who file for a rebate then return the item. OK, I get that one .... Other categories like customers who only buy during sales are more interesting. Best Buy declined comment on how they are dealing with those customers. Some stores have actually "fired" customers. Welcome to the end result of all that customer information data mining."

referenced article

Bayesian Truth Serum?

Consumers are often asked their opinions regarding products and their intention to buy. That information is used to guide research, product development and merchandising. It is well known that the truth is not always told. Is there a way to extract truth from groups of questions? Below an interesting area of research, using Bayesian statistics. Also here is another item on research in truth detection.

Mathematical "truth serum" promotes honesty , NewScientist.com news service
Few questions come with a clear-cut "right" answer, a solution as clear as the difference between black and white. Speculating on the future, making inferences about the past, and judging the present all involve a seemingly endless palette of greys.

But eliciting truthful responses from people - who give subjective answers - is crucial to the surveys and expert analyses that determine government and financial policies. Finding out if someone is answering a question truthfully is a murky process, often inaccessible to the questioner - sometimes even the person answering cannot be sure.

Now, Drazen Prelec, a psychologist at the Massachusetts Institute of Technology in Cambridge, US, has devised a scoring system, or �Bayesian truth serum� to encourage people to divulge their honest opinions ...

Examining the Hockey Stick

Last year I attended a talk by a Harvard geologist who discussed global warming as a complex process. One of the exhibits shown was some form of the graph here. This is the well known hockey-stick plot, showing rapid changes in temperature in the last 200 years. This is a remarkable graph, very convincing and impressive, you cannot come away without the belief that something very radical has happened. The origin of the data is usually only touched upon. Here is a case where you are basing your conclusions on the validity of the visual. Visuals are convincing and powerful, but always deserve close examination. I wanted to and should have asked, where did all the data come from? Were any transformations of the data performed? But it was an informal presentation, not a dissertation defense, and asking lots of very detailed questions would be considered off track ... and this was a professor from Harvard.

Yet, applying the principle of Occam�s Razor (that the simplest explanation of some observation is often best) this deserves further examination. There is something odd going on the graph in the last 200 years, the temperatures are increasing and their variability is decreasing. Very radically. One possible explanation is the industrial revolution�s use of fossil fuels, causing a greenhouse effect. A simpler explanation is that something about the data or it use has changed. The influence of the first cause is possible, but hard to prove. The second explanation is much easier to check.

Someone has finally checked the simpler hypothesis and found that some of the transformations used to create the hockey-stick are incorrect. Some side-effects of the method, Principal Components Analysis, could be causing the dramatic changes in the plot. See a recent article in the MIT Tech Review for more overview details, and also in the authors of the finding, Stephen McIntyre and Ross McKitrick's web site for technical detail.

Unfortunately this topic has become so politicized of late that it�s difficult to separate the math from the politics. This finding, if confirmed, does NOT mean that global warming is not happening nor imply that human beings have not caused it. It only means that the hockey stick model may be fundamentally incorrect. Unfortunately, too, the first things that will be checked will be the researchers funding and political affiliations, rather than their analyses.

This also has broader implications, in the area of commercial modeling, whenever preconceived opinion and science meet. In my career have seen a number of examples where internal politics have trumped science, often to unfortunate results. So there is more learning here than just the hockey stick example.

See also the comments section of the Tech review article for more debate about the results., and links to the development of the original model.

I plan to take a semi-serious dive, if anyone has taken a closer look at the math, would appreciate comments to help me navigate the details.

Is it How Many People You Know? ... Plus Reflections on Causality

An article in the NYTimes Business section by Nobel prize winning economist Kenneth Arrow got quite a bit of attention last week. It suggested that differences in salary and net worth could, in part, be explained by how many personal network connections you have:

... In fact, in a recent working paper, Professor Arrow and Mr. Borzekowski conclude that a worker's net worth can have a lot to do with the worker's network. In their model - and it is just a model, not based on empirical data - a person with one corporate connection would be expected to earn $19,570. By contrast, a person with links to five companies would be expected to earn $30,410. Ultimately, they conclude, "the difference in the number of ties can induce substantial inequality and can explain 15-20 percent of the unexplained variation in wages." ...

When I see analyses like this, I think, can we show that this is causal? Alternatively you could suggest that differences in net worth influence the number of connections you have. This whole causality thing (what caused what?) is treated very differently by economists versus statisticians. For another view see a splendid presentation by Judea Perl, regards his 2000 book "Causality : Models, Reasoning, and Inference". Here is Arrow's working paper on the network problem, the authors describe their construction of a model, running simulations and calibration of the results. This is an example of theoretical economics, as practiced at Stanford.

Adaptive Decision Making in the Global Economy

Now that we have the ability to put sensors in many kinds of systems, how do we get benefit from that expense? In a very general sense, this should allow us to adapt to situations faster and more accurately than ever before. A key aspect to this is the fact that adapting needs to be continual, so optimizing on say readings of sensors some time ago may not be optimal now. This contrasts with classical operations research approaches, which gather aggregated data, optimize, and then seek to apply the results to a changed and changing system. So, OK, we can place sensors everywhere, but how do we use that data? How do we address decision making under this new wealth of data?

Here is an innovative paper that addresses that issue by collaborator Tom Gibbs, Worldwide Director of Industry Strategy, Intel and Shoumen Datta, MIT Forum for Supply Chain Innovation:

Adaptive Decision Making in the Evolving Global Economy

.... rapid developments in the field of automatic identification technologies (AIT) such as radio frequency identification (RFID) promise an unprecedented new management resource, wrapped in an equally unprecedented operational challenge�how to structure and support real-time analysis and response concurrently at local, regional and central decision points.

This paper will examine some of the tools and technologies that are emerging for decentralized real-time decision making, particularly as they impact traditional supply chain systems and their related business processes. We believe these technologies, along with innovative business practices, can be used to meet the challenges and harvest the opportunities presented by the new global economy ....

Articles on information visualization

There were two mildly interesting articles (lots of company names, which is nice) about the information explosion and increasing need for visualization tools, both in real-time and large data sets.

Take a look at 'It's a vision thing' by Chris Nuttall in the Financial Times (requires a subscription) and 'Intelligence Or Info Overload?' by Rick Whiting from Infoweek.