A snapshot of pressure and temperature from the “crackling” jet run during the scaling study.

At the one million core level, previously innocuous parts of simulation codes may suddenly become bottlenecks, and massive parallelism through all aspects of the software architecture is critical. Joe and other researchers from Stanford’s PSAAP program and LLNL computing staff have been working closely together for a few weeks now to prepare for this unprecedented opportunity. So, together, they were glued to their terminals (more than usual) Tuesday afternoon and into the evening during the first “full-system scaling” window of the early science testing period to see whether initial extreme-scale science runs would achieve stable run-time performance after starting up. As the first CFD simulation passed successfully through its initialization phase, all were thrilled as they saw the code performance continue to scale all the way to and beyond one million cores. This means that the time-to-solution continued to reduce, enabling more science to be done with ever faster turnaround times. These early science runs represent at least an order-of-magnitude increase in computational power over the largest runs performed using CharLES to date, enabling unprecedented fidelity and dramatically reducing time to solution.

Calculations were performed on the Sequoia supercomputer installed at Lawrence Livermore National Lab.

Finally – the last two episodes of our interview with Prof. Moin! In these two short videos, the major upcoming challenges for Large Eddy Simulation fundamental research and its industrial applications are outlined, from the development of sub-grid scale models for multi-physics applications to the description of what it takes to be “exascale computing-ready”.

The third episode of our interview with Prof. Moin is finally here! In this short video, Prof. Moin explains what LES stands for, where the term originated, and the main components that make up a robust Large Eddy Simulation solver. Of particular interest, a detailed history of the development of sub-grid modeling is presented, including the breakthrough introduction of dynamic sub-grid models in 1991, originally developed at Stanford’s Center for Turbulence Research.

Here is the second part of our interview series with Prof. Parviz Moin. In this episode, a number of milestones in the development of Large Eddy Simulation technology are discussed, including the role of Direct Numerical Simulation and the rise of computing resources, as well as the first applications of LES in predicting large scale mixing in turbulent combustion and for low Reynolds number internal flows.

This is the first of a 5-parts interview with Prof. Parviz Moin. In addition to his role as Cascade Technologies’s founder, Prof. Moin is the founding director of the Center for Turbulence Research at Stanford and Ames. Professor Moin pioneered the use of direct and Large Eddy Simulation techniques for the study of turbulence physics, control and modelling concepts and has written widely on the structure of turbulent shear flows.

Intrepid consists of 40 Blue Gene/P racks of 4096 cores each for a total of 163840 cores. A visual snapshot of the daily activity on Intrepid can be seen below, where each of the smallest squares is made up of 128 cores, every larger block represents 2048 cores (1/2 of a complete rack), and every color represents a different project:

Quite amazing isn’t it? Lots of activity and simulations running concurrently.

However, one of our jet noise, aeroacoustics Large Eddy Simulation runs on a 500 million unstructured grid; needing as much computing power as possible, we ended up using every single one of the 163840 available cores for 4 days in a row, for a total of 16 million core-hours of computational time.

Here is how the Intrepid activity screen looked during the simulation:

Definitely easier on the eyes in our opinion….

(our thanks to Joseph Nichols and Prof. Sanjiva Lele)

Basically, blockbuster allows you to open a collection of images output from your favorite visualization software and animate them, zooming in and out to better interrogate multi-scale details of the flow structure. The use of multiple image files rather than a single movie in a movie player allows you to easily modify the length of a sequence (by adding more images), and also prevents any lossy artifacts related to compression. Just like a flipbook, images can be stepped through one at a time in full fidelity, and even played backwards! Here is an example of the complicated shock and mixing structures associated with sonic injection of hydrogen into a supersonic flow. Animation of the image sequence in blockbuster reveals clear interactions and instabilities not obvious from these stills.

If you are using a compatible browser, you can test Blockbuster here: www.cascadetechnologies.com/codes/blockbuster. There is also a collection of images from the simulation mentioned above: a sonic hydrogen jet in supersonic crossflow simulation performed for a US Air Force STTR program. Enjoy.

Recent rule changes and technological innovations have resulted in an increasing reliance on simulation for the design of Formula 1 racecars. With the freeze on engine and electronics development, aerodynamic performance is now the primary means to gain an advantage. Moreover, to reduce costs, teams have agreed to limit track testing and wind tunnel measurements. Exact CFD capabilities are closely guarded, but some teams rely on computational facilities in the ~80 teraflop peak range.

Ferrari's aesthetically-challenged stepped nose design

The most visually striking change to the cars in 2012 are the stepped noses. They are a consequence of new safety rules that mandate a maximum height of the noses at several points, with most teams choosing noses that follow this maximum envelope. High noses are beneficial because they feed more air to the undertray and diffuser, which produce approximately 50% of the car’s total downforce. New regulations will also ban blown diffusers, where high-energy exhaust gases are directed into the diffuser to increase downforce. Many teams developed special engine maps last season to keep exhaust gas pressure high, even when the driver is off the throttle (producing a distinct exhaust note). Returning this season is the so-called Drag Reduction System (DRS), a driver-controlled movable rear wing element. To encourage passing, drivers can decrease the angle of attack of their rear wing via an actuator when close to a car ahead. The DRS is meant to counteract the disadvantage experienced by the following car in the highly turbulent wake of an open-wheeled Formula 1 car.

Drag reduction system open (top) and closed - notice also the special green paint used to visualize flow structures during practice sessions

The 2012 season promises to be exciting – for the first time ever, six world champion drivers will be on the grid. Formula 1 may also gain traction in the United States with the recent addition to the calendar of the US Grand Prix in Austin, Texas. In a sport where tenths matter, high-fidelity simulation will inevitably become more widespread and sophisticated. Stay abreast of technological developments in the sport with two of the best English-language F1 blogs: James Allen on F1 and ScarbsF1.

Leveraging the incredible advancements in high-performance computing in these past few years, numerical techniques such as Large Eddy Simulation (LES) are finally emerging as an accurate yet cost-effective computational tools for the prediction of turbulent flows.

But what exactly is turbulence? A few, everyday examples, as described in the ‘Tackling Turbulence with Supercomputers” article by Prof. P.Moin and Prof. J.Kim (originally appeared in the Scientific American magazine a few years ago), can be illuminating:

“Open a kitchen tap only a bit, and the water that flows from the faucet will be smooth and glassy. This flow is known as laminar. Open the tap a little further, and the flow becomes more roiled and sinuous: turbulent, in other words. The same phenomenon can be seen in the smoke streaming upward into still air from a burning cigarette. Immediately above the cigarette, the flow is laminar. A little higher up, it becomes rippled and diffusive – i.e. turbulent, again. Turbulence is therefore composed of eddies: patches of zigzazzing, often swirling fluid, moving randomly around and about the overall direction of motion. Technically, the chaotic state of fluid motion arises when the speed of the fluid exceeds a specific threshold, below which viscous forces damp out the chaotic behavior.”

Despite the tremendous complexity in finding a mathematical description of turbulence (Richard Feynman, the Nobel Prize-winning physicist, once called turbulence “the most important unsolved problem of classical physics”, and the Clay Mathematics Institute is still offering a 1,000,000 USD prize for a more solid understanding of the Navier-Stokes equations existence and smoothness), the good news is that, after decades of continuous improvements, high-performance computing is finally enabling us to visualize and predict turbulence with a high degree of fidelity.

Here is such an example of a high-fidelity turbulent flow, representing the very fine structures of the shear layer originating from the merging of the turbulent boundary layers on the pressure and suction sides of a hydrofoil. Starting from the larger picture and moving clockwise, the new pictures represents each red box, culminating – on the top right – with the numerical grid necessary to perform this direct numerical simulation.