Q: Which Cascade solver is right for my application?
A: Cascade supports a suite of massively-parallel high-fidelity Computational Fluid Dynamics (CFD) flow solvers, available for different flow regimes and containing various modeling capabilities. A brief description of these solvers is provided below where each is based on the Large-Eddy Simulation (LES) methodology.
- Charles: An explicit, compressible flow solver ideal for aero-acoustics.
- Chris: A compressible, reacting flow solver targeted at simulating problems involving high speed combustion.
- Cliff: An incompressible Navier-Stokes solver applicable to problems ranging from low Mach number aerodynamics to scalar transport and mixing.
- Vida: A variable density solver for low-Mach number combustion suitable for gas turbine combustors.
Additionally, Cascade offers several mesh preparation and post-processing tools.
- Prepro: Import a computational mesh, interpolate solution data between meshes, transform coordinates.
- Adapt: An automated mesh transformation tool capable of local, anisotropic refinement based on target length scales.
- Postpro: Write visualization data from simulation results.
Q: What are the hardware and software dependencies for running the Cascade suite of solvers?
A: A Linux-based computer or cluster is required with the following software:
1. C and C++ compilers (GNU, Intel, etc.)
2. MPI including MPI-I/O*
*It is possible to run the solvers and pre/post-processing tools in a single-processor mode without MPI. Otherwise, the MPI 1.2 standard library is required (mpich, mvapich, openmpi or native implementation) and must include MPI-I/O.
3. ParMETIS/METIS libraries for parallel graph partitioning (version 3.X)
The following software is not required, however interfaces to these packages are provided in the infrastructure, and will improve the performance of certain solvers in some cases:
2. Hypre libraries (linear solver package from LLNL)
3. Trilinos libraries (linear solver package from Sandia)
4. Tecplot (for visualizations)
5. Paraview (for visualization)
6. Matlab (for post-processing)
Q: Where can I download the Cascade solver suite?
A: To interact most efficiently with developments made by the Cascade team, you will need git installed on your compute systems. Git is a free and open source distributed version control system. As a licensed user of the tools, you will be granted read privileges to Cascade’s code repositories hosted on github.com. Instructions for accessing the solvers, keeping your version up to date, and even making branches to do your own developments, are available at https://github.com/cascadetech.
In some cases, you may need to install the code from a zip file. You can request a zip file version of the repository from github, and then extract it somewhere in your user directory of the machine you are installing on.
Q: How do I build an executable for my system?
A: It is recommended that you set the environment variable CTI_HOME to the location of the solver suite obtained from GitHub. If you are using bash, you can do this by changing to the directory and entering:
where $PWD should contain the full path to the cti directory.
All system-dependent settings are specified in the file Makefile.in in this directory. Note that this file is not included with the source. You will need to construct a customized version of this file for your system. It is not an automated process. There are many examples in the subdirectory makefiles that you can use as a starting template. In general, we recommend that you create your own configuration file (e.g. Makefile.MYSYSTEM.in) in makefiles folder and then create a link to it in CTI_HOME. This will allow you to more efficiently manage multiple systems, and even multiple configurations on a single system.
ln -s makefiles/Makefile.MYSYSTEM.in Makefile.in
You should now be ready to make.
This will create “vanilla” versions of the following binaries in the bin directory: cliff.exe, chris.exe, charles.exe, vida.exe, prepro.exe, postpro.exe, adapt.exe, as well as other tools including: lesinfo.exe, createChemtable.exe, and snapshotConvert.exe. At this time we want to apologize for the .exe extensions. In addition to letting you know which files are the executables, this extension simplifies the code’s make process.
Q: How do I build a custom executable?
A: Assuming the CTI_HOME environment variable has been set to the location of the source code directory and the “vanilla” version of the solver has been successfully compiled, you can create a version of one of the solvers as a starting point for your own customizations as follows:
1. cd to the working directory mycase where you are going to build and/or run your case. This should preferably be at some location outside of the source code distribution
$ mkdir mycase
$ cd mycase
2. Copy “Makefile” from the directory “makefiles” in the distribution to this running directory. This is a special Makefile set up to build customized versions of the tools.
$ cp $CTI_HOME/makefiles/Makefile ./
3. You can now make a local version of the solver, for instance charles.
$ make charles
If not already present, the make command will copy over the default file (for instance charles.cpp) and then build a local executable (e.g. charles). If the file was copied over, this executable will be identical to the vanilla version of the code in the distribution. You can now edit this file, adding your own customizations, and then issue the make command again.
Programming your own class and hook routines requires that you have some knowledge of how the grid and solution data is laid out and accessed in the solver infrastructure. Please see the Cascade user manual for further details.
Q: How do I set solution parameters like time step and boundary conditions?
A: Upon execution, each Cascade solver reads parameters from a file. The default name of the parameter file is .in, where is the name of the particular solver or utility being used. For example, the charles compressible flow solver uses the parameter file charles.in.
The parameter file is an ASCII file where each line has the format KEYWORD = VALUE. Keywords are case sensitive, and comments can be added to any line using #, as in the following snippet:
# this is the location of the restart file...
RESTART = ./case21/restart.les
# solver controls
DT = 0.01
NSTEPS = 1000 # run for 1000 time steps
WRITE_DATA NAME=u_and_p INTERVAL=100 \
GEOM=ALL VARS=U P
The “=“ character is parsed as white space, and can be omitted if desired. As illustrated by the parameters PRESSURE SOLVER and WRITE DATA, the value for any keyword can involve multiple tokens. Long parameters can also be continued on the next line or lines using a backslash.
Normally the order of parameters in the parameter file is not important, however order is preserved which may be important in some cases (e.g. multiple grid transformations based on parameters).
Parameters can also be added at the command line using the syntax –KEYWORD = VALUE. Their behavior is identical to parameters coming from an input file. The use of a double- hypen is required to disambiguate multiple keywords from negative numbers. Command line parameters will appear at the top of the parameter list, giving them priority over any similar parameters in the input file. This command line feature can be useful for either running utilities where few parameters are required (because this saves the user from having to write an input file), or for running multiple cases in a batch mode where one parameter is changed.
This is an example of adding parameters at the command line:
$ mpirun -np 128 ./prepro --RESTART ./mesh/mymesh.msh --TECPLOT
This has the same behavior as running the preprocessing tool prepro without any command
line parameters and with the following prepro.in:
RESTART = ./mesh/mymesh.msh
Q: How do I run a Cascade solver?
A: Once your solver executable has been successfully compiled, the syntax for running the solver will depend on your cluster and its associated queueing system. For example, on some systems you will be able to launch the job using the following command:
$ mpirun -np 300 ./charles
This will launch charles on 300 cores. Note that this is the only time that the number of cores needs to be specified. Any mesh and solution data will be automatically repartitioned if required.
Q: How can I create a restart.les file?
A: The prepro utility can be used to prepare the restart.les file for your simulation. Its most common use is to add periodicity to a mesh coming from another source, such as a previous restart file or an ASCII Fluent msh or cas file, or to interpolate data from one restart file onto a new restart file. prepro reads parameters from the input file prepro.in, and also from the command line using the double-hyphen notation.
For example, to simply read in a ”.msh” file and output an “.les” file:
$ mpirun -np 32 ./prepro --RESTART fluent.msh
Alternatively, the RESTART parameter also supports the generation of simple Cartesian grids using the following syntax:
RESTART = SIMPLE_CART DOMAIN -5 5 -5 5 -0.1 0.1 SIZE 32 32 1 PERIODIC_X PERIODIC_Y PERIODIC_Z
This feature can be helpful for testing new capabilities and for generating very large meshes to test speed, scalability, and I/O rates. Using the example above will produce a triply-periodic mesh, and no boundary conditions will be required. Omitting any of the PERIODIC_* tokens will result in face boundary zone names x0, x1, y0, y1, etc.
Q: How can I stop a running simulation?
A: The default behavior for all solvers is to run forever. If the solver is running interactively, breaking out (ctrl-c) is not recommended, unless it is certain that the cluster will clean up after itself properly. Additionally, any result that has been computed will be lost. There are several ways to stop the solver more gracefully:
At regular intervals, the charles solver checks its run directory for the presence of the file “killcharles”. If it is found, it is deleted, and the solver stops cleanly, writing a restart file. One of the simplest ways to produce this file in a linux setting is to use the “touch” command. For example,
$ touch killcharles
produces a zero-size file named “killcharles”. The files killchris, killcliff and
killvida have the same function for the respective solvers.
The number of steps that the solver takes can be controlled by using the NSTEPS parameter
in the input file. For example, the parameter
NSTEPS = 1000
will cause the solver to take 1000 steps and then stop, writing a result file after computing step 1000.
If the total allowed runtime (wall clock time) for the simulation is known, as is often the case in a queueing environment, use the RUNTIME parameter in the input file. For example, for an allowed runtime of 12 hours, add the following line to the input file:
RUNTIME = 11.9
and the solver will integrate ahead as many steps as it can fit in this allowed time.
Q: How can I write visualization data?
A: The WRITE DATA parameter allows you to inspect all or a part of your simulation data at a
user-specified interval. The syntax is as follows:
WRITE_DATA FORMAT=<format> NAME=<prefix> INTERVAL=<int> GEOM=<geom-data> VARS=<var1> [<var2>...]
You can introduce as many WRITE DATA parameters into your input file as you would like, although excessive I/O may slow down the simulation considerably. The default format is TECPLOT, although Paraview is also supported with the keyword VTK.