reduces run time for analyses that require a large number of increments;
reduces run time for analyses that contain a large number of nodes and elements;
produces analysis results that are independent of the number of processors used for the
analysis;
divides the compute work among the processors by decomposing the model into multiple
domains;
is available as a thread-based implementation to take advantage of shared memory on a
single node computer;
is available as an MPI-based implementation whether
using a single node or a large compute cluster;
is available in hybrid mode that uses a combination of
MPI and threads to exploit the
NUMAarchitecture whether using a single node or a large
cluster; and
can use one or more compute-capable GPGPU resources in
combination with regular CPU processors, spread over one
node or multiple nodes of a cluster.
Parallelization in Abaqus/Explicit is implemented using a domain-decomposition method that splits the model into topological
domains and assigns each domain to a processor.
Parallelization is invoked by specifying the number of processors to use.
Input File Usage
Enter the following input on the command line:
abaqusjob=job-namecpus=n
For example, the following input runs the job “beam” on two processors:
abaqus job=beam cpus=2
Abaqus/CAE Usage
Job module: job editor: Parallelization: toggle on Use multiple processors, and specify the number of processors, n
Domain Decomposition
The domain-decomposition method used in Abaqus/Explicit splits the model into a number of topological domains. These domains contain a subset of
the nodes and elements and all the modeling features necessary to compute the solution. The
domains are distributed among the available processors. The analysis is then carried out
independently in each domain. However, information must be passed between the domains in
each increment because the domains share common boundaries. The domain-decomposition method
is supported with MPI, thread-based parallelization, and in
hybrid mode (a combination of MPI and the thread-based
method).
During initialization, the domain-decomposition method divides the model so that the
resulting domains take an appropriate amount of computational expense. When using
GPGPU hardware, the domain sizes are uneven with the
large domains assigned for GPGPU execution. Otherwise, when
running only on the CPU hardware, the decomposition aims to
create domains of equal size.
For a CPU-only execution, the load balance is defined as
the ratio of the computational expense of the domains in the most expensive process to that
of the domains in the least expensive process. For cases exhibiting significant load
imbalance, either because the initial load balancing is not adequate (static imbalance) or
because imbalance develops over time (dynamic imbalance), Abaqus/Explicit recommends using either hybrid execution or thread-based execution.
Abaqus/Explicit also provides the dynamic load balancing technique (see Abaqus/Standard and Abaqus/Explicit Execution). This technique is suitable for certain workflows,
particularly when using a single node either in a pure
MPI-based execution mode or a thread-based execution mode.
It is based on over-decomposition: you select a number of domains that is a multiple
(typically double) of the number of processors. During the calculation, Abaqus/Explicit regularly measures the computational expense and redistributes the domains over the
processors to minimize the load imbalance.
Increasing the number of domains also increases the computational cost slightly as the
compute work is subdivided and the communication overhead increases. Therefore, the
efficiency of the dynamic load balancing scheme depends on the load imbalance inherent to
the problem and on the communication efficiency of the hardware. Problems with a strongly
time-dependent or spatially varying computational load are most likely to benefit from
dynamic load balancing. For example, models containing airbags (where contact-impact
activity is highly localized and time dependent) and coupled Lagrangian-Eulerian problems
(where constitutive activity follows the material as it moves through empty space) would
likely be suited to dynamic load balancing.
The following functionality is not supported with dynamic load balancing:
You should specify the number of domains only when you are using the dynamic load balancing
technique; in all other cases Abaqus/Explicit chooses an optimal value based on the hardware resources.
Element and node sets are created for each domain and can be
inspected in Abaqus/CAE. The sets are named domain_n,
where n is the domain number.
For example, the following input runs the job “beam” on two processors:
abaqus job=beam cpus=2 domains=4
Abaqus/CAE Usage
Job module: job editor: Parallelization: toggle on Use multiple processors
and specify the number of processors, n; Number of domains:m;
toggle on Activate dynamic load balancing; Parallelization method:
Domain
You can activate dynamic load balancing when the number of domains is a multiple of
the number of processors.
Thread-Based Execution
Abaqus/Explicit can be executed in thread mode within one node of a compute cluster, which takes
advantage of the shared memory available to the threads that are running on different
cores.
Input File Usage
To select this execution mode, set the command line option
threads_per_mpi_process equal to the number of
cpus:
abaqusjob=job-namecpus=nthreads_per_mpi_process=n
For example, the following input runs the job “beam” on 20 cores with thread-based
domain-level parallelization using 20 threads:
Job module: job editor: Parallelization: toggle on Use multiple processors
and specify the number of processors, n;
Multiprocessing mode: Threads
MPI-Based Execution
Abaqus/Explicit can be executed in pure MPI mode, which uses the
message passing interface (MPI) to communicate between
processes running on different cores that might be spread over a single or multiple
compute nodes on an HPC cluster. To select this execution
mode, set the command line option
threads_per_mpi_process equal to 1.
For example, the following input runs the job “beam” on 20 cores with
MPI-based domain-level parallelization using 20
MPI processes:
abaqus job=beam cpus=20 threads_per_mpi_process=1
Abaqus/CAE Usage
Job module: job editor: Parallelization: toggle on Use multiple processors
and specify the number of processors, n;
Multiprocessing mode: MPI
Hybrid Execution
Abaqus/Explicit can be executed in hybrid mode using a combination of
MPI and threads with each
MPI process launching a user-specified number of
threads. For hybrid execution, the total number of cpus must be divisible by the number of
threads per MPI process, which is specified using the
command line option threads_per_mpi_process.
HPC clusters typically consist of compute nodes with
multiple sockets, and each socket contains numerous cores. In addition, the cores within a
socket can be further organized into multiple NUMA
(nonuniform memory access) nodes with dedicated resources, such as local memory, for
faster access. Hybrid execution of Abaqus/Explicit takes advantage of the NUMA architecture and the trend
of increasing number of cores available on each NUMA
node. It is recommended that the threads per MPI process
parameter be set equal to the number of cores in a NUMA
node, resulting in one MPI rank per
NUMA node. However, when the number of cores in a
NUMA node is large, you can experiment to allow
multiple MPI ranks within a
NUMA node. When submitting to a batch queuing system,
the processors per node (ppn) should be set to the total number of cores available per
compute node.
The number of domains is selected by Abaqus/Explicit automatically and does not need to be specified. If specified, Abaqus/Explicit can tune the number of domains for optimal performance. The resulting number of domains
is listed in the status (job-name.sta)
file.
Input File Usage
Enter the following input on the command line:
abaqusjob=job-namecpus=nthreads_per_mpi_process=m
For example, the following input runs the job “beam” on 80 cores with a hybrid MPI-
and thread-based domain-level parallelization method using 4
MPI processes and 20 threads per
MPI process:
Job module: job editor: Parallelization: toggle on Use multiple processors
and specify the number of processors, n; threads_per_mpi_process:m;
Multiprocessing mode: Hybrid
GPGPU Acceleration
Abaqus/Explicit can be accelerated by combining the GPGPU devices with
the CPU cores. When using cores of a single node with one
GPGPU device, Abaqus/Explicit can be executed in thread mode. Multiple GPGPU devices
can be used with cores of either a single node or multiple nodes when running Abaqus/Explicit in hybrid mode.
Input File Usage
Enter the following input on the command line for threaded execution with one
GPGPU device added:
For example, the following input runs the job “beam” on 20 cores and 1
GPGPU device with
MPI-based domain-level parallelization using 20
MPI processes:
For example, the following input runs the job “beam” on 80 cores and 4
GPGPU devices in a hybrid method using 4
MPI processes with 20 threads and 1
GPGPU device per MPI
process:
While most of the existing features are supported, the current release of
GPGPU implementation of Abaqus/Explicit comes with the following limitations, which will be addressed in the subsequent
releases:
Works only on GPGPU devices from
NVIDIA.
Works only on x86 processors running
Linux operating system.
User subroutines are not supported.
Eulerian (CEL) and Arbitrary Lagrangian-Eulerian (ALE) elements are not
supported.
Particle methods are not supported.
Selective subcycling analysis technique is not supported.
Selecting an Execution Mode
By selecting a suitable value for the
threads_per_mpi_process option on the command line,
you can vary execution between the thread mode, the hybrid mode, and the pure
MPI mode. When using all the physical cores of the
assigned nodes for the current job and not using any
gpus, the Abaqus/Explicit driver can query the hardware and select an optimal execution mode. Therefore, in this
scenario it is recommended that you do not specify the
threads_per_mpi_process option on the command line.
Consistency of Results
To obtain consistent analysis results from run to run, the domain decomposition should be
the same, even if the number of processors is varied. Except for cases in which the
single- and multiple-domain models are different because of features that are not yet
available with multiple parallel domains (discussed below), any results differences on
account of the varied decomposition should be triggered only by finite precision effects.
For example, the order of the nodal force assembly can depend on the number of parallel
domains, which can result in differences in trailing digits in the computed force. Some
physical systems are highly sensitive to small perturbations, so a tiny difference in the
force applied in one increment can result in noticeable differences in results in
subsequent increments. Simulations involving buckling and other bifurcations tend to be
sensitive to small perturbations.
Features that Do Not Allow Parallelization
The use of the parallelization is not allowed with the following features:
Extreme value output. An alternative is to filter the output.
Steady-state detection. This feature is typically used with ALE adaptive mesh
domains.
If you include these features in a parallel run, Abaqus issues an error message.
Features That Cannot Be Split across Domains
Certain features cannot be split across domains. The domain decomposition algorithm
automatically takes this into account and forces these features to be contained entirely
within one domain. If fewer domains than requested processors are created, Abaqus/Explicit issues an error message. Even if the algorithm succeeds in creating the requested
number of domains, the load might be balanced unevenly.
If you define tracer particles in the adaptive mesh domains, the adaptive mesh domains
cannot span parallel domain boundaries. In this case, the nodes on the boundary between an
adaptive mesh domain and a nonadaptive domain as well as the adaptive nodes on the surface
of the adaptive mesh domain cannot be shared with another parallel domain. To enforce this
in a consistent manner when you specify parallel domains, Abaqus/Explicit sets all nodes shared by adjacent adaptive mesh domains as nonadaptive. The analysis
results might be significantly different from that of a serial run with no parallel
domains. If this behavior is undesirable, set the number of parallel domains to 1, and
switch to the loop-level parallelization method. See Defining ALE Adaptive Mesh Domains in Abaqus/Explicit for details.
A contact pair cannot be split across parallel domains, but separate contact pairs are
not restricted to be in the same parallel domain. A contact pair that uses the kinematic
contact algorithm requires that all the nodes associated with the involved surfaces be
within a single parallel domain and not be shared with any other parallel domains. A
contact pair that uses the penalty contact algorithm requires that the associated nodes be
part of a single parallel domain, but these nodes might also be part of other parallel
domains if they are not defined in an adaptive mesh domain. Analyses in which a large
percentage of nodes are involved in contact might not scale well if contact pairs are
used, especially with kinematic enforcement of contact constraints. On the other hand, the
general contact has no such limitations on the decomposition.
Nodes involved in kinematic constraints (About Kinematic Constraints), except for
surface-based shell-to-solid constraints, will be within a single parallel domain; and
they will not be shared with another parallel domain. However, two kinematic constraints
that do not share nodes can be placed within different parallel domains.
In some cases beam elements that share a node might be forced into the same parallel
domain. This happens only for beams whose center of mass does not coincide with the
location of the beam node or for beams with additional inertia (see Adding Inertia to the Beam Section Behavior for Timoshenko Beams).
User Influence on Domain Decomposition
You can influence the domain decomposition by specifying one or more regions that are
independently decomposed into a user-specified number of parallel domains or by specifying
that an element set should be constrained to the same parallel domain.
Specifying a domain decomposition region can be useful when a local region of the model
is computationally intensive. Performance gains can be achieved by identifying the local
region as an independent domain decomposition region, thereby distributing computation of
the local regions among all processors. You can specify the domain decomposition region by
defining an element set directly, or Abaqus/Explicit can generate the domain decomposition region consisting of all
elements within a user-specified box. The part of the model that is not included in any
user-specified domain decomposition region is considered as the global region and is also
decomposed into the user-specified number of parallel domains. You can specify that each
decomposition region can be decomposed using a recursive coordinate bisection
(RCB) algorithm or a graph partitioning algorithm that
minimizes the number of shared nodes. The RCB algorithm
is the default for all domain decomposition regions. You can also specify that each domain
decomposition region can be decomposed into domains by specifying a decompose factor N. The
domains from each independent domain decomposition are distributed evenly among the
available processors, but these domains can be reassigned to different processors during
the analysis if dynamic load balancing is activated. The total number of parallel domains
for the simulation is
where
is the number of local regions identified as independent domain decomposition
regions;
is equal to 1 if any elements are not included in local regions identified as
independent domain decomposition regions; otherwise, is 0;
is the decompose factor for domain decomposition region ;
is the decompose factor for the global domain decomposition region; and
is the user-specified number of domains per domain decomposition region (see Domain Decomposition).
Separate domain decomposition regions might be desired, for example, in bird-strike
models (where contact-impact activity is highly localized and time dependent) and coupled
Eulerian-Lagrangian problems with localized adaptive mesh refinement (where elements are
refined adding to the computational cost). The example below (Figure 1) shows a spherical projectile impacting a flat plate with a failure model, thus
allowing the projectile to perforate the plate. One of the domains contains the projectile
as well as a significant portion of the impact area. Specifying a domain decomposition
region consisting of the projectile as well as the computationally intensive impact area
results in a more balanced parallel processing (Figure 2). In this example and ; therefore, .
Figure 1. Original domain decomposition. Figure 2. Modified domain decomposition.
Multiple domain decomposition regions can be specified. In the case of overlap between
the domain decomposition regions, by default, the first specified decomposition keeps the
overlapped elements. Some modeling features cannot be split across domains, and Abaqus/Explicit automatically merges the domain decomposition regions that contain features that cannot
be split.
Input File Usage
Use the following option to define a domain decomposition region using a
user-specified element set:
Use the following option to define a domain decomposition region by specifying that
Abaqus/Explicit should generate the domain decomposition region consisting of
all elements contained within the user-specified box:
Domain decomposition is not supported in Abaqus/CAE.
Restart
There are certain restrictions for restart when using domain-level parallelization. To
ensure that optimal parallel speedup is achieved, the number of processors used for the
restart analysis must be chosen so that the number of parallel domains used during the
original analysis can be distributed evenly among the processors. Because the domain
decomposition is based on the features specified in the original analysis and steps
defined therein, features that affect domain decomposition are restricted from being
defined in restart steps only if they would invalidate the original domain decomposition.
Because the newly added features will be added to existing domains, there is a potential
for load imbalance and a corresponding degradation of parallel performance.
The restart analysis requires that the separate state and selected results files created
during the original analysis be converted into single files, as described in Abaqus/Standard and Abaqus/Explicit Execution. This should be done automatically at the conclusion
of the original analysis. If the original analysis fails to complete successfully, you
must convert the state and selected results files prior to restart.
Co-Simulation
The co-simulation technique (About Co-Simulation) for run-time
coupling of Abaqus/Explicit to Abaqus/Standard or to third-party analysis programs can be used with Abaqus/Explicit running either in serial or parallel.
Measuring Parallel Performance
Parallel performance is measured by comparing the total time required to run on a single
processor (serial run) to the total time required to run on multiple processors (parallel
run). This ratio is referred to as the speedup factor. Even if the computation were
perfectly divided and executed at the same rate as in serial execution, a parallel will have
overhead in terms of data communication. Thus, the speedup factor will likely be less than
the number of processors used for the parallel run. The ratio of the speedup factor and the
number of processors is often referred to as the parallelism efficiency. Scaling refers to
the behavior of the speedup factor as the number of processors is increased. Perfect scaling
indicates that the speedup factor increases linearly with the number of processors, i.e.,
the efficiency remains constant. It is typical that the load imbalance among the processors
increases and the opportunity for vector operations reduces as a given model gets fragmented
onto increasing number of processors; thus, reducing the parallel efficiency with increasing
processors. The speedup factors and scaling behavior are heavily problem dependent, with
larger models scaling more efficiently to a higher number of processors.