--- title: "ParallelArray" language: "en" type: "Symbol" summary: "ParallelArray[f, n] generates in parallel a list of length n, with elements f[i], evaluated. ParallelArray[f, {n1, n2, ...}] generates in parallel an n1*n2*... array of nested lists, with elements f[i1, i2, ...]. ParallelArray[f, {n1, n2, ...}, {r1, r2, ...}] generates in parallel a list using the index origins ri (default 1). ParallelArray[f, dims, origin, h] uses head h, rather than List, for each level of the array." canonical_url: "https://reference.wolfram.com/language/ref/ParallelArray.html" source: "Wolfram Language Documentation" related_guides: - title: "Data Parallelism" link: "https://reference.wolfram.com/language/guide/DataParallelism.en.md" related_functions: - title: "Array" link: "https://reference.wolfram.com/language/ref/Array.en.md" - title: "ParallelTable" link: "https://reference.wolfram.com/language/ref/ParallelTable.en.md" - title: "Parallelize" link: "https://reference.wolfram.com/language/ref/Parallelize.en.md" --- # ParallelArray ⚠ *Unsupported in Public Cloud* ParallelArray[f, n] generates in parallel a list of length n, with elements f[i], evaluated. ParallelArray[f, {n1, n2, …}] generates in parallel an $n_1\times n_2\times \ldots$ array of nested lists, with elements f[i1, i2, …]. ParallelArray[f, {n1, n2, …}, {r1, r2, …}] generates in parallel a list using the index origins ri (default 1). ParallelArray[f, dims, origin, h] uses head h, rather than List, for each level of the array. ## Details and Options * ``ParallelArray`` is a parallel version of ``Array``, which automatically distributes different evaluations of ``expr`` among different kernels and processors. * ``ParallelArray`` will give the same results as ``Array``, except for side effects during the computation. * ``Parallelize[Array[f, n]]`` is equivalent to ``ParallelArray[f, n]``. * If an instance of ``ParallelArray`` cannot be parallelized, it is evaluated using ``Array``. * The following options can be given: | | | | | -------------------- | --------------------- | ------------------------------------------------------------ | | Method | Automatic | granularity of parallelization | | DistributedContexts | \$DistributedContexts | contexts used to distribute symbols to parallel computations | | ProgressReporting | \$ProgressReporting | whether to report the progress of the computation | * The ``Method`` option specifies the parallelization method to use. Possible settings include: | | | | --------------------------- | ------------------------------------------------------------------------- | | "CoarsestGrained" | break the computation into as many pieces as there are available kernels | | "FinestGrained" | break the computation into the smallest possible subunits | | "EvaluationsPerKernel" -> e | break the computation into at most e pieces per kernel | | "ItemsPerEvaluation" -> m | break the computation into evaluations of at most m subunits each | | Automatic | compromise between overhead and load balancing | * ``Method -> "CoarsestGrained"`` is suitable for computations involving many subunits, all of which take the same amount of time. It minimizes overhead, but does not provide any load balancing. * ``Method -> "FinestGrained"`` is suitable for computations involving few subunits whose evaluations take different amounts of time. It leads to higher overhead, but maximizes load balancing. * The ``DistributedContexts`` option specifies which symbols appearing in ``expr`` have their definitions automatically distributed to all available kernels before the computation. * The default value is ``DistributedContexts :> \$DistributedContexts`` with ``\$DistributedContexts := \$Context``, which distributes definitions of all symbols in the current context, but does not distribute definitions of symbols from packages. * The ``ProgressReporting`` option specifies whether to report the progress of the parallel computation. * The default value is ``ProgressReporting :> \$ProgressReporting``. --- ## Examples (19) ### Basic Examples (4) ``ParallelArray`` works like ``Array``, but in parallel: ```wl In[1]:= Array[(Pause[1];f[#])&, 4]//AbsoluteTiming Out[1]= {4.000318, {f[1], f[2], f[3], f[4]}} In[2]:= ParallelArray[(Pause[1];f[#])&, 4]//AbsoluteTiming Out[2]= {1.013779, {f[1], f[2], f[3], f[4]}} ``` --- Generate a $3\times 2$ array: ```wl In[1]:= ParallelArray[f, {3, 2}] Out[1]= {{f[1, 1], f[1, 2]}, {f[2, 1], f[2, 2]}, {f[3, 1], f[3, 2]}} ``` Generate a $3\times 4$ array: ```wl In[2]:= ParallelArray[10#1 + #2&, {3, 4}] Out[2]= {{11, 12, 13, 14}, {21, 22, 23, 24}, {31, 32, 33, 34}} ``` --- Use index origin ``0`` instead of ``1`` : ```wl In[1]:= ParallelArray[f, 10, 0] Out[1]= {f[0], f[1], f[2], f[3], f[4], f[5], f[6], f[7], f[8], f[9]} ``` Start with indices ``0`` and ``4`` instead of ``1``: ```wl In[2]:= ParallelArray[f, {2, 3}, {0, 4}] Out[2]= {{f[0, 4], f[0, 5], f[0, 6]}, {f[1, 4], f[1, 5], f[1, 6]}} ``` --- Used ``h`` instead of ``List`` for the result expression: ```wl In[1]:= ParallelArray[f, {3, 2}, 1, h] Out[1]= h[h[f[1, 1], f[1, 2]], h[f[2, 1], f[2, 2]], h[f[3, 1], f[3, 2]]] ``` ### Options (13) #### Method (6) Break the computation into the smallest possible subunits: ```wl In[1]:= ParallelArray[Labeled[Framed[#], $KernelID]&, 10, Method -> "FinestGrained"] Out[1]= {Labeled[1, 4], Labeled[2, 3], Labeled[3, 2], Labeled[4, 1], Labeled[5, 4], Labeled[6, 3], Labeled[7, 2], Labeled[8, 1], Labeled[9, 4], Labeled[10, 3]} ``` --- Break the computation into as many pieces as there are available kernels: ```wl In[1]:= ParallelArray[Labeled[Framed[#], $KernelID]&, 10, Method -> "CoarsestGrained"] Out[1]= {Labeled[1, 1], Labeled[2, 1], Labeled[3, 1], Labeled[4, 2], Labeled[5, 2], Labeled[6, 2], Labeled[7, 3], Labeled[8, 3], Labeled[9, 4], Labeled[10, 4]} ``` --- Break the computation into at most 2 evaluations per kernel for the entire job: ```wl In[1]:= ParallelArray[Labeled[Framed[#], $KernelID]&, 12, Method -> "EvaluationsPerKernel" -> 2] Out[1]= {Labeled[1, 4], Labeled[2, 4], Labeled[3, 3], Labeled[4, 3], Labeled[5, 2], Labeled[6, 2], Labeled[7, 1], Labeled[8, 1], Labeled[9, 4], Labeled[10, 3], Labeled[11, 2], Labeled[12, 1]} ``` --- Break the computation into evaluations of at most 5 elements each: ```wl In[1]:= ParallelArray[Labeled[Framed[#], $KernelID]&, 18, Method -> "ItemsPerEvaluation" -> 5] Out[1]= {Labeled[1, 4], Labeled[2, 4], Labeled[3, 4], Labeled[4, 4], Labeled[5, 4], Labeled[6, 3], Labeled[7, 3], Labeled[8, 3], Labeled[9, 3], Labeled[10, 3], Labeled[11, 2], Labeled[12, 2], Labeled[13, 2], Labeled[14, 2], Labeled[15, 1], Labeled[16, 1], Labeled[17, 1], Labeled[18, 1]} ``` --- The default option setting balances evaluation size and number of evaluations: ```wl In[1]:= ParallelArray[Labeled[Framed[#], $KernelID]&, 18, Method -> Automatic] Out[1]= {Labeled[1, 4], Labeled[2, 4], Labeled[3, 4], Labeled[4, 3], Labeled[5, 3], Labeled[6, 3], Labeled[7, 2], Labeled[8, 2], Labeled[9, 1], Labeled[10, 1], Labeled[11, 4], Labeled[12, 4], Labeled[13, 3], Labeled[14, 3], Labeled[15, 2], Labeled[16, 2], Labeled[17, 1], Labeled[18, 1]} ``` --- Calculations with vastly differing runtimes should be parallelized as finely as possible: ```wl In[1]:= ParallelArray[PrimeQ[2 ^ # - 1]&, 20, 4410, Method -> "FinestGrained"] Out[1]= {False, False, False, False, False, False, False, False, False, False, False, False, False, True, False, False, False, False, False, False} ``` A large number of simple calculations should be distributed into as few batches as possible: ```wl In[2]:= BinCounts[ParallelArray[Mod[Floor[# * Pi], 10]&, 10000, 1, Method -> "CoarsestGrained"], {0, 10}] Out[2]= {973, 1018, 973, 1019, 1016, 975, 1017, 974, 1017, 1018} ``` #### DistributedContexts (5) By default, definitions in the current context are distributed automatically: ```wl In[1]:= remote[x_] := Labeled[Framed[x], $KernelID] In[2]:= ParallelArray[remote, 4] Out[2]= {Labeled[1, 4], Labeled[2, 3], Labeled[3, 2], Labeled[4, 1]} ``` --- Do not distribute any definitions of functions: ```wl In[1]:= local[x_] := Labeled[Framed[x], $KernelID] In[2]:= ParallelArray[local, 4, DistributedContexts -> None] Out[2]= {Labeled[1, 0], Labeled[2, 0], Labeled[3, 0], Labeled[4, 0]} ``` --- Distribute definitions for all symbols in all contexts appearing in a parallel computation: ```wl In[1]:= a`f[x_] := Labeled[Framed[x], $KernelID] In[2]:= ParallelArray[a`f, 4, DistributedContexts -> Automatic] Out[2]= {Labeled[1, 4], Labeled[2, 3], Labeled[3, 2], Labeled[4, 1]} ``` --- Distribute only definitions in the given contexts: ```wl In[1]:= b`g[x_] := Labeled[Framed[x], $KernelID] In[2]:= ParallelArray[b`f, 4, DistributedContexts -> {"a`"}] Out[2]= {{0, 0}, {0, 0}, {0, 0}, {0, 0}} ``` --- Restore the value of the ``DistributedContexts`` option to its default: ```wl In[1]:= SetOptions[ParallelArray, DistributedContexts :> $DistributedContexts] Out[1]= {Method -> Automatic, DistributedContexts :> $DistributedContexts} ``` #### ProgressReporting (2) Do not show a temporary progress report: ```wl In[1]:= res = ParallelArray[PrimeQ[2 ^ # - 1]&, 10000, ProgressReporting -> False]; ``` --- Use ``Method -> "FinestGrained"`` for the most accurate progress report: ```wl In[1]:= res = ParallelArray[PrimeQ[2 ^ # - 1]&, 10000, Method -> "FinestGrained", ProgressReporting -> True]; During evaluation of In[2]:= Animator[0., {0., Infinity, 1.}, AnimationRate -> 1, AnimationRunTime -> 0.2135148048400879, AnimationTimeIndex -> 0.2135148048400879, AppearanceElements -> None, ImageSize -> {1, 1}] ``` ### Possible Issues (2) A function used that is not known on the parallel kernels may lead to sequential evaluation: ```wl In[1]:= ftest[n_] := Labeled[Framed[PrimeQ[2 ^ n - 1]], $KernelID] In[2]:= ParallelArray[ftest, 10, DistributedContexts -> None] Out[2]= {Labeled[False, 0], Labeled[True, 0], Labeled[True, 0], Labeled[False, 0], Labeled[True, 0], Labeled[False, 0], Labeled[True, 0], Labeled[False, 0], Labeled[False, 0], Labeled[False, 0]} ``` Define the function on all parallel kernels: ```wl In[3]:= DistributeDefinitions[ftest] Out[3]= {ftest} ``` The function is now evaluated on the parallel kernels: ```wl In[4]:= ParallelArray[ftest, 10, DistributedContexts -> None] Out[4]= {Labeled[False, 4], Labeled[True, 4], Labeled[True, 3], Labeled[False, 3], Labeled[True, 2], Labeled[False, 1], Labeled[True, 4], Labeled[False, 3], Labeled[False, 2], Labeled[False, 1]} ``` --- Definitions of functions in the current context are distributed automatically: ```wl In[1]:= gtest[n_] := Labeled[Framed[PrimeQ[2 ^ n - 1]], $KernelID] In[2]:= ParallelArray[gtest, 10] Out[2]= {Labeled[False, 4], Labeled[True, 4], Labeled[True, 3], Labeled[False, 3], Labeled[True, 2], Labeled[False, 1], Labeled[True, 4], Labeled[False, 3], Labeled[False, 2], Labeled[False, 1]} ``` Definitions from contexts other than the default context are not distributed automatically: ```wl In[3]:= ctx`mtest[n_] := Labeled[Framed[PrimeQ[2 ^ n - 1]], $KernelID] In[4]:= ParallelArray[ctx`mtest, 10] Out[4]= {Labeled[False, 0], Labeled[True, 0], Labeled[True, 0], Labeled[False, 0], Labeled[True, 0], Labeled[False, 0], Labeled[True, 0], Labeled[False, 0], Labeled[False, 0], Labeled[False, 0]} ``` Use ``DistributeDefinitions`` to distribute such definitions: ```wl In[5]:= DistributeDefinitions[ctx`mtest]; In[6]:= ParallelArray[ctx`mtest, 10] Out[6]= {Labeled[False, 4], Labeled[True, 4], Labeled[True, 3], Labeled[False, 3], Labeled[True, 2], Labeled[False, 1], Labeled[True, 4], Labeled[False, 3], Labeled[False, 2], Labeled[False, 1]} ``` Alternatively, set the ``DistributedContexts`` option to include all contexts: ```wl In[7]:= cty`mtest[n_] := Labeled[Framed[PrimeQ[2 ^ n - 1]], $KernelID] In[8]:= ParallelArray[cty`mtest, 10, DistributedContexts -> Automatic] Out[8]= {Labeled[False, 4], Labeled[True, 4], Labeled[True, 3], Labeled[False, 3], Labeled[True, 2], Labeled[False, 1], Labeled[True, 4], Labeled[False, 3], Labeled[False, 2], Labeled[False, 1]} ``` ## See Also * [`Array`](https://reference.wolfram.com/language/ref/Array.en.md) * [`ParallelTable`](https://reference.wolfram.com/language/ref/ParallelTable.en.md) * [`Parallelize`](https://reference.wolfram.com/language/ref/Parallelize.en.md) ## Related Guides * [Data Parallelism](https://reference.wolfram.com/language/guide/DataParallelism.en.md) ## History * [Introduced in 2008 (7.0)](https://reference.wolfram.com/language/guide/SummaryOfNewFeaturesIn70.en.md) \| [Updated in 2010 (8.0)](https://reference.wolfram.com/language/guide/SummaryOfNewFeaturesIn80.en.md) ▪ [2021 (13.0)](https://reference.wolfram.com/language/guide/SummaryOfNewFeaturesIn130.en.md)