##bbobECDFslegendrlbased## Bootstrapped empirical cumulative distribution of the number of f-evaluations divided by dimension (FEvals/DIM) for all functions and subgroups in -D. The targets are chosen from !!TARGET-RANGES-IN-ECDF!! such that !!THE-REF-ALG!! just not reached them within a given budget of k × DIM, with !!NUM−OF−TARGETS−IN−ECDF!! different values of k chosen equidistant in logscale within the interval {0.5, ..., 50}. As reference algorithm, !!THE-REF-ALG!! is shown as light thick line with diamond markers.
##bbobppfigslegendrlbased## Expected running time (ERT in number of f-evaluations as log10 value) divided by dimension versus dimension. The target function value is chosen such that !!THE-REF-ALG!! just failed to achieve an ERT of !!PPFIGS−FTARGET!!×DIM. Different symbols correspond to different algorithms given in the legend of f1 and f24. Light symbols give the maximum number of evaluations from the longest trial divided by dimension. Black stars (if present) indicate a better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendrlbased## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of number of f-evaluations divided by search space dimension D, to fall below fopt+∆f where ∆f is the target just not reached by the best algorithm from BBOB 2009 within a budget of k×DIM evaluations, where k is the first value in the legend. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆f for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... f-evaluations (from right to left cycling cyan-magenta-black...) and final ∆f-value (red), where ∆fand Df denote the difference to the optimal function value. Light brown lines in the background show ECDFs for the most difficult target of all algorithms benchmarked during BBOB-2009.
##bbobpprldistrlegendtworlbased## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 5-D (left) and 20-D (right). Left sub-columns: ECDF of the number of f-evaluations divided by dimension D (FEvals/D) to fall below fopt+∆f for algorithmA (°) and algorithmB (♦ ) where ∆f is the target just not reached by the best algorithm from BBOB 2009 within a budget of k×DIM f-evaluations, with k being the value in the legend. Right sub-columns:ECDF of f-evaluations ratiosof algorithmA divided by algorithmB for run-length-based targets; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legends indicate the target budget of k×DIM FEvals and, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendrlbased## Scaling of runtime with dimension to reach certain target values ∆f. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for targets just not reached by the best algorithm from BBOB 2009 within the given budget k×DIM, where k is shown in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. The light thick line with diamonds indicates the best algorithm from BBOB 2009 for the most difficult target. Slanted grid lines indicate a scaling with O(DIM) compared to O(1) when using the respective reference algorithm.
##bbobpptablecaptionrlbased## Expected runtime (ERT in number of f-evaluations) divided by the ERT of the best algorithm from BBOB 2009 in different dimensions. This ERT  ratio and, in braces as dispersion measure, the half difference between 90 and 10%-tile of (bootstrapped) run lengths appear in the second row of each cell, the best ERT  in the first. The different target ∆f-values are shown in the top row. #succ is the number of trials that reached the (final) target fopt + 10−8. The median number of conducted evaluations is additionally given in italics, if the target in the last column was never reached. Bold entries are better than the reference algorithm the best algorithm from BBOB 2009 with a p-value of at least 0.05 or 10−k where k is the number following the ↓ symbol (computed in the rank-sum test with Bonferroni correction by the number of functions 24).
##bbobpptablesmanylegendrlbased## Expected runtime (ERT in number of f-evaluations) divided by the respective best ERT measured during BBOB-2009 (when finite) in different dimensions. This ERT ratio and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) run lengths appear for each algorithm and run-length based target, the corresponding reference ERT  (preceded by the target ∆f-value in italics) in the first row. #succ is the number of trials that reached the target value of the last column. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. The best entry is marked by a star when the largest p-value of pairwise rank-sum tests with the other algorithms either obeys 0.01 < p ≤ 0.05 or 10−k−1 < p ≤ 10−k when the star is followed by the number k, with Bonferroni correction by the number of functions (24). A ↑ signifies the number of trials that were worse than the ERT of the best algorithm from BBOB 2009 shown only when less than 10 percent were worse and the ERT was better. Best results are printed in bold.
##bbobppscatterlegendrlbased## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! runlength-based target values for budgets between !!NBLOW!! and !!NBUP!! evaluations. Each runlength-based target !!F!!-value is chosen such that the ERTs of !!THE-REF-ALG!! for the given and a slightly easier target bracket the reference budget. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 2:+, 3:\triangledown, 5:∗, 10:°, 20:[¯], 40:\Diamond.
##bbobloglosstablecaptionrlbased## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target f-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if the best algorithm from BBOB 2009 reached a better target within the budget, divided by the ERT of the best algorithm from BBOB 2009 to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.
##bbobloglossfigurecaptionrlbased## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendfixed## Bootstrapped empirical cumulative distribution of the number of f-evaluations divided by dimension (FEvals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D. As reference algorithm, !!THE-REF-ALG!! is shown as light thick line with diamond markers.
##bbobppfigslegendfixed## Expected running time (ERT in number of f-evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f24. Light symbols give the maximum number of evaluations from the longest trial divided by dimension. Black stars (if present) indicate a better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendfixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of f-evaluations divided by search space dimension D, to fall below fopt+∆f with ∆f =10k, where k is the first value in the legend. The thick red line represents the most difficult target value fopt+ 10−8. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆f for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... f-evaluations (from right to left cycling cyan-magenta-black...) and final ∆f-value (red), where ∆fand Df denote the difference to the optimal function value. Light brown lines in the background show ECDFs for the most difficult target of all algorithms benchmarked during BBOB-2009.
##bbobpprldistrlegendtwofixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 5-D (left) and 20-D (right). Left sub-columns: ECDF of the number of f-evaluations divided by dimension D (FEvals/D) to reach a target value fopt+∆f with ∆f =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Light beige lines show the ECDF of evals for target value ∆f =10−8 of all algorithms benchmarked during BBOB-2009. Right sub-columns: ECDF of FEvals ratios of algorithmA divided by algorithmB for fixed target precision values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendfixed## Scaling of runtime with dimension to reach certain target values ∆f. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed target precision values of 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. The light thick line with diamonds indicates the best algorithm from BBOB 2009 for the most difficult target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionfixed## Expected runtime (ERT in number of f-evaluations) divided by the ERT of the best algorithm from BBOB 2009 in different dimensions. This ERT  ratio and, in braces as dispersion measure, the half difference between 90 and 10%-tile of (bootstrapped) run lengths appear in the second row of each cell, the best ERT  (preceded by the target ∆f-value in italics) in the first. #succ is the number of trials that reached the target value of the last column. The median number of conducted evaluations is additionally given in italics, if the target in the last column was never reached. Bold entries are better than the reference algorithm the best algorithm from BBOB 2009 with a p-value of at least 0.05 or 10−k where k is the number following the ↓ symbol (computed in the rank-sum test with Bonferroni correction by the number of functions 24).
##bbobpptablesmanylegendfixed## Expected runtime (ERT in number of f-evaluations) divided by the respective best ERT measured during BBOB-2009 (when finite) in different dimensions. This ERT ratio and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) run lengths appear for each algorithm and target, the corresponding reference ERT  in the first row. The different target ∆f-values are shown in the top row. #succ is the number of trials that reached the (final) target fopt+ 10−8. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. The best entry is marked by a star when the largest p-value of pairwise rank-sum tests with the other algorithms either obeys 0.01 < p ≤ 0.05 or 10−k−1 < p ≤ 10−k when the star is followed by the number k, with Bonferroni correction by the number of functions (24). A ↑ signifies the number of trials that were worse than the ERT of the best algorithm from BBOB 2009 shown only when less than 10 percent were worse and the ERT was better. Best results are printed in bold.
##bbobppscatterlegendfixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f24. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 2:+, 3:\triangledown, 5:∗, 10:°, 20:[¯], 40:\Diamond.
##bbobloglosstablecaptionfixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target f-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if the best algorithm from BBOB 2009 reached a better target within the budget, divided by the ERT of the best algorithm from BBOB 2009 to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.
##bbobloglossfigurecaptionfixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendbiobjfixed## Bootstrapped empirical cumulative distribution of the number of f-evaluations divided by dimension (FEvals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D. As reference algorithm, !!THE-REF-ALG!! is shown as light thick line with diamond markers.
##bbobppfigslegendbiobjfixed## Expected running time (ERT in number of f-evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f55. Light symbols give the maximum number of evaluations from the longest trial divided by dimension. Black stars (if present) indicate a better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendbiobjfixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of f-evaluations divided by search space dimension D, to fall below Iref+∆I with ∆I =10k, where k is the first value in the legend. The thick red line represents the most difficult target value Iref+ 10−5. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆I for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... f-evaluations (from right to left cycling cyan-magenta-black...) and final ∆I-value (red), where ∆Iand Df denote the difference to the optimal function value. Shown are aggregations over functions where the single objectives are in the same BBOB function class, as indicated on the left side and the aggregation over all 55 functions in the last row.
##bbobpprldistrlegendtwobiobjfixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 5-D (left) and 20-D (right). Left sub-columns: ECDF of the number of f-evaluations divided by dimension D (FEvals/D) to reach a target value Iref+∆I with ∆I =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Right sub-columns: ECDF of FEvals ratios of algorithmA divided by algorithmB for fixed target precision values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendbiobjfixed## Scaling of runtime with dimension to reach certain target values ∆I. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed target precision values of 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. The light thick line with diamonds indicates the best algorithm from BBOB 2016 for the most difficult target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionbiobjfixed## Expected runtime (ERT in number of f-evaluations) divided by the ERT of the best algorithm from BBOB 2016 in different dimensions. This ERT  ratio and, in braces as dispersion measure, the half difference between 90 and 10%-tile of run lengths appear in the second row of each cell, the best ERT  (preceded by the target ∆I-value in italics) in the first. #succ is the number of trials that reached the target value of the last column. The median number of conducted evaluations is additionally given in italics, if the target in the last column was never reached. Bold entries are better than the reference algorithm the best algorithm from BBOB 2016 with a p-value of at least 0.05 or 10−k where k is the number following the ↓ symbol (computed in the rank-sum test with Bonferroni correction by the number of functions 55).
##bbobpptablesmanylegendbiobjfixed## Expected runtime (ERT in number of f-evaluations) divided by the respective best ERT measured during BBOB-2016 (when finite) in different dimensions. This ERT ratio and, in braces as dispersion measure, the half difference between 10 and 90%-tile of run lengths appear for each algorithm and target, the corresponding reference ERT  in the first row. The different target ∆I-values are shown in the top row. #succ is the number of trials that reached the (final) target Iref+ 10−5. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. The best entry is marked by a star when the largest p-value of pairwise rank-sum tests with the other algorithms either obeys 0.01 < p ≤ 0.05 or 10−k−1 < p ≤ 10−k when the star is followed by the number k, with Bonferroni correction by the number of functions (55). A ↑ signifies the number of trials that were worse than the ERT of the best algorithm from BBOB 2016 shown only when less than 10 percent were worse and the ERT was better. Best results are printed in bold.
##bbobppscatterlegendbiobjfixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f55. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 2:+, 3:\triangledown, 5:∗, 10:°, 20:[¯], 40:\Diamond.
##bbobloglosstablecaptionbiobjfixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target IHVCOCO-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if the best algorithm from BBOB 2016 reached a better target within the budget, divided by the ERT of the best algorithm from BBOB 2016 to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.
##bbobloglossfigurecaptionbiobjfixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendbiobjrlbased## Bootstrapped empirical cumulative distribution of the number of f-evaluations divided by dimension (FEvals/DIM) for all functions and subgroups in -D. The targets are chosen from !!TARGET-RANGES-IN-ECDF!! such that !!THE-REF-ALG!! just not reached them within a given budget of k × DIM, with !!NUM−OF−TARGETS−IN−ECDF!! different values of k chosen equidistant in logscale within the interval {0.5, ..., 50}. As reference algorithm, !!THE-REF-ALG!! is shown as light thick line with diamond markers.
##bbobppfigslegendbiobjrlbased## Expected running time (ERT in number of f-evaluations as log10 value) divided by dimension versus dimension. The target function value is chosen such that !!THE-REF-ALG!! just failed to achieve an ERT of !!PPFIGS−FTARGET!!×DIM. Different symbols correspond to different algorithms given in the legend of f1 and f55. Light symbols give the maximum number of evaluations from the longest trial divided by dimension. Black stars (if present) indicate a better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendbiobjrlbased## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of number of f-evaluations divided by search space dimension D, to fall below Iref+∆I where ∆I is the target just not reached by the best algorithm from BBOB 2016 within a budget of k×DIM evaluations, where k is the first value in the legend. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆I for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... f-evaluations (from right to left cycling cyan-magenta-black...) and final ∆I-value (red), where ∆Iand Df denote the difference to the optimal function value. Shown are aggregations over functions where the single objectives are in the same BBOB function class, as indicated on the left side and the aggregation over all 55 functions in the last row.
##bbobpprldistrlegendtwobiobjrlbased## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 5-D (left) and 20-D (right). Left sub-columns: ECDF of the number of f-evaluations divided by dimension D (FEvals/D) to fall below Iref+∆I for algorithmA (°) and algorithmB (♦ ) where ∆I is the target just not reached by the best algorithm from BBOB 2016 within a budget of k×DIM f-evaluations, with k being the value in the legend. Right sub-columns:ECDF of f-evaluations ratiosof algorithmA divided by algorithmB for run-length-based targets; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legends indicate the target budget of k×DIM FEvals and, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendbiobjrlbased## Scaling of runtime with dimension to reach certain target values ∆I. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for targets just not reached by the best algorithm from BBOB 2016 within the given budget k×DIM, where k is shown in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. The light thick line with diamonds indicates the best algorithm from BBOB 2016 for the most difficult target. Slanted grid lines indicate a scaling with O(DIM) compared to O(1) when using the respective reference algorithm.
##bbobpptablecaptionbiobjrlbased## Expected runtime (ERT in number of f-evaluations) divided by the ERT of the best algorithm from BBOB 2016 in different dimensions. This ERT  ratio and, in braces as dispersion measure, the half difference between 90 and 10%-tile of run lengths appear in the second row of each cell, the best ERT  in the first. The different target ∆I-values are shown in the top row. #succ is the number of trials that reached the (final) target Iref + 10−5. The median number of conducted evaluations is additionally given in italics, if the target in the last column was never reached. Bold entries are better than the reference algorithm the best algorithm from BBOB 2016 with a p-value of at least 0.05 or 10−k where k is the number following the ↓ symbol (computed in the rank-sum test with Bonferroni correction by the number of functions 55).
##bbobpptablesmanylegendbiobjrlbased## Expected runtime (ERT in number of f-evaluations) divided by the respective best ERT measured during BBOB-2016 (when finite) in different dimensions. This ERT ratio and, in braces as dispersion measure, the half difference between 10 and 90%-tile of run lengths appear for each algorithm and run-length based target, the corresponding reference ERT  (preceded by the target ∆I-value in italics) in the first row. #succ is the number of trials that reached the target value of the last column. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. The best entry is marked by a star when the largest p-value of pairwise rank-sum tests with the other algorithms either obeys 0.01 < p ≤ 0.05 or 10−k−1 < p ≤ 10−k when the star is followed by the number k, with Bonferroni correction by the number of functions (55). A ↑ signifies the number of trials that were worse than the ERT of the best algorithm from BBOB 2016 shown only when less than 10 percent were worse and the ERT was better. Best results are printed in bold.
##bbobppscatterlegendbiobjrlbased## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! runlength-based target values for budgets between !!NBLOW!! and !!NBUP!! evaluations. Each runlength-based target !!F!!-value is chosen such that the ERTs of !!THE-REF-ALG!! for the given and a slightly easier target bracket the reference budget. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 2:+, 3:\triangledown, 5:∗, 10:°, 20:[¯], 40:\Diamond.
##bbobloglosstablecaptionbiobjrlbased## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target IHVCOCO-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if the best algorithm from BBOB 2016 reached a better target within the budget, divided by the ERT of the best algorithm from BBOB 2016 to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.
##bbobloglossfigurecaptionbiobjrlbased## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendbiobjextfixed## Bootstrapped empirical cumulative distribution of the number of f-evaluations divided by dimension (FEvals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D.
##bbobppfigslegendbiobjextfixed## Expected running time (ERT in number of f-evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f92. Light symbols give the maximum number of evaluations from the longest trial divided by dimension. Black stars (if present) indicate a better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendbiobjextfixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of f-evaluations divided by search space dimension D, to fall below Iref+∆I with ∆I =10k, where k is the first value in the legend. The thick red line represents the most difficult target value Iref+ 10−5. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆I for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... f-evaluations (from right to left cycling cyan-magenta-black...) and final ∆I-value (red), where ∆Iand Df denote the difference to the optimal function value. Shown are aggregations over functions where the single objectives are in the same BBOB function class, as indicated on the left side and the aggregation over all 92 functions in the last row.
##bbobpprldistrlegendtwobiobjextfixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 5-D (left) and 20-D (right). Left sub-columns: ECDF of the number of f-evaluations divided by dimension D (FEvals/D) to reach a target value Iref+∆I with ∆I =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Right sub-columns: ECDF of FEvals ratios of algorithmA divided by algorithmB for fixed target precision values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendbiobjextfixed## Scaling of runtime with dimension to reach certain target values ∆I. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed target precision values of 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionbiobjextfixed## Expected runtime (ERT) to reach given targets, measured in number of f-evaluations in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of runtimes is shown for the different target ∆I-values as shown in the top row. #succ is the number of trials that reached the last target Iref+ 10−5. The median number of conducted evaluations is additionally given in italics, if the target in the last column was never reached.
##bbobpptablesmanylegendbiobjextfixed## Expected runtime (ERT) to reach given targets, measured in number of f-evaluations, in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of runtimes is shown for the different target ∆I-values as shown in the top row. #succ is the number of trials that reached the last target Iref+ 10−5. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. The best entry is marked by a star when the largest p-value of pairwise rank-sum tests with the other algorithms either obeys 0.01 < p ≤ 0.05 or 10−k−1 < p ≤ 10−k when the star is followed by the number k, with Bonferroni correction by the number of functions (92). Best results are printed in bold.
##bbobppscatterlegendbiobjextfixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f92. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 2:+, 3:\triangledown, 5:∗, 10:°, 20:[¯], 40:\Diamond.
##bbobloglosstablecaptionbiobjextfixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target IHVCOCO-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if reached a better target within the budget, divided by the ERT of to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.
##bbobloglossfigurecaptionbiobjextfixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendconstrainedfixed## Bootstrapped empirical cumulative distribution of the number of evaluations divided by dimension (evals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D.
##bbobppfigslegendconstrainedfixed## Expected running time (ERT in number of evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f54. Light symbols give the maximum number of evaluations from the longest trial divided by dimension. Black stars (if present) indicate a better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendconstrainedfixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of evaluations divided by search space dimension D, to fall below fopt+∆f with ∆f =10k, where k is the first value in the legend. The thick red line represents the most difficult target value fopt+ 10−6. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆f for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... evaluations (from right to left cycling cyan-magenta-black...) and final ∆f-value (red), where ∆fand Df denote the difference to the optimal function value. Shown are aggregations over problems where the objective functions are in the same BBOB function class and the aggregation over all 48 functions in the last row.
##bbobpprldistrlegendtwoconstrainedfixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 5-D (left) and 20-D (right). Left sub-columns: ECDF of the number of evaluations divided by dimension D (evals/D) to reach a target value fopt+∆f with ∆f =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Right sub-columns: ECDF of evals ratios of algorithmA divided by algorithmB for fixed target precision values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendconstrainedfixed## Scaling of runtime with dimension to reach certain target values. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed target precision values of 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionconstrainedfixed## Expected runtime (ERT) to reach given targets, measured in number of evaluations in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target precision values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−6. The median number of conducted evaluations is additionally given in italics, if the target in the last column was never reached.
##bbobpptablesmanylegendconstrainedfixed## Expected runtime (ERT) to reach given targets, measured in number of evaluations, in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target precision values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−6. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. The best entry is marked by a star when the largest p-value of pairwise rank-sum tests with the other algorithms either obeys 0.01 < p ≤ 0.05 or 10−k−1 < p ≤ 10−k when the star is followed by the number k, with Bonferroni correction by the number of functions (54). Best results are printed in bold.
##bbobppscatterlegendconstrainedfixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f54. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 2:+, 3:\triangledown, 5:∗, 10:°, 20:[¯], 40:\Diamond.
##bbobloglosstablecaptionconstrainedfixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target f-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if reached a better target within the budget, divided by the ERT of to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.
##bbobloglossfigurecaptionconstrainedfixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobppfigconslegendconstrainedfixed## Expected running time (ERT in number of evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus number of constraints. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 - f54. If the success ratio is < 1/2 and > 0 the symbol is annotated with the number of successes. Light symbols give the maximum number of evaluations from the longest trial divided by dimension. Black stars (if present) indicate a better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of constraints (six).
##bbobppfigconsonelegendconstrainedfixed## Scaling of runtime with number of constraints to reach certain target values. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed target precision values of 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobECDFslegendlargescalefixed## Bootstrapped empirical cumulative distribution of the number of f-evaluations divided by dimension (FEvals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D.
##bbobppfigslegendlargescalefixed## Expected running time (ERT in number of f-evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f24. Light symbols give the maximum number of evaluations from the longest trial divided by dimension. Black stars (if present) indicate a better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendlargescalefixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of f-evaluations divided by search space dimension D, to fall below fopt+∆f with ∆f =10k, where k is the first value in the legend. The thick red line represents the most difficult target value fopt+ 10−8. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆f for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... f-evaluations (from right to left cycling cyan-magenta-black...) and final ∆f-value (red), where ∆fand Df denote the difference to the optimal function value.
##bbobpprldistrlegendtwolargescalefixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 80-D (left) and 320-D (right). Left sub-columns: ECDF of the number of f-evaluations divided by dimension D (FEvals/D) to reach a target value fopt+∆f with ∆f =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Right sub-columns: ECDF of FEvals ratios of algorithmA divided by algorithmB for fixed target precision values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendlargescalefixed## Scaling of runtime with dimension to reach certain target values ∆f. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed target precision values of 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionlargescalefixed## Expected runtime (ERT) to reach given targets, measured in number of f-evaluations in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−8. The median number of conducted evaluations is additionally given in italics, if the target in the last column was never reached.
##bbobpptablesmanylegendlargescalefixed## Expected runtime (ERT) to reach given targets, measured in number of f-evaluations, in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−8. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. The best entry is marked by a star when the largest p-value of pairwise rank-sum tests with the other algorithms either obeys 0.01 < p ≤ 0.05 or 10−k−1 < p ≤ 10−k when the star is followed by the number k, with Bonferroni correction by the number of functions (24). Best results are printed in bold.
##bbobppscatterlegendlargescalefixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f24. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 20:+, 40:\triangledown, 80:∗, 160:°, 320:[¯], 640:\Diamond.
##bbobloglosstablecaptionlargescalefixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target f-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if reached a better target within the budget, divided by the ERT of to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.
##bbobloglossfigurecaptionlargescalefixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendmixintfixed## Bootstrapped empirical cumulative distribution of the number of f-evaluations divided by dimension (FEvals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D.
##bbobppfigslegendmixintfixed## Expected running time (ERT in number of f-evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f24. Light symbols give the maximum number of evaluations from the longest trial divided by dimension. Black stars (if present) indicate a better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendmixintfixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of f-evaluations divided by search space dimension D, to fall below fopt+∆f with ∆f =10k, where k is the first value in the legend. The thick red line represents the most difficult target value fopt+ 10−8. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆f for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... f-evaluations (from right to left cycling cyan-magenta-black...) and final ∆f-value (red), where ∆fand Df denote the difference to the optimal function value.
##bbobpprldistrlegendtwomixintfixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 10-D (left) and 40-D (right). Left sub-columns: ECDF of the number of f-evaluations divided by dimension D (FEvals/D) to reach a target value fopt+∆f with ∆f =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Right sub-columns: ECDF of FEvals ratios of algorithmA divided by algorithmB for fixed target precision values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendmixintfixed## Scaling of runtime with dimension to reach certain target values ∆f. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed target precision values of 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionmixintfixed## Expected runtime (ERT) to reach given targets, measured in number of f-evaluations in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−8. The median number of conducted evaluations is additionally given in italics, if the target in the last column was never reached.
##bbobpptablesmanylegendmixintfixed## Expected runtime (ERT) to reach given targets, measured in number of f-evaluations, in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−8. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. The best entry is marked by a star when the largest p-value of pairwise rank-sum tests with the other algorithms either obeys 0.01 < p ≤ 0.05 or 10−k−1 < p ≤ 10−k when the star is followed by the number k, with Bonferroni correction by the number of functions (24). Best results are printed in bold.
##bbobppscatterlegendmixintfixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f24. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 5:+, 10:\triangledown, 20:∗, 40:°, 80:[¯], 160:\Diamond.
##bbobloglosstablecaptionmixintfixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target f-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if reached a better target within the budget, divided by the ERT of to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.
##bbobloglossfigurecaptionmixintfixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendbiobjmixintfixed## Bootstrapped empirical cumulative distribution of the number of f-evaluations divided by dimension (FEvals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D.
##bbobppfigslegendbiobjmixintfixed## Expected running time (ERT in number of f-evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f24. Light symbols give the maximum number of evaluations from the longest trial divided by dimension. Black stars (if present) indicate a better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendbiobjmixintfixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of f-evaluations divided by search space dimension D, to fall below fopt+∆f with ∆f =10k, where k is the first value in the legend. The thick red line represents the most difficult target value fopt+ 10−8. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆f for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... f-evaluations (from right to left cycling cyan-magenta-black...) and final ∆f-value (red), where ∆fand Df denote the difference to the optimal function value.
##bbobpprldistrlegendtwobiobjmixintfixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 10-D (left) and 40-D (right). Left sub-columns: ECDF of the number of f-evaluations divided by dimension D (FEvals/D) to reach a target value fopt+∆f with ∆f =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Right sub-columns: ECDF of FEvals ratios of algorithmA divided by algorithmB for fixed target precision values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendbiobjmixintfixed## Scaling of runtime with dimension to reach certain target values ∆f. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed target precision values of 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionbiobjmixintfixed## Expected runtime (ERT) to reach given targets, measured in number of f-evaluations in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−8. The median number of conducted evaluations is additionally given in italics, if the target in the last column was never reached.
##bbobpptablesmanylegendbiobjmixintfixed## Expected runtime (ERT) to reach given targets, measured in number of f-evaluations, in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−8. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. The best entry is marked by a star when the largest p-value of pairwise rank-sum tests with the other algorithms either obeys 0.01 < p ≤ 0.05 or 10−k−1 < p ≤ 10−k when the star is followed by the number k, with Bonferroni correction by the number of functions (24). Best results are printed in bold.
##bbobppscatterlegendbiobjmixintfixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f24. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 5:+, 10:\triangledown, 20:∗, 40:°, 80:[¯], 160:\Diamond.
##bbobloglosstablecaptionbiobjmixintfixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target f-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if reached a better target within the budget, divided by the ERT of to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.
##bbobloglossfigurecaptionbiobjmixintfixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendsboxcostfixed## Bootstrapped empirical cumulative distribution of the number of f-evaluations divided by dimension (FEvals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D.
##bbobppfigslegendsboxcostfixed## Expected running time (ERT in number of f-evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f24. Light symbols give the maximum number of evaluations from the longest trial divided by dimension. Black stars (if present) indicate a better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendsboxcostfixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of f-evaluations divided by search space dimension D, to fall below fopt+∆f with ∆f =10k, where k is the first value in the legend. The thick red line represents the most difficult target value fopt+ 10−8. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆f for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... f-evaluations (from right to left cycling cyan-magenta-black...) and final ∆f-value (red), where ∆fand Df denote the difference to the optimal function value.
##bbobpprldistrlegendtwosboxcostfixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 5-D (left) and 20-D (right). Left sub-columns: ECDF of the number of f-evaluations divided by dimension D (FEvals/D) to reach a target value fopt+∆f with ∆f =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Right sub-columns: ECDF of FEvals ratios of algorithmA divided by algorithmB for fixed target precision values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendsboxcostfixed## Scaling of runtime with dimension to reach certain target values ∆f. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed target precision values of 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionsboxcostfixed## Expected runtime (ERT) to reach given targets, measured in number of f-evaluations in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−8. The median number of conducted evaluations is additionally given in italics, if the target in the last column was never reached.
##bbobpptablesmanylegendsboxcostfixed## Expected runtime (ERT) to reach given targets, measured in number of f-evaluations, in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−8. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. The best entry is marked by a star when the largest p-value of pairwise rank-sum tests with the other algorithms either obeys 0.01 < p ≤ 0.05 or 10−k−1 < p ≤ 10−k when the star is followed by the number k, with Bonferroni correction by the number of functions (24). Best results are printed in bold.
##bbobppscatterlegendsboxcostfixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f24. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 2:+, 3:\triangledown, 5:∗, 10:°, 20:[¯], 40:\Diamond.
##bbobloglosstablecaptionsboxcostfixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target f-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if reached a better target within the budget, divided by the ERT of to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.
##bbobloglossfigurecaptionsboxcostfixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
###

File translated from TEX by TTH, version 4.16.
On 12 Jun 2024, 15:35.