##bbobECDFslegendrlbased## Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension (FEvals/DIM) for all functions and subgroups in -D. The targets are chosen from !!TARGET-RANGES-IN-ECDF!! such that !!THE-REF-ALG!! just not reached them within a given budget of k × DIM, with !!NUM−OF−TARGETS−IN−ECDF!! different values of k chosen equidistant in logscale within the interval {0.5, ..., 50}. As reference algorithm, !!THE-REF-ALG!! is shown as light thick line with diamond markers.
##bbobppfigslegendrlbased## Expected running time (ERT in number of f-evaluations as log10 value) divided by dimension versus dimension. The target function value is chosen such that !!THE-REF-ALG!! just failed to achieve an ERT of !!PPFIGS−FTARGET!!×DIM. Different symbols correspond to different algorithms given in the legend of f1 and f24. Light symbols give the maximum number of function evaluations from the longest trial divided by dimension. Black stars indicate a statistically better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendrlbased## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of number of function evaluations (FEvals) divided by search space dimension D, to fall below fopt+∆f where ∆f is the target just not reached by the best algorithm from BBOB 2009 within a budget of k×DIM evaluations, where k is the first value in the legend. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆f for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... function evaluations (from right to left cycling cyan-magenta-black...) and final ∆f-value (red), where ∆fand Df denote the difference to the optimal function value. Light brown lines in the background show ECDFs for the most difficult target of all algorithms benchmarked during BBOB-2009.
##bbobpprldistrlegendtworlbased## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 5-D (left) and 20-D (right). Left sub-columns: ECDF of the number of function evaluations divided by dimension D (FEvals/D) to fall below fopt+∆f for algorithmA (°) and algorithmB (♦ ) where ∆f is the target just not reached by the best algorithm from BBOB 2009 within a budget of k×DIM evaluations, with k being the value in the legend. Right sub-columns: ECDF of FEval ratios of algorithmA divided by algorithmB for run-length-based targets; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legends indicate the target budget of k×DIM evaluations and, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendrlbased## Scaling of runtime with dimension to reach certain target values ∆f. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for targets just not reached by the best algorithm from BBOB 2009 within the given budget k×DIM, where k is shown in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. The light thick line with diamonds indicates the best algorithm from BBOB 2009 for the most difficult target. Slanted grid lines indicate a scaling with O(DIM) compared to O(1) when using the respective reference algorithm.
##bbobpptablecaptionrlbased## Expected running time (ERT in number of function evaluations) divided by the ERT of the best algorithm from BBOB 2009 in different dimensions. This ERT  ratio and, in braces as dispersion measure, the half difference between 90 and 10%-tile of bootstrapped run lengths appear in the second row of each cell, the best ERT  in the first. The different target ∆f-values are shown in the top row. #succ is the number of trials that reached the (final) target fopt + 10−8. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Bold entries are statistically significantly better (according to the rank-sum test) compared to the best algorithm from BBOB 2009, with p = 0.05 or p = 10−k when the number k > 1 is following the ↓ symbol, with Bonferroni correction by the number of functions (24).??COCOVERSION??
##bbobpptablesmanylegendrlbased## Expected runtime (ERT in number of function evaluations) divided by the respective best ERT measured during BBOB-2009 in different dimensions. This ERT ratio and, in braces as dispersion measure, the half difference between 10 and 90%-tile of bootstrapped run lengths appear for each algorithm and run-length based target, the corresponding reference ERT  (preceded by the target ∆f-value in italics) in the first row. #succ is the number of trials that reached the target value of the last column. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Entries, succeeded by a star, are statistically significantly better (according to the rank-sum test) when compared to all other algorithms of the table, with p = 0.05 or p = 10−k when the number k following the star is larger than 1, with Bonferroni correction by the number of functions (24). A ↓ indicates the same tested against the best algorithm from BBOB 2009. Best results are printed in bold. ??COCOVERSION??
##bbobppscatterlegendrlbased## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! runlength-based target values for budgets between !!NBLOW!! and !!NBUP!! evaluations. Each runlength-based target !!F!!-value is chosen such that the ERTs of !!THE-REF-ALG!! for the given and a slightly easier target bracket the reference budget. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 2:+, 3:\triangledown, 5:∗, 10:°, 20:[¯], 40:\Diamond.
##bbobloglosstablecaptionrlbased## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target f-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if the best algorithm from BBOB 2009 reached a better target within the budget, divided by the ERT of the best algorithm from BBOB 2009 to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.??COCOVERSION??
##bbobloglossfigurecaptionrlbased## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendfixed## Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension (FEvals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D. As reference algorithm, !!THE-REF-ALG!! is shown as light thick line with diamond markers.
##bbobppfigslegendfixed## Expected running time (ERT in number of f-evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f24. Light symbols give the maximum number of function evaluations from the longest trial divided by dimension. Black stars indicate a statistically better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendfixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of function evaluations (FEvals) divided by search space dimension D, to fall below fopt+∆f with ∆f =10k, where k is the first value in the legend. The thick red line represents the most difficult target value fopt+ 10−8. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆f for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... function evaluations (from right to left cycling cyan-magenta-black...) and final ∆f-value (red), where ∆fand Df denote the difference to the optimal function value. Light brown lines in the background show ECDFs for the most difficult target of all algorithms benchmarked during BBOB-2009.
##bbobpprldistrlegendtwofixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 5-D (left) and 20-D (right). Left sub-columns: ECDF of the number of function evaluations divided by dimension D (FEvals/D) to reach a target value fopt+∆f with ∆f =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Light beige lines show the ECDF of FEvals for target value ∆f =10−8 of all algorithms benchmarked during BBOB-2009. Right sub-columns: ECDF of FEval ratios of algorithmA divided by algorithmB for target function values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendfixed## Scaling of runtime with dimension to reach certain target values ∆f. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed values of ∆f = 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. The light thick line with diamonds indicates the best algorithm from BBOB 2009 for the most difficult target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionfixed## Expected running time (ERT in number of function evaluations) divided by the ERT of the best algorithm from BBOB 2009 in different dimensions. This ERT  ratio and, in braces as dispersion measure, the half difference between 90 and 10%-tile of bootstrapped run lengths appear in the second row of each cell, the best ERT  (preceded by the target ∆f-value in italics) in the first. #succ is the number of trials that reached the target value of the last column. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Bold entries are statistically significantly better (according to the rank-sum test) compared to the best algorithm from BBOB 2009, with p = 0.05 or p = 10−k when the number k > 1 is following the ↓ symbol, with Bonferroni correction by the number of functions (24).??COCOVERSION??
##bbobpptablesmanylegendfixed## Expected runtime (ERT in number of function evaluations) divided by the respective best ERT measured during BBOB-2009 in different dimensions. This ERT ratio and, in braces as dispersion measure, the half difference between 10 and 90%-tile of bootstrapped run lengths appear for each algorithm and target, the corresponding reference ERT  in the first row. The different target ∆f-values are shown in the top row. #succ is the number of trials that reached the (final) target fopt+ 10−8. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Entries, succeeded by a star, are statistically significantly better (according to the rank-sum test) when compared to all other algorithms of the table, with p = 0.05 or p = 10−k when the number k following the star is larger than 1, with Bonferroni correction by the number of functions (24). A ↓ indicates the same tested against the best algorithm from BBOB 2009. Best results are printed in bold. ??COCOVERSION??
##bbobppscatterlegendfixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f24. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 2:+, 3:\triangledown, 5:∗, 10:°, 20:[¯], 40:\Diamond.
##bbobloglosstablecaptionfixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target f-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if the best algorithm from BBOB 2009 reached a better target within the budget, divided by the ERT of the best algorithm from BBOB 2009 to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.??COCOVERSION??
##bbobloglossfigurecaptionfixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendbiobjfixed## Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension (FEvals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D. As reference algorithm, !!THE-REF-ALG!! is shown as light thick line with diamond markers.
##bbobppfigslegendbiobjfixed## Expected running time (ERT in number of f-evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f55. Light symbols give the maximum number of function evaluations from the longest trial divided by dimension. Black stars indicate a statistically better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendbiobjfixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of function evaluations (FEvals) divided by search space dimension D, to fall below Iref+∆I with ∆I =10k, where k is the first value in the legend. The thick red line represents the most difficult target value Iref+ 10−5. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆I for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... function evaluations (from right to left cycling cyan-magenta-black...) and final ∆I-value (red), where ∆Iand Df denote the difference to the optimal function value. Shown are aggregations over functions where the single objectives are in the same BBOB function class, as indicated on the left side and the aggregation over all 55 functions in the last row.
##bbobpprldistrlegendtwobiobjfixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 5-D (left) and 20-D (right). Left sub-columns: ECDF of the number of function evaluations divided by dimension D (FEvals/D) to reach a target value Iref+∆I with ∆I =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Right sub-columns: ECDF of FEval ratios of algorithmA divided by algorithmB for target function values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendbiobjfixed## Scaling of runtime with dimension to reach certain target values ∆I. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed values of ∆I = 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. The light thick line with diamonds indicates the best algorithm from BBOB 2016 for the most difficult target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionbiobjfixed## Expected running time (ERT in number of function evaluations) divided by the ERT of the best algorithm from BBOB 2016 in different dimensions. This ERT  ratio and, in braces as dispersion measure, the half difference between 90 and 10%-tile of bootstrapped run lengths appear in the second row of each cell, the best ERT  (preceded by the target ∆I-value in italics) in the first. #succ is the number of trials that reached the target value of the last column. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Bold entries are statistically significantly better (according to the rank-sum test) compared to the best algorithm from BBOB 2016, with p = 0.05 or p = 10−k when the number k > 1 is following the ↓ symbol, with Bonferroni correction by the number of functions (55).??COCOVERSION??
##bbobpptablesmanylegendbiobjfixed## Expected runtime (ERT in number of function evaluations) divided by the respective best ERT measured during BBOB-2016 in different dimensions. This ERT ratio and, in braces as dispersion measure, the half difference between 10 and 90%-tile of bootstrapped run lengths appear for each algorithm and target, the corresponding reference ERT  in the first row. The different target ∆I-values are shown in the top row. #succ is the number of trials that reached the (final) target Iref+ 10−5. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Entries, succeeded by a star, are statistically significantly better (according to the rank-sum test) when compared to all other algorithms of the table, with p = 0.05 or p = 10−k when the number k following the star is larger than 1, with Bonferroni correction by the number of functions (55). A ↓ indicates the same tested against the best algorithm from BBOB 2016. Best results are printed in bold. ??COCOVERSION??
##bbobppscatterlegendbiobjfixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f55. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 2:+, 3:\triangledown, 5:∗, 10:°, 20:[¯], 40:\Diamond.
##bbobloglosstablecaptionbiobjfixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target IHVCOCO-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if the best algorithm from BBOB 2016 reached a better target within the budget, divided by the ERT of the best algorithm from BBOB 2016 to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.??COCOVERSION??
##bbobloglossfigurecaptionbiobjfixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendbiobjrlbased## Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension (FEvals/DIM) for all functions and subgroups in -D. The targets are chosen from !!TARGET-RANGES-IN-ECDF!! such that !!THE-REF-ALG!! just not reached them within a given budget of k × DIM, with !!NUM−OF−TARGETS−IN−ECDF!! different values of k chosen equidistant in logscale within the interval {0.5, ..., 50}. As reference algorithm, !!THE-REF-ALG!! is shown as light thick line with diamond markers.
##bbobppfigslegendbiobjrlbased## Expected running time (ERT in number of f-evaluations as log10 value) divided by dimension versus dimension. The target function value is chosen such that !!THE-REF-ALG!! just failed to achieve an ERT of !!PPFIGS−FTARGET!!×DIM. Different symbols correspond to different algorithms given in the legend of f1 and f55. Light symbols give the maximum number of function evaluations from the longest trial divided by dimension. Black stars indicate a statistically better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendbiobjrlbased## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of number of function evaluations (FEvals) divided by search space dimension D, to fall below Iref+∆I where ∆I is the target just not reached by the best algorithm from BBOB 2016 within a budget of k×DIM evaluations, where k is the first value in the legend. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆I for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... function evaluations (from right to left cycling cyan-magenta-black...) and final ∆I-value (red), where ∆Iand Df denote the difference to the optimal function value. Shown are aggregations over functions where the single objectives are in the same BBOB function class, as indicated on the left side and the aggregation over all 55 functions in the last row.
##bbobpprldistrlegendtwobiobjrlbased## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 5-D (left) and 20-D (right). Left sub-columns: ECDF of the number of function evaluations divided by dimension D (FEvals/D) to fall below Iref+∆I for algorithmA (°) and algorithmB (♦ ) where ∆I is the target just not reached by the best algorithm from BBOB 2016 within a budget of k×DIM evaluations, with k being the value in the legend. Right sub-columns: ECDF of FEval ratios of algorithmA divided by algorithmB for run-length-based targets; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legends indicate the target budget of k×DIM evaluations and, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendbiobjrlbased## Scaling of runtime with dimension to reach certain target values ∆I. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for targets just not reached by the best algorithm from BBOB 2016 within the given budget k×DIM, where k is shown in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. The light thick line with diamonds indicates the best algorithm from BBOB 2016 for the most difficult target. Slanted grid lines indicate a scaling with O(DIM) compared to O(1) when using the respective reference algorithm.
##bbobpptablecaptionbiobjrlbased## Expected running time (ERT in number of function evaluations) divided by the ERT of the best algorithm from BBOB 2016 in different dimensions. This ERT  ratio and, in braces as dispersion measure, the half difference between 90 and 10%-tile of bootstrapped run lengths appear in the second row of each cell, the best ERT  in the first. The different target ∆I-values are shown in the top row. #succ is the number of trials that reached the (final) target Iref + 10−5. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Bold entries are statistically significantly better (according to the rank-sum test) compared to the best algorithm from BBOB 2016, with p = 0.05 or p = 10−k when the number k > 1 is following the ↓ symbol, with Bonferroni correction by the number of functions (55).??COCOVERSION??
##bbobpptablesmanylegendbiobjrlbased## Expected runtime (ERT in number of function evaluations) divided by the respective best ERT measured during BBOB-2016 in different dimensions. This ERT ratio and, in braces as dispersion measure, the half difference between 10 and 90%-tile of bootstrapped run lengths appear for each algorithm and run-length based target, the corresponding reference ERT  (preceded by the target ∆I-value in italics) in the first row. #succ is the number of trials that reached the target value of the last column. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Entries, succeeded by a star, are statistically significantly better (according to the rank-sum test) when compared to all other algorithms of the table, with p = 0.05 or p = 10−k when the number k following the star is larger than 1, with Bonferroni correction by the number of functions (55). A ↓ indicates the same tested against the best algorithm from BBOB 2016. Best results are printed in bold. ??COCOVERSION??
##bbobppscatterlegendbiobjrlbased## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! runlength-based target values for budgets between !!NBLOW!! and !!NBUP!! evaluations. Each runlength-based target !!F!!-value is chosen such that the ERTs of !!THE-REF-ALG!! for the given and a slightly easier target bracket the reference budget. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 2:+, 3:\triangledown, 5:∗, 10:°, 20:[¯], 40:\Diamond.
##bbobloglosstablecaptionbiobjrlbased## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target IHVCOCO-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if the best algorithm from BBOB 2016 reached a better target within the budget, divided by the ERT of the best algorithm from BBOB 2016 to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.??COCOVERSION??
##bbobloglossfigurecaptionbiobjrlbased## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendbiobjextfixed## Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension (FEvals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D.
##bbobppfigslegendbiobjextfixed## Expected running time (ERT in number of f-evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f92. Light symbols give the maximum number of function evaluations from the longest trial divided by dimension. Black stars indicate a statistically better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendbiobjextfixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of function evaluations (FEvals) divided by search space dimension D, to fall below Iref+∆I with ∆I =10k, where k is the first value in the legend. The thick red line represents the most difficult target value Iref+ 10−5. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆I for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... function evaluations (from right to left cycling cyan-magenta-black...) and final ∆I-value (red), where ∆Iand Df denote the difference to the optimal function value. Shown are aggregations over functions where the single objectives are in the same BBOB function class, as indicated on the left side and the aggregation over all 92 functions in the last row.
##bbobpprldistrlegendtwobiobjextfixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 5-D (left) and 20-D (right). Left sub-columns: ECDF of the number of function evaluations divided by dimension D (FEvals/D) to reach a target value Iref+∆I with ∆I =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Right sub-columns: ECDF of FEval ratios of algorithmA divided by algorithmB for target function values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendbiobjextfixed## Scaling of runtime with dimension to reach certain target values ∆I. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed values of ∆I = 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionbiobjextfixed## Expected runtime (ERT) to reach given targets, measured in number of function evaluations in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆I-values as shown in the top row. #succ is the number of trials that reached the last target Iref+ 10−5. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached.
##bbobpptablesmanylegendbiobjextfixed## Expected runtime (ERT) to reach given targets, measured in number of function evaluations, in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆I-values as shown in the top row. #succ is the number of trials that reached the last target Iref+ 10−5. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Entries, succeeded by a star, are statistically significantly better (according to the rank-sum test) when compared to all other algorithms of the table, with p = 0.05 or p = 10−k when the number k following the star is larger than 1, with Bonferroni correction by the number of functions (92). Best results are printed in bold. ??COCOVERSION??
##bbobppscatterlegendbiobjextfixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f92. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 2:+, 3:\triangledown, 5:∗, 10:°, 20:[¯], 40:\Diamond.
##bbobloglosstablecaptionbiobjextfixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target IHVCOCO-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if reached a better target within the budget, divided by the ERT of to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.??COCOVERSION??
##bbobloglossfigurecaptionbiobjextfixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendconstrainedfixed## Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension (FEvals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D.
##bbobppfigslegendconstrainedfixed## Expected running time (ERT in number of f-evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f48. Light symbols give the maximum number of function evaluations from the longest trial divided by dimension. Black stars indicate a statistically better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendconstrainedfixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of function evaluations ((f+g)-evals) divided by search space dimension D, to fall below fopt+∆f with ∆f =10k, where k is the first value in the legend. The thick red line represents the most difficult target value fopt+ 10−6. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆f for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... function evaluations (from right to left cycling cyan-magenta-black...) and final ∆f-value (red), where ∆fand Df denote the difference to the optimal function value. Shown are aggregations over problems where the objective functions are in the same BBOB function class and the aggregation over all 48 functions in the last row.
##bbobpprldistrlegendtwoconstrainedfixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 5-D (left) and 20-D (right). Left sub-columns: ECDF of the number of function evaluations divided by dimension D (FEvals/D) to reach a target value fopt+∆f with ∆f =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Right sub-columns: ECDF of FEval ratios of algorithmA divided by algorithmB for target function values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendconstrainedfixed## Scaling of runtime with dimension to reach certain target values ∆f. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed values of ∆f = 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionconstrainedfixed## Expected runtime (ERT) to reach given targets, measured in number of function evaluations in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−6. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached.
##bbobpptablesmanylegendconstrainedfixed## Expected runtime (ERT) to reach given targets, measured in number of function evaluations, in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−6. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Entries, succeeded by a star, are statistically significantly better (according to the rank-sum test) when compared to all other algorithms of the table, with p = 0.05 or p = 10−k when the number k following the star is larger than 1, with Bonferroni correction by the number of functions (48). A ↓ indicates the same tested against . Best results are printed in bold. ??COCOVERSION??
##bbobppscatterlegendconstrainedfixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f48. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 2:+, 3:\triangledown, 5:∗, 10:°, 20:[¯], 40:\Diamond.
##bbobloglosstablecaptionconstrainedfixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target f-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if reached a better target within the budget, divided by the ERT of to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.??COCOVERSION??
##bbobloglossfigurecaptionconstrainedfixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendlargescalefixed## Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension (FEvals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D.
##bbobppfigslegendlargescalefixed## Expected running time (ERT in number of f-evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f24. Light symbols give the maximum number of function evaluations from the longest trial divided by dimension. Black stars indicate a statistically better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendlargescalefixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of function evaluations (FEvals) divided by search space dimension D, to fall below fopt+∆f with ∆f =10k, where k is the first value in the legend. The thick red line represents the most difficult target value fopt+ 10−8. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆f for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... function evaluations (from right to left cycling cyan-magenta-black...) and final ∆f-value (red), where ∆fand Df denote the difference to the optimal function value.
##bbobpprldistrlegendtwolargescalefixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 80-D (left) and 320-D (right). Left sub-columns: ECDF of the number of function evaluations divided by dimension D (FEvals/D) to reach a target value fopt+∆f with ∆f =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Right sub-columns: ECDF of FEval ratios of algorithmA divided by algorithmB for target function values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendlargescalefixed## Scaling of runtime with dimension to reach certain target values ∆f. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed values of ∆f = 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionlargescalefixed## Expected runtime (ERT) to reach given targets, measured in number of function evaluations in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−8. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached.
##bbobpptablesmanylegendlargescalefixed## Expected runtime (ERT) to reach given targets, measured in number of function evaluations, in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−8. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Entries, succeeded by a star, are statistically significantly better (according to the rank-sum test) when compared to all other algorithms of the table, with p = 0.05 or p = 10−k when the number k following the star is larger than 1, with Bonferroni correction by the number of functions (24). Best results are printed in bold. ??COCOVERSION??
##bbobppscatterlegendlargescalefixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f24. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 20:+, 40:\triangledown, 80:∗, 160:°, 320:[¯], 640:\Diamond.
##bbobloglosstablecaptionlargescalefixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target f-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if reached a better target within the budget, divided by the ERT of to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.??COCOVERSION??
##bbobloglossfigurecaptionlargescalefixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendmixintfixed## Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension (FEvals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D.
##bbobppfigslegendmixintfixed## Expected running time (ERT in number of f-evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f24. Light symbols give the maximum number of function evaluations from the longest trial divided by dimension. Black stars indicate a statistically better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendmixintfixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of function evaluations (FEvals) divided by search space dimension D, to fall below fopt+∆f with ∆f =10k, where k is the first value in the legend. The thick red line represents the most difficult target value fopt+ 10−8. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆f for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... function evaluations (from right to left cycling cyan-magenta-black...) and final ∆f-value (red), where ∆fand Df denote the difference to the optimal function value.
##bbobpprldistrlegendtwomixintfixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 10-D (left) and 40-D (right). Left sub-columns: ECDF of the number of function evaluations divided by dimension D (FEvals/D) to reach a target value fopt+∆f with ∆f =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Right sub-columns: ECDF of FEval ratios of algorithmA divided by algorithmB for target function values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendmixintfixed## Scaling of runtime with dimension to reach certain target values ∆f. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed values of ∆f = 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionmixintfixed## Expected runtime (ERT) to reach given targets, measured in number of function evaluations in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−8. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached.
##bbobpptablesmanylegendmixintfixed## Expected runtime (ERT) to reach given targets, measured in number of function evaluations, in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−8. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Entries, succeeded by a star, are statistically significantly better (according to the rank-sum test) when compared to all other algorithms of the table, with p = 0.05 or p = 10−k when the number k following the star is larger than 1, with Bonferroni correction by the number of functions (24). Best results are printed in bold. ??COCOVERSION??
##bbobppscatterlegendmixintfixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f24. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 5:+, 10:\triangledown, 20:∗, 40:°, 80:[¯], 160:\Diamond.
##bbobloglosstablecaptionmixintfixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target f-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if reached a better target within the budget, divided by the ERT of to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.??COCOVERSION??
##bbobloglossfigurecaptionmixintfixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
##bbobECDFslegendbiobjmixintfixed## Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension (FEvals/DIM) for !!NUM−OF−TARGETS−IN−ECDF!! targets with target precision in !!TARGET-RANGES-IN-ECDF!! for all functions and subgroups in -D.
##bbobppfigslegendbiobjmixintfixed## Expected running time (ERT in number of f-evaluations as log10 value), divided by dimension for target function value !!PPFIGS−FTARGET!! versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of f1 and f24. Light symbols give the maximum number of function evaluations from the longest trial divided by dimension. Black stars indicate a statistically better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six).
##bbobpprldistrlegendbiobjmixintfixed## Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of the number of function evaluations (FEvals) divided by search space dimension D, to fall below fopt+∆f with ∆f =10k, where k is the first value in the legend. The thick red line represents the most difficult target value fopt+ 10−8. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆f for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,... function evaluations (from right to left cycling cyan-magenta-black...) and final ∆f-value (red), where ∆fand Df denote the difference to the optimal function value.
##bbobpprldistrlegendtwobiobjmixintfixed## Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 10-D (left) and 40-D (right). Left sub-columns: ECDF of the number of function evaluations divided by dimension D (FEvals/D) to reach a target value fopt+∆f with ∆f =10k, where k is given by the first value in the legend, for algorithmA (°) and algorithmB (♦) . Right sub-columns: ECDF of FEval ratios of algorithmA divided by algorithmB for target function values 10k with k given in the legend; all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > 0 or < 1. The legend also indicates, after the colon, the number of functions that were solved in at least one trial (algorithmA first).
##bbobppfigdimlegendbiobjmixintfixed## Scaling of runtime with dimension to reach certain target values ∆f. Lines: expected runtime (ERT); Cross (+): median runtime of successful runs to reach the most difficult target that was reached at least once (but not always); Cross (×): maximum number of f-evaluations in any trial. Notched boxes: interquartile range with median of simulated runs; All values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for fixed values of ∆f = 10k with k given in the legend. Numbers above ERT-symbols (if appearing) indicate the number of trials reaching the respective target. Horizontal lines mean linear scaling, slanted grid lines depict quadratic scaling.
##bbobpptablecaptionbiobjmixintfixed## Expected runtime (ERT) to reach given targets, measured in number of function evaluations in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−8. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached.
##bbobpptablesmanylegendbiobjmixintfixed## Expected runtime (ERT) to reach given targets, measured in number of function evaluations, in different dimensions. For each function, the ERT  and, in braces as dispersion measure, the half difference between 10 and 90%-tile of (bootstrapped) runtimes is shown for the different target ∆f-values as shown in the top row. #succ is the number of trials that reached the last target fopt+ 10−8. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Entries, succeeded by a star, are statistically significantly better (according to the rank-sum test) when compared to all other algorithms of the table, with p = 0.05 or p = 10−k when the number k following the star is larger than 1, with Bonferroni correction by the number of functions (24). Best results are printed in bold. ??COCOVERSION??
##bbobppscatterlegendbiobjmixintfixed## Expected running time (ERT in log10 of number of function evaluations) of algorithmA (y-axis) versus algorithmB (x-axis) for !!NBTARGETS−SCATTER!! target values !!DF!! ∈ [!!NBLOW!!, !!NBUP!!] in each dimension on functions f1 - f24. Markers on the upper or right edge indicate that the respective target value was never reached. Markers represent dimension: 5:+, 10:\triangledown, 20:∗, 40:°, 80:[¯], 160:\Diamond.
##bbobloglosstablecaptionbiobjmixintfixed## ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target f-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if reached a better target within the budget, divided by the ERT of to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.??COCOVERSION??
##bbobloglossfigurecaptionbiobjmixintfixed## ERT loss ratios (see the previous figure for details).
Each cross (+) represents a single function, the line is the geometric mean.
###

File translated from TEX by TTH, version 4.08.
On 30 Jan 2020, 13:26.