Runaway process in -CURRENT

Thu Aug 24 18:27:50 UTC 2017

On Thu, Aug 24, 2017 at 8:15 PM, Dimitry Andric <dim at freebsd.org> wrote:
> On 24 Aug 2017, at 16:42, Fernando Apesteguía <fernando.apesteguia at gmail.com> wrote:
>>
>> On Thu, Aug 24, 2017 at 11:14 AM, Fernando Apesteguía
>> <fernando.apesteguia at gmail.com> wrote:
> ...
>> Just for the record, I got another email. Log URL:
>>
>> http://beefy11.nyi.freebsd.org/data/head-i386-default/p448640_s322824/logs/stepcode-0.8_1.log
> ...
>> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o -MF
>> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o.d -o
>> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o -c
>> schemas/sdai_wip210e3/SdaiAll.cc
>> ====>> Killing runaway build after 7200 seconds with no output
>
> Hm, I had not noticed two facts, one being that there are a great many
> copies of SdaiAll.cc files, which all seem to be generated, and the
> other that this is occurring on i386.  There are also other generated
> files, and specifically the compstructs.cc files can take extremely long
> to compile.
>
> On amd64, the longest compile time in the whole port is ~185 seconds,
> for a generated file of ~28000 lines:
>
> time     lines  filename
> =======  =====  ============================================================================================
>  184.90  28345  schemas/sdai_wip210e3/compstructs.cc
>  115.08    103  schemas/sdai_ap239/SdaiAP239_PRODUCT_LIFE_CYCLE_SUPPORT_ARM_LF_unity_types.cc
>  110.00    440  schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_AND_DESIGN_MIM_LF_unity_types.cc
>  106.70    292  schemas/sdai_wip210e3/SdaiAP210_ELECTRONIC_ASSEMBLY_INTERCONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_types.cc
>   90.73    290  schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGINEERING_MIM_LF_unity_types.cc
>   88.77   2232  schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_AND_DESIGN_MIM_LF_unity_entities.cc
>   81.87   2174  schemas/sdai_wip210e3/SdaiAP210_ELECTRONIC_ASSEMBLY_INTERCONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_entities.cc
>   73.92  19658  schemas/sdai_cd209/compstructs.cc
>   64.82   1734  schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGINEERING_MIM_LF_unity_entities.cc
>   61.42    189  schemas/sdai_ap210e2/SdaiAP210_ELECTRONIC_ASSEMBLY_INTERCONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_types.cc
>
> On i386, this time is enormously increased, especially for the files
> with lots of lines:
>
> time     lines  filename
> =======  =====  ============================================================================================
> 4809.23  28345  schemas/sdai_wip210e3/compstructs.cc
> 2164.37  19658  schemas/sdai_cd209/compstructs.cc
> 2056.54  15806  schemas/sdai_ap210e2/compstructs.cc
> 1545.19  16679  schemas/sdai_cd242/compstructs.cc
>  446.49   7667  schemas/sdai_ap203e2/compstructs.cc
>  273.96   6297  schemas/sdai_ap214e3/compstructs.cc
>  187.16    103  schemas/sdai_ap239/SdaiAP239_PRODUCT_LIFE_CYCLE_SUPPORT_ARM_LF_unity_types.cc
>  163.39    440  schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_AND_DESIGN_MIM_LF_unity_types.cc
>  130.02   2232  schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_AND_DESIGN_MIM_LF_unity_entities.cc
>  122.15    290  schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGINEERING_MIM_LF_unity_types.cc
>

I did some tests:

-CURRENT (i326) in VirtualBox, 1 CPU, 8Gb RAM

cd /usr/ports/cad/stepcode && time make

* gcc 6.4.0

real 27m17.867s
user 50m.0.428s
sys 2m47.619

* clang38

real 18m28.996s
user 33m22.790s
sys  1m22.711s

* clang 5.0.0

It takes a lot of time compiling
schemas/sdai_{wip210e3,ap210e2}/SdaiAll.cc and other SdaAll.cc files.
It's still compiling after several hours.

> The compstructs.cc and SdaiAll.cc files contain *extremely* large
> generated functions, in most cases embodying the whole file, even.  It
> seems that this is the cause for the long compile times.  A test case
> tarball with the preprocessed version of the top compstructs.cc file,
> and a shell script to run it, is here:
> http://www.andric.com/freebsd/clang/stepcode.tar.xz
>
> With clang 4.0.0, the compile times are more at the level of the amd64
> builds, even on i386.  So this regressed somewhere along the way to
> 5.0.0.  I will have to try to find out where, to see if there is an
> existing bug report or workaround.
>
> Note that in r321664, about three weeks ago, I already committed a
> couple of upstream fixes for some situations where compile times could
> get excessive (see also https://bugs.llvm.org/show_bug.cgi?id=33900),
> but it doesn't seem to have helped in this case.
>
> As a quick workaround, the port can be modified to compile these files
> using a lower optimization level.  If -O1 is used for the longest case,
> e.g. schemas/sdai_wip210e3/compstructs.cc, the compile time drops down
> to ~30s.  I don't think it will matter very much for the speed or size
> of the output.

I'll have a look at this.

Thanks!

>
> -Dimitry
>