Runaway process in -CURRENT

Fri Aug 25 15:10:44 UTC 2017

On Thu, Aug 24, 2017 at 8:27 PM, Fernando Apesteguía
<fernando.apesteguia at gmail.com> wrote:
> On Thu, Aug 24, 2017 at 8:15 PM, Dimitry Andric <dim at freebsd.org> wrote:
>> On 24 Aug 2017, at 16:42, Fernando Apesteguía <fernando.apesteguia at gmail.com> wrote:
>>>
>>> On Thu, Aug 24, 2017 at 11:14 AM, Fernando Apesteguía
>>> <fernando.apesteguia at gmail.com> wrote:
>> ...
>>> Just for the record, I got another email. Log URL:
>>>
>>> http://beefy11.nyi.freebsd.org/data/head-i386-default/p448640_s322824/logs/stepcode-0.8_1.log
>> ...
>>> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o -MF
>>> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o.d -o
>>> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o -c
>>> schemas/sdai_wip210e3/SdaiAll.cc
>>> ====>> Killing runaway build after 7200 seconds with no output
>>
>> Hm, I had not noticed two facts, one being that there are a great many
>> copies of SdaiAll.cc files, which all seem to be generated, and the
>> other that this is occurring on i386.  There are also other generated
>> files, and specifically the compstructs.cc files can take extremely long
>> to compile.
>>
>> On amd64, the longest compile time in the whole port is ~185 seconds,
>> for a generated file of ~28000 lines:
>>
>> time     lines  filename
>> =======  =====  ============================================================================================
>>  184.90  28345  schemas/sdai_wip210e3/compstructs.cc
>>  115.08    103  schemas/sdai_ap239/SdaiAP239_PRODUCT_LIFE_CYCLE_SUPPORT_ARM_LF_unity_types.cc
>>  110.00    440  schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_AND_DESIGN_MIM_LF_unity_types.cc
>>  106.70    292  schemas/sdai_wip210e3/SdaiAP210_ELECTRONIC_ASSEMBLY_INTERCONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_types.cc
>>   90.73    290  schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGINEERING_MIM_LF_unity_types.cc
>>   88.77   2232  schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_AND_DESIGN_MIM_LF_unity_entities.cc
>>   81.87   2174  schemas/sdai_wip210e3/SdaiAP210_ELECTRONIC_ASSEMBLY_INTERCONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_entities.cc
>>   73.92  19658  schemas/sdai_cd209/compstructs.cc
>>   64.82   1734  schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGINEERING_MIM_LF_unity_entities.cc
>>   61.42    189  schemas/sdai_ap210e2/SdaiAP210_ELECTRONIC_ASSEMBLY_INTERCONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_types.cc
>>
>> On i386, this time is enormously increased, especially for the files
>> with lots of lines:
>>
>> time     lines  filename
>> =======  =====  ============================================================================================
>> 4809.23  28345  schemas/sdai_wip210e3/compstructs.cc
>> 2164.37  19658  schemas/sdai_cd209/compstructs.cc
>> 2056.54  15806  schemas/sdai_ap210e2/compstructs.cc
>> 1545.19  16679  schemas/sdai_cd242/compstructs.cc
>>  446.49   7667  schemas/sdai_ap203e2/compstructs.cc
>>  273.96   6297  schemas/sdai_ap214e3/compstructs.cc
>>  187.16    103  schemas/sdai_ap239/SdaiAP239_PRODUCT_LIFE_CYCLE_SUPPORT_ARM_LF_unity_types.cc
>>  163.39    440  schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_AND_DESIGN_MIM_LF_unity_types.cc
>>  130.02   2232  schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_AND_DESIGN_MIM_LF_unity_entities.cc
>>  122.15    290  schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGINEERING_MIM_LF_unity_types.cc
>>
>
> I did some tests:
>
> -CURRENT (i326) in VirtualBox, 1 CPU, 8Gb RAM
>
> cd /usr/ports/cad/stepcode && time make
>
> * gcc 6.4.0
>
> real 27m17.867s
> user 50m.0.428s
> sys 2m47.619
>
>
> * clang38
>
> real 18m28.996s
> user 33m22.790s
> sys  1m22.711s
>
> * clang 5.0.0
>
> It takes a lot of time compiling
> schemas/sdai_{wip210e3,ap210e2}/SdaiAll.cc and other SdaAll.cc files.
> It's still compiling after several hours.
>
>
>> The compstructs.cc and SdaiAll.cc files contain *extremely* large
>> generated functions, in most cases embodying the whole file, even.  It
>> seems that this is the cause for the long compile times.  A test case
>> tarball with the preprocessed version of the top compstructs.cc file,
>> and a shell script to run it, is here:
>> http://www.andric.com/freebsd/clang/stepcode.tar.xz
>>
>> With clang 4.0.0, the compile times are more at the level of the amd64
>> builds, even on i386.  So this regressed somewhere along the way to
>> 5.0.0.  I will have to try to find out where, to see if there is an
>> existing bug report or workaround.
>>
>> Note that in r321664, about three weeks ago, I already committed a
>> couple of upstream fixes for some situations where compile times could
>> get excessive (see also https://bugs.llvm.org/show_bug.cgi?id=33900),
>> but it doesn't seem to have helped in this case.
>>
>> As a quick workaround, the port can be modified to compile these files
>> using a lower optimization level.  If -O1 is used for the longest case,
>> e.g. schemas/sdai_wip210e3/compstructs.cc, the compile time drops down
>> to ~30s.  I don't think it will matter very much for the speed or size
>> of the output.
>
> I'll have a look at this.

Is this problem going to be reported upstream?

>
> Thanks!
>
>>
>> -Dimitry
>>