Runaway process in -CURRENT

[Dimitry Andric]

On 24 Aug 2017, at 16:42, Fernando Apesteguía <fernando.apesteguia at gmail.com> wrote:
> 
> On Thu, Aug 24, 2017 at 11:14 AM, Fernando Apesteguía
> <fernando.apesteguia at gmail.com> wrote:
...
> Just for the record, I got another email. Log URL:
> 
> http://beefy11.nyi.freebsd.org/data/head-i386-default/p448640_s322824/logs/stepcode-0.8_1.log
...
> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o -MF
> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o.d -o
> schemas/sdai_wip210e3/CMakeFiles/sdai_wip210e3.dir/SdaiAll.cc.o -c
> schemas/sdai_wip210e3/SdaiAll.cc
> ====>> Killing runaway build after 7200 seconds with no output

Hm, I had not noticed two facts, one being that there are a great many
copies of SdaiAll.cc files, which all seem to be generated, and the
other that this is occurring on i386.  There are also other generated
files, and specifically the compstructs.cc files can take extremely long
to compile.

On amd64, the longest compile time in the whole port is ~185 seconds,
for a generated file of ~28000 lines:

time     lines  filename
=======  =====  ============================================================================================
 184.90  28345  schemas/sdai_wip210e3/compstructs.cc
 115.08    103  schemas/sdai_ap239/SdaiAP239_PRODUCT_LIFE_CYCLE_SUPPORT_ARM_LF_unity_types.cc
 110.00    440  schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_AND_DESIGN_MIM_LF_unity_types.cc
 106.70    292  schemas/sdai_wip210e3/SdaiAP210_ELECTRONIC_ASSEMBLY_INTERCONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_types.cc
  90.73    290  schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGINEERING_MIM_LF_unity_types.cc
  88.77   2232  schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_AND_DESIGN_MIM_LF_unity_entities.cc
  81.87   2174  schemas/sdai_wip210e3/SdaiAP210_ELECTRONIC_ASSEMBLY_INTERCONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_entities.cc
  73.92  19658  schemas/sdai_cd209/compstructs.cc
  64.82   1734  schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGINEERING_MIM_LF_unity_entities.cc
  61.42    189  schemas/sdai_ap210e2/SdaiAP210_ELECTRONIC_ASSEMBLY_INTERCONNECT_AND_PACKAGING_DESIGN_MIM_LF_unity_types.cc

On i386, this time is enormously increased, especially for the files
with lots of lines:

time     lines  filename
=======  =====  ============================================================================================
4809.23  28345  schemas/sdai_wip210e3/compstructs.cc
2164.37  19658  schemas/sdai_cd209/compstructs.cc
2056.54  15806  schemas/sdai_ap210e2/compstructs.cc
1545.19  16679  schemas/sdai_cd242/compstructs.cc
 446.49   7667  schemas/sdai_ap203e2/compstructs.cc
 273.96   6297  schemas/sdai_ap214e3/compstructs.cc
 187.16    103  schemas/sdai_ap239/SdaiAP239_PRODUCT_LIFE_CYCLE_SUPPORT_ARM_LF_unity_types.cc
 163.39    440  schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_AND_DESIGN_MIM_LF_unity_types.cc
 130.02   2232  schemas/sdai_cd209/SdaiAP209_MULTIDISCIPLINARY_ANALYSIS_AND_DESIGN_MIM_LF_unity_entities.cc
 122.15    290  schemas/sdai_cd242/SdaiAP242_MANAGED_MODEL_BASED_3D_ENGINEERING_MIM_LF_unity_types.cc

The compstructs.cc and SdaiAll.cc files contain *extremely* large
generated functions, in most cases embodying the whole file, even.  It
seems that this is the cause for the long compile times.  A test case
tarball with the preprocessed version of the top compstructs.cc file,
and a shell script to run it, is here:
http://www.andric.com/freebsd/clang/stepcode.tar.xz

With clang 4.0.0, the compile times are more at the level of the amd64
builds, even on i386.  So this regressed somewhere along the way to
5.0.0.  I will have to try to find out where, to see if there is an
existing bug report or workaround.

Note that in r321664, about three weeks ago, I already committed a
couple of upstream fixes for some situations where compile times could
get excessive (see also https://bugs.llvm.org/show_bug.cgi?id=33900),
but it doesn't seem to have helped in this case.

As a quick workaround, the port can be modified to compile these files
using a lower optimization level.  If -O1 is used for the longest case,
e.g. schemas/sdai_wip210e3/compstructs.cc, the compile time drops down
to ~30s.  I don't think it will matter very much for the speed or size
of the output.

-Dimitry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 223 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20170824/ea2cdb38/attachment.sig>