Unknown error after AlignmentTrackChi2

1

Dear Corry experts,

I’m reconstructing data from a DESY test beam, and I’m having a problem during the alignment. Please see below (the log file didn’t include the error, that’s why the copy-paste from the terminal). I don’t know what the problem can be. I have checked the space available, and its more than 50Gb. Any thoughts?
Thanks in advance,
Arturo

[rodrigar@lxplus773 bin]$ ./corry -c …/…/ITk-stuff/alignment_mimosa26-itk.conf
|15:42:53.755| (STATUS) Welcome to Corryvreckan v1.0+482^g9187f93
|15:42:53.757| (STATUS) Loaded 7 detectors
|15:42:53.759| (WARNING) Main ROOT file /afs/cern.ch/work/r/rodrigar/Corry_Stuff/corryvreckan/ITk-stuff/ouput/histograms_run427.root exists and will be overwritten.
|15:42:56.079| (STATUS) Loading module AlignmentTrackChi2
|15:42:56.080| (INFO) [C:AlignmentTrackChi2] Aligning positions
|15:42:56.080| (INFO) [C:AlignmentTrackChi2] Aligning telescope
|15:42:56.080| (STATUS) Loaded 23 module instances
|15:42:56.081| (STATUS) =================| Initialising modules |==================
|15:42:57.938| (STATUS) [I:ClusteringSpatial:TLU_0] Initialising “ClusteringSpatial:TLU_0” Warning in TH1::TH1: nbins is <=0 - set to nbins = 1
Warning in TH2::TH2: nbinsy is <=0 - set to nbinsy = 1
|15:42:58.154| (STATUS) [I:Correlations:MIMOSA26_0] Initialising “Correlations:MIMOSA26_0”
|15:42:58.154| (WARNING) [I:Correlations:MIMOSA26_0] [further messages will be suppressed] Correlations module is enabled and will significantly increase the runtime
|15:42:58.694| (STATUS) [I:AlignmentTrackChi2] Initialising “AlignmentTrackChi2”
|15:42:58.695| (STATUS) ========================| Event loop |========================
|15:44:31.242| (STATUS) Ev: 19.9k Px: 3.23M Tr: 67.1k (3.37/ev) t = 76.4066s
|15:44:31.784| (STATUS) ===================| Finalising modules |===================
|15:44:31.784| (INFO) [F:EventLoaderEUDAQ2:TLU_0] Found 0 hits in the data.
|15:44:32.396| (INFO) [F:EventLoaderEUDAQ2:MIMOSA26_0] Found 266667 hits in the data.
|15:44:33.264| (INFO) [F:EventLoaderEUDAQ2:MIMOSA26_1] Found 984846 hits in the data.
|15:44:34.113| (INFO) [F:EventLoaderEUDAQ2:MIMOSA26_2] Found 786706 hits in the data.
|15:44:34.878| (INFO) [F:EventLoaderEUDAQ2:MIMOSA26_3] Found 306901 hits in the data.
|15:44:35.658| (INFO) [F:EventLoaderEUDAQ2:MIMOSA26_4] Found 257819 hits in the data.
|15:44:36.403| (INFO) [F:EventLoaderEUDAQ2:MIMOSA26_5] Found 645079 hits in the data.
|15:44:41.549| (STATUS) [F:AlignmentTrackChi2] Alignment iteration 1 of 3
|15:44:47.456| (INFO) [F:AlignmentTrackChi2] MIMOSA26_1/0 dT(-53.119um,52.333um,0) dR(0,0,0)
|15:44:53.757| (INFO) [F:AlignmentTrackChi2] MIMOSA26_2/0 dT(117.503um,-34.741um,0) dR(0,0,0)
|15:45:02.058| (INFO) [F:AlignmentTrackChi2] MIMOSA26_3/0 dT(84.022um,-23.199um,0) dR(0,0,0)
|15:45:08.009| (INFO) [F:AlignmentTrackChi2] MIMOSA26_4/0 dT(141.422um,-4.674um,0) dR(0,0,0)
|15:45:12.674| (INFO) [F:AlignmentTrackChi2] MIMOSA26_5/0 dT(-49.323um,6.99um,0) dR(0,0,0)
|15:45:12.674| (STATUS) [F:AlignmentTrackChi2] Alignment iteration 2 of 3
|15:45:19.199| (INFO) [F:AlignmentTrackChi2] MIMOSA26_1/1 dT(55.088um,-12.496um,0) dR(0,0,0)
|15:45:29.384| (INFO) [F:AlignmentTrackChi2] MIMOSA26_2/1 dT(36.156um,-4.239um,0) dR(0,0,0)
|15:45:36.007| (INFO) [F:AlignmentTrackChi2] MIMOSA26_3/1 dT(19.095um,-0.794um,0) dR(0,0,0)
|15:45:44.114| (INFO) [F:AlignmentTrackChi2] MIMOSA26_4/1 dT(-16.164um,2.249um,0) dR(0,0,0)
|15:45:49.006| (INFO) [F:AlignmentTrackChi2] MIMOSA26_5/1 dT(-0.722um,0.938um,0) dR(0,0,0)
|15:45:49.006| (STATUS) [F:AlignmentTrackChi2] Alignment iteration 3 of 3
|15:45:53.834| (INFO) [F:AlignmentTrackChi2] MIMOSA26_1/2 dT(10.96um,-1.09um,0) dR(0,0,0)
|15:45:58.742| (INFO) [F:AlignmentTrackChi2] MIMOSA26_2/2 dT(1.155um,0.258um,0) dR(0,0,0)
|15:46:03.302| (INFO) [F:AlignmentTrackChi2] MIMOSA26_3/2 dT(-0.539um,0.657um,0) dR(0,0,0)
|15:46:08.512| (INFO) [F:AlignmentTrackChi2] MIMOSA26_4/2 dT(0.878um,0.553um,0) dR(0,0,0)
|15:46:13.846| (INFO) [F:AlignmentTrackChi2] MIMOSA26_5/2 dT(0.641um,0.734um,0) dR(0,0,0)
|15:46:13.846| (STATUS) [F:AlignmentTrackChi2] Alignment finished, 3 iteration.
|15:46:13.846| (STATUS) [F:AlignmentTrackChi2] MIMOSA26_1 new alignment:
T(12.428um,38.247um,153mm) R(1.57283deg,0.297652deg,-0.214515deg)
|15:46:13.856| (STATUS) [F:AlignmentTrackChi2] MIMOSA26_2 new alignment:
T(154.358um,-39.178um,303mm) R(-0.0336899deg,0.0812454deg,-0.00343775deg)
|15:46:13.857| (STATUS) [F:AlignmentTrackChi2] MIMOSA26_3 new alignment:
T(102.04um,-23.872um,433mm) R(2.08156deg,1.19691deg,0.00699009deg)
|15:46:13.857| (STATUS) [F:AlignmentTrackChi2] MIMOSA26_4 new alignment:
T(125.627um,-2.38um,582mm) R(-2.53133deg,-0.753898deg,-0.354489deg)
|15:46:13.858| (STATUS) [F:AlignmentTrackChi2] MIMOSA26_5 new alignment:
T(-50.02um,8.046um,737mm) R(-3.04516deg,-0.866713deg,0.0708176deg)
|15:46:14.234| (STATUS) Wrote histogram output file to /afs/cern.ch/work/r/rodrigar/Corry_Stuff/corryvreckan/ITk-stuff/ouput/histograms_run427.root
|15:46:14.235| (WARNING) Detectors file /afs/cern.ch/work/r/rodrigar/Corry_Stuff/corryvreckan/ITk-stuff/geometries/geometry_mimosa26_telescope-updated.conf exists and will be overwritten.
|15:46:14.244| (STATUS) Wrote updated detector configuration to /afs/cern.ch/work/r/rodrigar/Corry_Stuff/corryvreckan/ITk-stuff/geometries/geometry_mimosa26_telescope-updated.conf
|15:46:14.245| (STATUS) ===============| Wall-clock timing (seconds) |================
|15:46:14.246| (STATUS) EventLoaderEUDAQ2 : TLU_0 – 4.59384s = 0.229692ms/evt
|15:46:14.246| (STATUS) EventLoaderEUDAQ2 : MIMOSA26_0 – 3.85715s = 0.192858ms/evt
|15:46:14.246| (STATUS) EventLoaderEUDAQ2 : MIMOSA26_1 – 4.16473s = 0.208237ms/evt
|15:46:14.246| (STATUS) EventLoaderEUDAQ2 : MIMOSA26_2 – 4.07199s = 0.203600ms/evt
|15:46:14.246| (STATUS) EventLoaderEUDAQ2 : MIMOSA26_3 – 3.82923s = 0.191462ms/evt
|15:46:14.246| (STATUS) EventLoaderEUDAQ2 : MIMOSA26_4 – 3.82069s = 0.191035ms/evt
|15:46:14.246| (STATUS) EventLoaderEUDAQ2 : MIMOSA26_5 – 4.17184s = 0.208592ms/evt
|15:46:14.246| (STATUS) ClusteringSpatial : TLU_0 – 0.08456s = 0.004228ms/evt
|15:46:14.246| (STATUS) ClusteringSpatial : MIMOSA26_0 – 1.50160s = 0.075080ms/evt
|15:46:14.246| (STATUS) ClusteringSpatial : MIMOSA26_1 – 2.51564s = 0.125782ms/evt
|15:46:14.246| (STATUS) ClusteringSpatial : MIMOSA26_2 – 2.11943s = 0.105972ms/evt
|15:46:14.246| (STATUS) ClusteringSpatial : MIMOSA26_3 – 0.96206s = 0.048103ms/evt
|15:46:14.246| (STATUS) ClusteringSpatial : MIMOSA26_4 – 0.94319s = 0.047159ms/evt
|15:46:14.246| (STATUS) ClusteringSpatial : MIMOSA26_5 – 1.86311s = 0.093155ms/evt
|15:46:14.246| (STATUS) Correlations : TLU_0 – 0.05022s = 0.002511ms/evt
|15:46:14.246| (STATUS) Correlations : MIMOSA26_0 – 2.81265s = 0.140633ms/evt
|15:46:14.246| (STATUS) Correlations : MIMOSA26_1 – 5.11734s = 0.255867ms/evt
|15:46:14.246| (STATUS) Correlations : MIMOSA26_2 – 4.70586s = 0.235293ms/evt
|15:46:14.246| (STATUS) Correlations : MIMOSA26_3 – 2.85870s = 0.142935ms/evt
|15:46:14.246| (STATUS) Correlations : MIMOSA26_4 – 2.55600s = 0.127800ms/evt
|15:46:14.246| (STATUS) Correlations : MIMOSA26_5 – 4.23480s = 0.211740ms/evt
|15:46:14.246| (STATUS) Tracking4D – 30.49105s = 1.524553ms/evt
|15:46:14.246| (STATUS) AlignmentTrackChi2 – 0.81946s = 0.040973ms/evt
|15:46:14.246| (STATUS) ==============================================================
*** Error in `./corry’: double free or corruption (!prev): 0x0000000013b77f90 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81299)[0x7f9580f70299]
/lib64/libc.so.6(__cxa_finalize+0x9a)[0x7f9580f2905a]
/afs/cern.ch/work/r/rodrigar/Corry_Stuff/corryvreckan/install/lib/libCorryvreckanModuleAlignmentTrackChi2.so(+0x81a3)[0x7f958bccf1a3]
======= Memory map: ========
00400000-00408000 r-xp 00000000 00:2b 902663228 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/corryvreckan/install/bin/corry
00608000-00609000 r–p 00008000 00:2b 902663228 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/corryvreckan/install/bin/corry
00609000-0060a000 rw-p 00009000 00:2b 902663228 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/corryvreckan/install/bin/corry
00bee000-17277000 rw-p 00000000 00:00 0 [heap]
7f950f233000-7f950f255000 r-xp 00000000 fc:01 6382649 /usr/lib64/ld-2.17.so
7f950f255000-7f950f454000 —p 00022000 fc:01 6382649 /usr/lib64/ld-2.17.so
7f950f454000-7f950f455000 r–p 00021000 fc:01 6382649 /usr/lib64/ld-2.17.so
7f950f455000-7f950f456000 rw-p 00022000 fc:01 6382649 /usr/lib64/ld-2.17.so
7f950f456000-7f950f457000 rw-p 00000000 00:00 0
7f956c000000-7f956c021000 rw-p 00000000 00:00 0
7f956c021000-7f9570000000 —p 00000000 00:00 0
7f95716af000-7f95718d3000 rw-p 00000000 00:00 0
7f95718d3000-7f9578b64000 r–p 00000000 00:4b 481023005 /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/etc/allDict.cxx.pch
7f9578b64000-7f9578b89000 r-xp 00000000 fc:01 6382745 /usr/lib64/libtinfo.so.5.9
7f9578b89000-7f9578d89000 —p 00025000 fc:01 6382745 /usr/lib64/libtinfo.so.5.9
7f9578d89000-7f9578d8d000 r–p 00025000 fc:01 6382745 /usr/lib64/libtinfo.so.5.9
7f9578d8d000-7f9578d8e000 rw-p 00029000 fc:01 6382745 /usr/lib64/libtinfo.so.5.9
7f9578d8e000-7f957bb6a000 r-xp 00000000 00:4b 481021670 /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCling.so
7f957bb6a000-7f957bd69000 —p 02ddc000 00:4b 481021670 /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCling.so
7f957bd69000-7f957bf4b000 r–p 02ddb000 00:4b 481021670 /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCling.so
7f957bf4b000-7f957bf56000 rw-p 02fbd000 00:4b 481021670 /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCling.so
7f957bf56000-7f957bf7f000 rw-p 00000000 00:00 0
7f957bf7f000-7f957bf86000 r-xp 00000000 00:2b 902657662 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_stcontrol.so
7f957bf86000-7f957c185000 —p 00007000 00:2b 902657662 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_stcontrol.so
7f957c185000-7f957c186000 r–p 00006000 00:2b 902657662 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_stcontrol.so
7f957c186000-7f957c187000 rw-p 00007000 00:2b 902657662 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_stcontrol.so
7f957c187000-7f957c18c000 r-xp 00000000 00:2b 902657654 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_tlu.so
7f957c18c000-7f957c38b000 —p 00005000 00:2b 902657654 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_tlu.so
7f957c38b000-7f957c38c000 r–p 00004000 00:2b 902657654 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_tlu.so
7f957c38c000-7f957c38d000 rw-p 00005000 00:2b 902657654 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_tlu.so
7f957c38d000-7f957c39c000 r-xp 00000000 00:2b 902657630 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_itkstrip.so
7f957c39c000-7f957c59b000 —p 0000f000 00:2b 902657630 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_itkstrip.so
7f957c59b000-7f957c59c000 r–p 0000e000 00:2b 902657630 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_itkstrip.so
7f957c59c000-7f957c59d000 rw-p 0000f000 00:2b 902657630 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_itkstrip.so
7f957c59d000-7f957c5be000 r-xp 00000000 00:2b 902657618 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_eudet.so
7f957c5be000-7f957c7be000 —p 00021000 00:2b 902657618 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_eudet.so
7f957c7be000-7f957c7bf000 r–p 00021000 00:2b 902657618 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_eudet.so
7f957c7bf000-7f957c7c0000 rw-p 00022000 00:2b 902657618 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_eudet.so
7f957c7c0000-7f957c7e4000 r-xp 00000000 00:2b 902657584 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_example.so
7f957c7e4000-7f957c9e4000 —p 00024000 00:2b 902657584 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_example.so
7f957c9e4000-7f957c9e5000 r–p 00024000 00:2b 902657584 /afs/cern.ch/work/r/rodrigar/Corry_Stuff/eudaq/lib/libeudaq_module_example.soAborted (core dumped)

2

Hi Arturo,
thanks for reporting this issue.
“double free or corruption” can happen when trying to delete something twice.

I am trying to remember if I’ve seen this before.

Anyhow, from the output
|15:42:53.755| (STATUS) Welcome to Corryvreckan v1.0+482^g9187f93
I can infer that you’re currently on this commit:

commit 9187f93fb392b36578c0267064667a5d07a4d629
Merge: 824390b1 d819fe2e
Author: Jens Kroeger <kroeger@physi.uni-heidelberg.de>
Date:   Tue Apr 14 16:09:57 2020 +0000

    Merge branch 'dictionary_compilation' into 'master'
    
    Improve Dictionary Compilation
    
    See merge request corryvreckan/corryvreckan!284

There has been a number of updates and changes, particularly related to GBL. Maybe you could try to checkout the latest master and see, if this issue still occurs?

Also, are you running corry on lxplus or on your local machine?

3

Hi!
Thanks for the reply. I read that too, but I think it doesn’t help too much in this case. I will check out the latest commits and run again. I’m running on lxplus. Interestingly enough, I get an output root file but for example, if I run the AnalysisTelescope module I get this other error.

Thanks

|16:30:06.314| (FATAL) [R:AnalysisTelescope] Fatal internal error
Track Object GblTrackDirection after the last telescope plane not defined
Cannot continue…

These are the configuration files I’m using.
analysis_m26-itk.conf

[Corryvreckan]
log_level = “INFO”
log_format = “DEFAULT”

detectors_file = “geometries/geometry_mimosa26_telescope-updated.conf”
histogram_file = “histograms_analysis_run427.root”
output_directory = “/afs/cern.ch/work/r/rodrigar/Corry_Stuff/corryvreckan/ITk-stuff/ouput”

number_of_events = 30000

[EventLoaderEUDAQ2]
name = “TLU_0”
get_time_residuals = true
file_name = “data/run000427_180609184340.raw”

[EventLoaderEUDAQ2]
type = “MIMOSA26”
file_name = “data/run000427_180609184340.raw”

[ClusteringSpatial]
use_trigger_timestamp = true

[Correlations]
make_correlations=true

#[TrackingSpatial]
#min_hits_on_track = 5
#track_model = “gbl”

[Tracking4D]
spatial_cut_abs = 400um,400um
min_hits_on_track = 5
time_cut_abs = 200us
track_model = “gbl”

[AnalysisTelescope]

alignment_mimosa26-itk.conf

[Corryvreckan]
log_level = “INFO”
log_format = “DEFAULT”
log_file = ‘log_corry’

detectors_file = “geometries/geometry_mimosa26_telescope.conf”
detectors_file_updated = “geometries/geometry_mimosa26_telescope-updated.conf”
histogram_file = “histograms_run427.root”
output_directory = “/afs/cern.ch/work/r/rodrigar/Corry_Stuff/corryvreckan/ITk-stuff/ouput”

number_of_events = 20000

[EventLoaderEUDAQ2]
name = “TLU_0”
get_time_residuals = true
file_name = “data/run000427_180609184340.raw”

[EventLoaderEUDAQ2]
type = “MIMOSA26”
file_name = “data/run000427_180609184340.raw”

[ClusteringSpatial]
charge_weighting = false
use_trigger_timestamp = true

[Correlations]
make_correlations = true

[Tracking4D]
spatial_cut_abs = 400um,400um
min_hits_on_track = 5
time_cut_abs = 200us

[AlignmentTrackChi2]
log_level = INFO
number_of_tracks = 300000
iterations = 1
align_orientation = false
align_position = true
max_track_chi2ndof = 10

4

Hi Arturo,

I think I remember now that others have seen this error, too, but not only the ROOT file but also the updated geometry file have been written out. Is that correct?

The error you’re reporting now has been fixed in the meantime. This will be solved by updating corry.
Please let me know how it goes.

Cheers, Jens

5

Hi Jens,

After updating the latest changes, the error is gone.
Thanks a lot!

Arturo

1 Like