I have been profiling the LSST Stack using various tools such as debuggers and profilers. During this process, I noticed that all the software tools had trouble linking to the source files. Specifically, they consistently pointed to non-existing files or directories.
A random example: the following path is reported as the source file path: /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_23/conda/envs/lsst-scipipe-8.0.0-exact/share/eups/EupsBuildDir/Linux64/afw-gadfd92a7e4+96127a2237/afw-gadfd92a7e4+96127a2237/src/image/Image.cc
whereas the correct path is: /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_23/conda/envs/lsst-scipipe-8.0.0-exact/share/eups/Linux64/afw/gadfd92a7e4+96127a2237/src/image/Image.cc
(Note how Linux64/afw/gadfd92a7e4+96127a2237 is wrongfully replaced by EupsBuildDir/Linux64/afw-gadfd92a7e4+96127a2237/afw-gadfd92a7e4+96127a2237.)
The software tools are obtaining the source paths from the .debug_line_str sections, which are created when using the -goption during compilation.
This issue occurs both with the lsstinstall.sh script and the /cvmfs method. However, when I use homemade versions of the packages recompiled with scons, the paths are correct.
I have traced the issue to the way eups installs packages. eups uses a temporary build directory and then moves the package to another location, which breaks the paths.
This should be fixable by either:
Changing the way eups builds the packages, making the build in place instead of using a specific build directory, or
I think if you build with lsstsw tool (which is what we use locally for all our dev builds) everything should work for you because the source files will still be in your source tree.
EUPS builds into a build tree and installs into an install tree like every other build tool I know about so I’m not sure what we can do there.
Setting that compiler flag would be possible for the lsstsw builds and that would likely solve the CernVMFS problem since the paths seem to be burned in during build. For the general person installing with eups distrib install we have no idea where the files will end up.
Thanks for the tips about lsstsw tool. I did a quick test and it seems solves my issues !
Thanks for the tips about the lsstsw tool. I did a quick test, and it seems to solve my issues!
I still think it would be cool to have this small change, as it would help with future-proofing. If you agree, could you guide me on which solution seems more aligned with the philosophy of the stack? I would then create a PR.
Another solution I’ve found is debugedit, which is used for this exact purpose by Fedora for rpm.
Can you explain how your option 1 can work? What tooling builds into the install location directly? We have discussed not installing the source code at all, and won’t be doing that if we eventually switch to a standard conda-style installation for end users.
Is it possible to define the debug-prefix relative to the shared library? That would allow it to work everywhere since we know the src are in ../src relative to the lib directory.
Debugedit sounds possible otherwise but would have to be an afterburner applied by eups distrib install for lsstinstall users. Do you know the macOS equivalent?
As I understand the way eups installs packages, it stores them in $EUPS_PATH/EupsBuildDir/<flavor>/<package-version>, builds them in $EUPS_PATH/EupsBuildDir/<flavor>/<package-version>/build and then moves or copies the directory to $EUPS_PATH/<flavor>/<package>/<version>/.
The idea of option 1 was to make this build directory directly inside the install directory, so no moving is required. However, libraries or binaries can be moved. Same as building something from source.
Both the -fdebug-prefix-map (and the more global -ffile-prefix-map) option and debugedit handle relative paths. However, as far as I know, it’s not possible to make it relative to the shared library or binary. The documentation of GDB, states that the path to the binary or library is never used to search for the path to the sources, therefore the debugger has no way of knowing relative to what the path is given.
The debug information present in a binary that is used to look for the sources is DW_AT_comp_dir (compilation directory) and DW_AT_name (path to the source file, as given on the command line). GDB then does some magic to try to find the corresponding file, but I think most tools just look at DW_AT_comp_dir/DW_AT_name.
This means we need some absolute path somewhere.
Unfortunately I don’t know anything about software like debugedit on MacOS.
At this point I’m thinking that if you are serious about profiling you are going to want to build the software yourself using lsstsw and then you will be using the correct source file paths for the debugger.