Editor integrations in LSST sconsUtils

jbosch · October 1, 2019, 2:29pm

How would people feel about having sconsUtils targets that generate files useful for editor integrations?

I’ve already got a script for VSCode+Python that works okay as something I run manually, but build integration would both make it more convenient and provide a nice way to make it available to others.

I started looking into doing the same for C++ today, but it looks like trying to tell VSCode about C++ include paths without doing something inside sconsUtils is probably impossible. I assume the same is true for any other editor, because you can’t currently process our dependency-declaration information (the stuff in ups/<package>.cfg) without actually being inside a SCons script.

I haven’t thought this through in detail, but I’m thinking maybe having explicit targets for different editors, maintained on a totally volunteer basis, which would write whatever files are helpful for that editor. Those targets would of course not be included in the default build target.

An alternative might be to just have sconsUtils export any information an editor or other tool might want in a generic format. I don’t know of any well-established standard for that for C++, aside from perhaps CMake’s CMAKE_EXPORT_COMPILE_COMMANDS JSON format - but that isn’t something editors can necessarily read natively, and it would require downstream tools to parse information out of command-line arguments that sconsUtils knows directly. A custom format might be better, such as a JSON or YAML file that maps more directly to the SCons environment content.

timj · October 1, 2019, 3:16pm

Would DM-9029 help: replace sconsUtils “.cfg” with pkg_config?

jbosch · October 1, 2019, 3:25pm

It’d make it bit cleaner - you could then write tooling for editors without putting it in our build tooling, just as if we had sconsUtils export something generic, but at least for me it’d probably make it slightly harder to implement, because then I’d have to learn how to extract that information recursively from pkgconfig. More importantly, I don’t want to make the better the enemy of the good enough - I can imagine adding sconsUtils support for VSCode in a couple of hours, and I could imagine DM-9029 lingering indefinitely.

rowen · October 1, 2019, 4:31pm

I’m curious: what does this generate? (I use Sublime Text and it seems to generate everything I need automatically – e.g. a list of symbols for lookup. I’m curious if I’m missing something.)

jbosch · October 1, 2019, 4:42pm

For C++, it would be generating a config file that includes the complete include path for all setup packages. This might be unnecessary for you if you either only care about your editor knowing about symbols from the package you’re working on, or if your workflow involves telling the editor about the full source tree (e.g. in lsstsw) rather than building against installed (e.g.) weeklies.

In Python, it’s creating a config file that propagates PATH, PYTHONPATH, LD_LIBRARY_PATH, and EUPS’s *_DIR variables to the editor without me having to start the editor from a shell with all of those set - that’s particularly relevant when doing remote editing over SSH (where what’s actually relevant is the editor server process one doesn’t normally start manually), but also something I find nice in general.

brianv0 · October 2, 2019, 12:04am

I think this is the closest to a standard. See Also:

https://clang.llvm.org/docs/JSONCompilationDatabase.html
https://www.jetbrains.com/help/clion/compilation-database.html

github.com

melver/scons-bare/blob/master/site_scons/site_tools/compile_commands.py

"""
SCons tool to emit compile_commands.json for files compiled in current
invocation.

Requires default CXXCOMSTR and CCOMSTR.
"""

import json
import os
import SCons

def make_strfunction(strfunction):
    def _strfunction(target, source, env, **kwargs):
        cwd = os.getcwd()
        cmd = strfunction(target, source, env, **kwargs)
        env._compile_commands.append({
            'directory' : cwd,
            'command'   : cmd,
            'file'      : source[0].rstr()
            })

This file has been truncated. show original

jbosch · November 4, 2019, 4:38pm

I’ve taken that SCons logic and integrated it with sconsUtils on DM-22074; the results and further prospects are disappointing but (barely) usable. Quoting from my commit message:

This only writes out the compile commands that are actually being run (i.e. not files that are intentionally not being rebuilt) and only when compile_commands.json is listed explicitly as a target.

That makes this barely usable for editor integration; if you start from a clean repo, you can build from scratch with that target included, making your editor happy, and then rebuild as usual (without that target) to avoid clobbering the file with a version that includes much less.

Unfortunately, doing better than this would seem to involve understanding much more of SCons’ internals than I have time for now, and I worry that it’s not even a well-formed operation, given the degree to which sconsUtils regularly disregards the super-important distinction between scons-execution-time and build-time logic. More time is probably better invested in just switching us to CMake.

I’ll try using that ticket locally and see what I think before asking for a review/merge (I’m pretty sure it at least does no harm). I’d be interested in getting feedback from others as well, if this is something you’re looking for. Good news is that just writing compile_commands.json was enough to make all of the C++ VSCode integration Just Work, which I hadn’t realized when making my original post.

mwittgen · October 26, 2020, 4:37pm

Reviving this old topic:
Bear seems suitable to create a compilation command DB.

I have compiled version 2.4.4 with devtoolset-8 on lsst-dev. Version 3.0.0 introduces some more dependencies like grpc, spdlog I didn’t want to deal with for a first test.
Running with
/home/wittgen/opt/bear/bin/bear scons -j8
creates
/home/wittgen/dev/afw/compile_commands.json