Like many other teams, we have a fair amount of 3rd-party dependencies in our project (a C++-based SDK, that is). Like fewer other teams, we store those dependencies source code right in our project repository and we build them together with the project sources every time. This is of course a tremendous waste of time and CPU cycles, as it would be much more efficient to build dependencies just once (per version) and link to already pre-built binaries.

But where to store pre-built dependencies and how to fetch the correct variants for various platforms, toolchains and configurations? Our investigation on the matter led us to Conan package manager.

Why bother

Straight to the point, here’s a build of my simple application that has a few dependencies: glad, GLFW and Dear ImGui - everything is built from sources:

$ time cmake --build .
[29/29] Linking CXX executable glfw-imgui

real    0m13.472s

$ cat .ninja_log
# ninja log v5
2    344    1643144116191484343    CMakeFiles/glfw-imgui.dir/_dependencies/DearImGui/imgui_impl_glfw.cpp.o    538b240abc3a3bc4
3    386    1643144116228441376    CMakeFiles/glfw-imgui.dir/_dependencies/DearImGui/imgui_impl_opengl3.cpp.o    cb2977cd0cf0a02
1    838    1643144116674414843    CMakeFiles/glfw-imgui.dir/_dependencies/DearImGui/imgui_stdlib.cpp.o    72c8c50464634e52
4    1459    1643144117295845649    _dependencies/glfw/src/CMakeFiles/glfw.dir/init.c.o    ec7a84d90e09be1
1    1473    1643144117313538409    CMakeFiles/glfw-imgui.dir/functions.cpp.o    369a9f32cfe69e82
4    1600    1643144117436078293    _dependencies/glfw/src/CMakeFiles/glfw.dir/context.c.o    2ae159d1c31a5c3c
386    1707    1643144117535843892    _dependencies/glfw/src/CMakeFiles/glfw.dir/vulkan.c.o    e5f6409e22e990ab
345    1855    1643144117682858278    _dependencies/glfw/src/CMakeFiles/glfw.dir/monitor.c.o    aa0b3f727ad17763
5    1972    1643144117794902750    _dependencies/glfw/src/CMakeFiles/glfw.dir/input.c.o    556c75d1b24c1fec
0    2122    1643144117958231086    CMakeFiles/glfw-imgui.dir/main.cpp.o    3c409eca7f63548
841    2612    1643144118445759410    _dependencies/glfw/src/CMakeFiles/glfw.dir/window.c.o    4cd69de2cf698676
2    3040    1643144118878103647    CMakeFiles/glfw-imgui.dir/_dependencies/DearImGui/imgui_tables.cpp.o    d613d3d1e47520ee
1855    3420    1643144119254115013    _dependencies/glfw/src/CMakeFiles/glfw.dir/cocoa_time.c.o    8fe5aa7e55c4a4ee
1973    3522    1643144119337803512    _dependencies/glfw/src/CMakeFiles/glfw.dir/posix_thread.c.o    ca35990aafd46022
2613    4220    1643144120055526527    _dependencies/glfw/src/CMakeFiles/glfw.dir/egl_context.c.o    4a0f25900af8ffc7
3    4312    1643144120151590718    CMakeFiles/glfw-imgui.dir/_dependencies/DearImGui/imgui_demo.cpp.o    ad04f8180049818f
1474    4379    1643144120210864651    _dependencies/glfw/src/CMakeFiles/glfw.dir/cocoa_joystick.m.o    8f8a50953484faa7
1467    4428    1643144120252642979    _dependencies/glfw/src/CMakeFiles/glfw.dir/cocoa_init.m.o    ec5d33d5098b5c52
1600    4543    1643144120379829813    _dependencies/glfw/src/CMakeFiles/glfw.dir/cocoa_monitor.m.o    eb2653b5a4204ff1
3043    4578    1643144120418062550    _dependencies/glfw/src/CMakeFiles/glfw.dir/osmesa_context.c.o    7d802e40cc6041e4
2123    4695    1643144120536939362    _dependencies/glfw/src/CMakeFiles/glfw.dir/nsgl_context.m.o    787dd493c62ff124
1708    4810    1643144120650860676    _dependencies/glfw/src/CMakeFiles/glfw.dir/cocoa_window.m.o    5bd62d5af52211f1
4810    4863    1643144120718256000    _dependencies/glfw/src/libglfw3.a    39988b2ad15bd49d
3    5676    1643144121524078659    CMakeFiles/glad.dir/_dependencies/glad/src/glad.c.o    4875fb68b660e707
5676    5716    1643144121571422000    libglad.a    af027585fb9b185b
2    5824    1643144121672025107    CMakeFiles/glfw-imgui.dir/_dependencies/DearImGui/imgui_widgets.cpp.o    ce787c92a020884c
1    7045    1643144122892928204    CMakeFiles/glfw-imgui.dir/_dependencies/DearImGui/imgui.cpp.o    6d6193214989032
2    12906    1643144128754775483    CMakeFiles/glfw-imgui.dir/_dependencies/DearImGui/imgui_draw.cpp.o    ed7c0bc7c73c224b
12906    13075    1643144128921561250    glfw-imgui    c2d88c723185913b

So it is 29 items to build and 13.5 seconds of building time.

And here’s the same project building only the project sources, while all the dependencies are already pre-built:

$ time cmake --build .
[3/3] Linking CXX executable glfw-imgui

real    0m1.068s

$ cat .ninja_log
# ninja log v5
1    727    1643144024851560807    CMakeFiles/glfw-imgui.dir/functions.cpp.o    abc14ffe6cd16ea3
0    958    1643144025083386756    CMakeFiles/glfw-imgui.dir/main.cpp.o    49263edbd6a7bdab
958    1174    1643144025294166462    glfw-imgui    eb2bea70f24ef99e

Only 3 items to build and 1 second of building time - 13 times faster! Certainly worth the effort, intit? And for bigger projects the gain can be even greater.

For us this became especially important, as our project is re-built on every commit in our CI/CD to run all the QA tests, so the builds are run many times a day, wasting a lot of resources re-building non-changing dependencies over and over.

But pre-building dependencies is not enough, it isn’t even a half of the job. First of all, dependencies need to be built for several platforms and different toolchains. Then every dependency can (will) have different versions and also dependencies of its own. And finally, you need to store pre-built artifacts somewhere and fetch the right variant for the build.

To handle/manage all that you’ll need a combination of CI/CD builbots, an artifacts storage and a package manager.

Conan

Conan is a package/dependency manager, mostly for C++ development. It has a good and easy enough integration with CMake (and seemingly other build systems too).

In short, Conan packages are regular bin/cmake/include/lib/etc folder structures packed into .tgz compressed archives, accompanied with various metadata. Conan client uses that metadata to determine correct packages for concrete platforms/toolchains/configurations, resolve dependencies, etc.

What might come as a shock for some is that Conan is written in Python. I’ve seen people looking down on it because of that and saying something like “how can this pitiful indent-based scripting language handle dependencies for our mighty C++”. But after testing Conan for a couple of months, I can say that it is doing a good job. There are not that many package managers for C++ projects, and out of those that I tried so far I like Conan the most.

I suggest you to get familiar with Conan by reading its documentation and trying out their examples, as it’s too much to cover in one article.

Out of all the functionality that Conan provides, at the moment we are using only packing pre-built binaries and of course installing dependencies.

Packages storage

There are several options for where the Conan packages can be hosted and fetched from.

conan_server

The default option is to use conan_server that comes out of the box. If you decide to use that one, as usual, I would recommend not to expose it to the internet directly but rather via a reverse-proxy, such as NGINX.

In short, this would be your server.conf:

[server]
ssl_enabled: False
port: 9300
host_name: your.host
...

[write_permissions]
*/*@*/*: ADMIN-USER

[read_permissions]
*/*@*/*: ?

[users]
ADMIN-USER: HERE-GOES-ADMIN-PASSWORD
# has nothing to do with the system user
packages: HERE-GOES-USER-PASSWORD

this is systemd service:

[Unit]
Description=Conan server

[Service]
# isn't really used, set just in case
WorkingDirectory=/var/www/conan-server/
ExecStart=/usr/local/bin/conan_server
Restart=always
RestartSec=10
SyslogIdentifier=conan_server
User=packages

[Install]
WantedBy=multi-user.target

and that’s NGINX config:

location /packages/conan/ {
    # Conan server has its own authentication
    #auth_basic           "restricted area";
    #auth_basic_user_file /etc/nginx/packages.htpasswd;
    proxy_pass http://127.0.0.1:9300/;
    # these might not be needed
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection keep-alive;
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

We started with conan_server, and it was working fine, but then we discovered that despite read_permissions is set to */*@*/*: ?, which means allowing access only for authenticated users(?), the server was still available to anonymous users too, so everyone could list and download our packages, which certainly was not the intention. I thought about reporting this in the Conan bugtracker, but when I had another question, along with the answer I got a recommendation to start using JFrog Artifactory, so I reckoned it won’t be worth wasting everyone’s time asking about conan_server any more.

Conan Center

Not much to say here - it’s a public Conan packages storage, a great source of lots of pre-built libraries.

But we are not using it. First of all, because of the same reasons listed below; and secondly, we’d like to ensure that both our project and all the dependencies are built in the same environment, on the same buildbots, and in general we prefer to build our dependencies ourselves.

JFrog Artifactory

JFrog Artifactory is a great piece of software for hosting packages, either in their cloud (no) or on your own server (yes). Probably that is what’s running behind the Conan Center too.

You can take the Community Edition, which is graciously available free of charge (and limited to Conan packages only). Trying to find it using the links from the referred documentation if futile, so here’s the downloads page.

As we need to distribute many other package formats beside Conan (deb, NuGet, npm, Maven, etc), it was grand time for us to get a full-featured Artifactory, which supports all these formats:

It is expensive (cheapest variant is 3200 USD per year!), but it saves a lot of trouble for maintaining several different storages/services - instead you only need to maintain one Artifactory. Also it nicely integrates with Azure AD SAML SSO, which helps a lot too.

Dependencies

Source code repositories

Now we have a place for storing pre-built dependencies in packages, but where to store their source code?

If you are your own evil twin (сам себе злой Буратино), then you can just fetch it from original repositories on GitHub, GitLab or wherever. If you really are an enemy to yourself, then you can also store your own project source code in an external repository too.

The main reason why this is not a great idea is that any external resource might suddenly become unavailable, and you can’t do anything about it, as that’s not your server and you have no control over it. Here are some examples of such unavailability:

any service can go down anytime, including GitHub;
Facebook can update its DNS records and bring the rest of the internet down;
malicious/righteous person(s) can commit destructive changes into a repository;
a repository can be blocked on whatever grounds;
just as well users from certain sanctioned countries can be denied using a service;
the entire country can lose internet completely, because authorities shut it down;
fucking sharks can destroy internet cables.

A better idea would be to store copies (clones) of your dependencies source code in a self-hosted in-house repository. It can be the GitLab, Gitea, some other service or just bare Git repositories.

An even better idea would be to have those repositories not as mirrors, but as regular clones, so they are updated to the next version manually, once it is verified to be safe/stable to use. And that is what we started doing by gradually moving 3rd-party sources out from our main repository to their own repositories:

Project files

Almost always libraries sources already have project files for various build systems. Ideally that would be CMake, because that is our main build system.

If a library doesn’t have a project file, then it could be that author intended it to be built as a part of your project, for example Dear ImGui is like that. But you still might want to pre-build it separately, so creating a CMakeLists.txt for it will be on you.

Another scenario would be so-called “header-only” libraries, such as pdqsort, so there is nothing to build there. But even in that case you might want to consider making CMakeLists.txt for it with install instructions to generate CMake configs. Furthermore, some header-only libraries, for example json-nlohmann, do have CMakeLists.txt and sometimes even include a number of tests.

For storing those missing CMakeLists.txt’s, conanfiles, shared CMake modules and also patches for special cases we created a separate repository called “Dependencies resources”, which looks like this:

├── README.md
├── _cmake
│   ├── Config.cmake.in
│   └── Installing.cmake
├── _conan
│   └── functions.py
├── cpp-base64
│   ├── CMakeLists.txt
│   └── conanfile.py
├── dear-imgui
│   ├── CMakeLists.txt
│   └── conanfile.py
├── e57format
│   ├── conanfile.py
│   └── patches
│       └── 2.2.0
│           └── CMakeLists.txt.patch
├── glad
│   ├── CMakeLists.txt
│   └── conanfile.py
├── glfw
│   └── conanfile.py
├── json-nlohmann
│   └── conanfile.py
├── jsoncpp
│   └── conanfile.py
├── pdqsort
│   ├── CMakeLists.txt
│   └── conanfile.py
├── poco
│   └── conanfile.py
├── rapidxml
│   ├── CMakeLists.txt
│   └── conanfile.py
├── xerces-c
│   ├── conanfile.py
│   └── patches
│       └── 3.2.3
│           └── CMakeLists.txt.patch
└── zstandard
    └── conanfile.py

That way we don’t interfere with the original repositories/clones and keep all the dependencies-related resources in one place.

conanfile

The conanfile.py contains various information about the package and it is almost the same for all the dependencies:

from conans import ConanFile, tools


def normalizeVersion(rawVersion):
    if rawVersion.startswith("v"):
        rawVersion = rawVersion[1:]
    if rawVersion.count(".") < 2:
        rawVersion = f"{rawVersion}.0"
    return rawVersion


class DearImGuiConan(ConanFile):
    projectName = "DearImGui"
    name = projectName
    version = "0.0.0"
    user = "YOUR-PREFIX"
    channel = "public"
    settings = "os", "compiler", "build_type", "arch"
    description = "Bloat-free Graphical User interface for C++ with minimal dependencies"
    homepage = "https://github.com/ocornut/imgui"
    url = "https://github.com/ocornut/imgui"
    license = "https://github.com/ocornut/imgui/blob/master/LICENSE.txt"
    author = projectName

    def set_version(self):
        self.version = normalizeVersion(self.version)

    def package(self):
        self.copy("*")

    def package_info(self):
        self.cpp_info.libs = tools.collect_libs(self)

You will be right to assume that normalizeVersion function is redundantly duplicated across all of them. This is because sadly Conan does not support importing relative files/modules. There is Python requires functionality, but it looks too complex for such a purpose and also it is marked as experimental at the moment, so we haven’t tried it yet.

Why version is set to 0.0.0? At first we were setting it based in Git tags:

def getVersionFromGit():
    rawVersion = ""
    git = tools.Git(folder=self.recipe_folder)
    try:
        # rawVersion = git.run("describe --tags --abbrev=0")
        rawVersion = normalizeVersion(git.get_tag())
    except Exception as ex:
        print(f"Exception: {str(ex)}")
        # rawVersion = "0.0.0"
        raise SystemExit(
            "##teamcity[message text='Package version identification failed' "
            "errorDetails='Conan couldn't get the version tag from Git' "
            "status='ERROR']"
        )
    return rawVersion

But that is not very reliable (different tag formats, missing tags, etc), and so now we set it explicitly from CI/CD variable by replacing version = "0.0.0" pattern with sed:

sed -i 's/version = \"0\.0\.0\"$/version = \"%system.current-version%\"/' ./conanfile.py

In addition, if you are checking out sources from different repositories into a common path, there might be no .git data available.

The name, version, user and channel properties comprise the package formula/reference, for example:

DearImGui/1.86.0@YOUR-PREFIX/public

You might want to use channel property for organizing packages by platforms, but that would be redundant and rather confusing, as that is already taken care of by Conan via so-called profiles (there is an illustration for that on the last screenshot in the packing section).

What user and channel properties should be used for is identifying your package as yours to prevent possible collisions with other people packages. The user property here (YOUR-PREFIX) can be your company or team name, and channel can be whatever you’d like (public, experimental, customerspecific).

CI/CD

Projects and configurations

Hopefully your infrastructure already has dedicated buildbots for building, testing and packing your product, managed by a CI/CD system.

We use TeamCity, so instructions below will be mostly TeamCity-specific, but it’s rather trivial to apply them to whatever CI/CD you might have.

As we build and distribute our SDK for several platforms/toolchains, there is a configuration for every target:

Libraries have versions, and usually those are marked with Git tags, which can be used as branch specifications. We set a project parameter current-version for each dependency:

This parameter can be then used in VCS root default branch specification:

Note the v in refs/tags/v%system.current-version%. Different repositories have different naming schema for tags: some add v to the version number, some don’t and others do something else (like SDL or cpp-base64). So for consistency we have a “plain” version set as a system variable in SemVer format and then prefix it with necessary values. If the original tags format is absolutely horrible (or if they are no tags at all), then you’ll need to add the tags with versions yourself (it’s your clone, after all).

Once you set the VCS root like that, you’ll be able to build the specific version of a library, instead of looking for the commit of interest:

Quite handy, wouldn’t you say. And here’s how the builds will look like:

The START configuration generates a common build number (optional) and triggers the builds themselves:

Here you can specify that builds won’t trigger for any version other than the default one that is set in the project parameters (note the v prefix again).

Finally, how to organize the sources checkout:

The library sources checkout rule is trivial:

+:. => ./

And dependencies resources are checked out like this:

+:%system.projectNameNormalized% => ./
+:_cmake => ./cmake

So everything specific to this library (CMakeLists.txt, conanfile.py, patches, etc) is fetched to the top level, and common resources (CMake modules) are fetched into subfolders.

Since it’s two different repositories that are fetched into a common path, you won’t have .git data there, so if you do need it, then you’ll need to checkout those into separate paths.

Building

The build step in most cases is simple:

mkdir build
mkdir install

cd build
cmake -G "Visual Studio 16 2019" -DCMAKE_INSTALL_PREFIX="../install" -DCMAKE_DEBUG_POSTFIX="d" ..
cmake --build . --target install --config Debug
sh.exe -c "cp ./install_manifest.txt ../install-manifest-debug.txt"
cmake --build . --target install --config Release
sh.exe -c "cp ./install_manifest.txt ../install-manifest-release.txt"

The install_manifest.txt files are a very useful feature of CMake: when you build install target (or run cmake --install), CMake generates a list of files that were installed. These files can be then used to separate Debug and Release build artifacts, which is what’s happening here.

You can of course set different installation prefixes for Debug/Release configurations, but we need them in one folder (that way you also avoid duplication of some files between configurations).

On GNU/Linux the build step is somewhat less simple:

#!/bin/bash

mkdir build
mkdir install

cd build

cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX="../install" -DCMAKE_DEBUG_POSTFIX="d" ..
cmake --build . --target install
cp ./install_manifest.txt ../install-manifest-debug.txt
rm -r ./* && rm .ninja*

cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX="../install" ..
cmake --build . --target install
cp ./install_manifest.txt ../install-manifest-release.txt

Without cleaning the build folder (rm -r ./* && rm .ninja*) after building and installing Debug configuration you’ll get a surprising result on installing the Release configuration:

-- Old export file "/home/build/buildAgent/work/externals/dear-imgui/src/install/cmake/DearImGuiConfig.cmake" will be replaced
Removing files [/home/build/buildAgent/work/externals/dear-imgui/src/install/cmake/DearImGuiConfig-debug.cmake]

And so you’ll be missing that file in your package (thanks a lot!).

Dependencies of dependencies

If (when) a dependency has dependencies of its own, the project configuration and build become slightly more complicated, but thanks to Conan not too complicated.

For example, let’s take E57Format library, which depends on Xerces-C++ library and expects to discover it with find_package().

In that case, naturally, first you need to build Xerces-C++ and publish it to your Conan packages storage. Then add the following to E57Format’s conanfile.py:

class E57FormatConan(ConanFile):
    projectName = "E57Format"
    # ...
    requires = [
        "Xerces-C/3.2.3@YOUR-PREFIX/public"
    ]

But that’s not enough, because E57Format’s CMakeLists.txt does not contain required commands for running Conan stuff, and that means that we need to make a patch (dependencies-resources/e57format/patches/2.2.0/CMakeLists.txt.patch) for adding them:

44a45,47
> include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
> conan_basic_setup()
>
106c109
<         DEBUG_POSTFIX "-d"
---
>         DEBUG_POSTFIX "d"

Here we also override their hardcoded DEBUG_POSTFIX, because we use d in our libraries and we would like this to be consistent. By the way, that hardcoding is a good example of doing bad things in CMake, because in that case user loses the ability to override this value with a -D argument (so a better patch would be to delete this line instead of replacing it).

The build step for E57Format project in CI/CD looks like this:

mkdir build
mkdir install

sh.exe -c "patch ./CMakeLists.txt ./patches/%system.current-version%/CMakeLists.txt.patch"

cd build
conan install .. -s compiler.version=16 -s build_type=Debug -r YOUR-CONAN-REMOTE
conan install .. -s compiler.version=16 -s build_type=Release -r YOUR-CONAN-REMOTE

cmake -G "Visual Studio 16 2019" -DCMAKE_INSTALL_PREFIX="../install" ..
cmake --build . --target install --config Debug
sh.exe -c "cp ./install_manifest.txt ../install-manifest-debug.txt"
cmake --build . --target install --config Release
sh.exe -c "cp ./install_manifest.txt ../install-manifest-release.txt"

Packing

When creating Conan package, you can put both Debug and Release configurations into one package. Thanks to CMake this work work just fine: find_package() will find both Debug (DearImGuiTargets-debug.cmake) and Release (DearImGuiTargets-release.cmake) configurations. And conan export-pkg wouldn’t care, as it packs everything it finds in the path specified in -pf argument.

But that would not be exactly correct, because if one tries to install such package like this:

$ conan install -s build_type=Debug -r YOUR-CONAN-REMOTE

then Conan will say that there are no available packages for this configuration. And why it works without providing -s build_type is because it defaults to Release (both for conan export-pkg and conan install).

It is actually possible to have Debug and Release in one package, so it would work for any -s build_type, but it’s not as simple as just splitting configurations into different packages.

The splitting can be done by using install-manifest-*.txt lists produced on the building step. But they store absolute paths to the files, and so we need to trim them with sed. After that the installation folder can be split into Debug/Release with cp --parents to preserve the folders structure.

After splitting is done, you can create packages with conan export-pkg and publish them to your Artifactory with conan upload.

Here are all the commands executing on the packing step:

mkdir package-debug
sh.exe -c "sed -i 's/.*install\///g' ./install-manifest-debug.txt"

mkdir package-release
sh.exe -c "sed -i 's/.*install\///g' ./install-manifest-release.txt"

cd install
sh.exe -c "cp --parents $(<../install-manifest-debug.txt) ../package-debug || :"
sh.exe -c "cp --parents $(<../install-manifest-release.txt) ../package-release || :"
cd ..

sh.exe -c "sed -i 's/version = \"0\.0\.0\"$/version = \"%system.current-version%\"/' ./conanfile.py"
conan export-pkg conanfile.py -s compiler.toolset="v142" -pf="./package-debug" -s build_type=Debug -f && conan upload %system.projectName% --all --remote=YOUR-CONAN-REMOTE --check --confirm
conan export-pkg conanfile.py -s compiler.toolset="v142" -pf="./package-release" -s build_type=Release -f && conan upload %system.projectName% --all --remote=YOUR-CONAN-REMOTE --check --confirm

conan remove "%system.projectName%*" -f

Setting the package version with sed is explained in conanfile description.

Removing the package (conan remove) from local cache after publishing it to Artifactory is optional.

As usual, on Mac OS Bash commands behave differently, so you’ll need to adjust sed arguments and use rsync instead of cp:

#!/bin/bash

mkdir ./package-debug
sed -i "" -E 's/.*install\///g' ./install-manifest-debug.txt

mkdir ./package-release
sed -i "" -E 's/.*install\///g' ./install-manifest-release.txt

cd install
rsync -R $(<../install-manifest-debug.txt) ../package-debug || :
rsync -R $(<../install-manifest-release.txt) ../package-release || :
cd ..

sed -i "" -E 's/version = "0\.0\.0"$/version = "%system.current-version%"/' ./conanfile.py
conan export-pkg conanfile.py -s arch=armv8 -pf="./package-debug" -s build_type=Debug -f && conan upload %system.projectName% --all --remote=YOUR-CONAN-REMOTE --check --confirm
conan export-pkg conanfile.py -s arch=armv8 -pf="./package-release" -s build_type=Release -f && conan upload %system.projectName% --all --remote=YOUR-CONAN-REMOTE --check --confirm

conan remove "%system.projectName%*" -f

By the way, since we are talking about Mac OS special needs, if you’ll need to RegEx a \w+ pattern, for example this one: 's/.*install\/\w\+\///g', then you’ll have to do it like that: 's/.*install\/[a-zA-Z0-9_]+\///g'.

After you run building and packing steps on all the platforms/targets, here’s what you’ll get in the Artifactory:

So it is 10 packages in total: 5 platforms, each having 2 configurations (Debug/Release).

Resolving dependencies in a project

As an example, let’s take the same application that was used for benchmarking build times in the beginning.

I have all of its dependencies pre-built and published in Artifactory, so I can fetch them with Conan. For that I can either create conanfile.txt:

[requires]
glad/0.1.36@YOUR-PREFIX/public
GLFW/3.3.5@YOUR-PREFIX/public
DearImGui/1.86.0@YOUR-PREFIX/public

or conanfile.py:

from conans import ConanFile, CMake

prefixChannel = "@YOUR-PREFIX/public"


class SandboxConan(ConanFile):
    settings = "os", "arch", "compiler"
    generators = "cmake"
    requires = [
        f"glad/0.1.36{prefixChannel}",
        f"GLFW/3.3.5{prefixChannel}",
        f"DearImGui/1.86.0{prefixChannel}"
    ]

Now I can fetch them all:

$ mkdir build && cd $_

$ conan install .. -r YOUR-CONAN-REMOTE
Configuration:
[settings]
arch=x86_64
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=12.0
os=Macos
[options]
[build_requires]
[env]

glad/0.1.36@YOUR-PREFIX/public: Retrieving from server 'YOUR-CONAN-REMOTE'
glad/0.1.36@YOUR-PREFIX/public: Trying with 'YOUR-CONAN-REMOTE'...
Downloading conanmanifest.txt completed [0.06k]
Downloading conanfile.py completed [0.94k]
glad/0.1.36@YOUR-PREFIX/public: Downloaded recipe revision 0
GLFW/3.3.5@YOUR-PREFIX/public: Retrieving from server 'YOUR-CONAN-REMOTE'
GLFW/3.3.5@YOUR-PREFIX/public: Trying with 'YOUR-CONAN-REMOTE'...
Downloading conanmanifest.txt completed [0.06k]
Downloading conanfile.py completed [0.86k]
GLFW/3.3.5@YOUR-PREFIX/public: Downloaded recipe revision 0
DearImGui/1.86.0@YOUR-PREFIX/public: Retrieving from server 'YOUR-CONAN-REMOTE'
DearImGui/1.86.0@YOUR-PREFIX/public: Trying with 'YOUR-CONAN-REMOTE'...
Downloading conanmanifest.txt completed [0.06k]
Downloading conanfile.py completed [0.92k]
DearImGui/1.86.0@YOUR-PREFIX/public: Downloaded recipe revision 0
conanfile.py: Installing package
Requirements
    DearImGui/1.86.0@YOUR-PREFIX/public from 'YOUR-CONAN-REMOTE' - Downloaded
    GLFW/3.3.5@YOUR-PREFIX/public from 'YOUR-CONAN-REMOTE' - Downloaded
    glad/0.1.36@YOUR-PREFIX/public from 'YOUR-CONAN-REMOTE' - Downloaded
Packages
    DearImGui/1.86.0@YOUR-PREFIX/public:f83037eff23ab3a94190d7f3f7b37a2d6d522241 - Download
    GLFW/3.3.5@YOUR-PREFIX/public:f83037eff23ab3a94190d7f3f7b37a2d6d522241 - Download
    glad/0.1.36@YOUR-PREFIX/public:f83037eff23ab3a94190d7f3f7b37a2d6d522241 - Download

Installing (downloading, building) binaries...
DearImGui/1.86.0@YOUR-PREFIX/public: Retrieving package f83037eff23ab3a94190d7f3f7b37a2d6d522241 from remote 'YOUR-CONAN-REMOTE'
Downloading conanmanifest.txt completed [0.63k]
Downloading conaninfo.txt completed [0.40k]
Downloading conan_package.tgz completed [577.86k]
Decompressing conan_package.tgz completed [0.00k]
DearImGui/1.86.0@YOUR-PREFIX/public: Package installed f83037eff23ab3a94190d7f3f7b37a2d6d522241
DearImGui/1.86.0@YOUR-PREFIX/public: Downloaded package revision 0
GLFW/3.3.5@YOUR-PREFIX/public: Retrieving package f83037eff23ab3a94190d7f3f7b37a2d6d522241 from remote 'YOUR-CONAN-REMOTE'
Downloading conanmanifest.txt completed [0.56k]
Downloading conaninfo.txt completed [0.40k]
Downloading conan_package.tgz completed [123.99k]
Decompressing conan_package.tgz completed [0.00k]
GLFW/3.3.5@YOUR-PREFIX/public: Package installed f83037eff23ab3a94190d7f3f7b37a2d6d522241
GLFW/3.3.5@YOUR-PREFIX/public: Downloaded package revision 0
glad/0.1.36@YOUR-PREFIX/public: Retrieving package f83037eff23ab3a94190d7f3f7b37a2d6d522241 from remote 'YOUR-CONAN-REMOTE'
Downloading conanmanifest.txt completed [0.39k]
Downloading conaninfo.txt completed [0.40k]
Downloading conan_package.tgz completed [333.16k]
Decompressing conan_package.tgz completed [0.00k]
glad/0.1.36@YOUR-PREFIX/public: Package installed f83037eff23ab3a94190d7f3f7b37a2d6d522241
glad/0.1.36@YOUR-PREFIX/public: Downloaded package revision 0
conanfile.py: Generator cmake created conanbuildinfo.cmake
conanfile.py: Generator txt created conanbuildinfo.txt
conanfile.py: Aggregating env generators
conanfile.py: Generated conaninfo.txt
conanfile.py: Generated graphinfo

Lucky me, the auto-generated Conan profile for my environment matches the one on the buildbots, so I got everything found and installed.

But actually it doesn’t match, and here’s what I was really getting with auto-generated profile:

Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=clang
compiler.libcxx=libstdc++
compiler.version=13
os=Macos
os_build=Macos
[options]
[build_requires]
[env]

glad/0.1.36@YOUR-PREFIX/public: Retrieving from server 'YOUR-CONAN-REMOTE'
glad/0.1.36@YOUR-PREFIX/public: Trying with 'YOUR-CONAN-REMOTE'...
Downloading conanmanifest.txt completed [0.06k]
Downloading conanfile.py completed [0.94k]
glad/0.1.36@YOUR-PREFIX/public: Downloaded recipe revision 0
GLFW/3.3.5@YOUR-PREFIX/public: Retrieving from server 'YOUR-CONAN-REMOTE'
GLFW/3.3.5@YOUR-PREFIX/public: Trying with 'YOUR-CONAN-REMOTE'...
Downloading conanmanifest.txt completed [0.06k]
Downloading conanfile.py completed [0.86k]
GLFW/3.3.5@YOUR-PREFIX/public: Downloaded recipe revision 0
DearImGui/1.86.0@YOUR-PREFIX/public: Retrieving from server 'YOUR-CONAN-REMOTE'
DearImGui/1.86.0@YOUR-PREFIX/public: Trying with 'YOUR-CONAN-REMOTE'...
Downloading conanmanifest.txt completed [0.06k]
Downloading conanfile.py completed [0.92k]
DearImGui/1.86.0@YOUR-PREFIX/public: Downloaded recipe revision 0
conanfile.py: Installing package
Requirements
    DearImGui/1.86.0@YOUR-PREFIX/public from 'YOUR-CONAN-REMOTE' - Downloaded
    GLFW/3.3.5@YOUR-PREFIX/public from 'YOUR-CONAN-REMOTE' - Downloaded
    glad/0.1.36@YOUR-PREFIX/public from 'YOUR-CONAN-REMOTE' - Downloaded
Packages
    DearImGui/1.86.0@YOUR-PREFIX/public:c48a49ca4467bb2c7f002b84b691f213c49e91b4 - Missing
    GLFW/3.3.5@YOUR-PREFIX/public:c48a49ca4467bb2c7f002b84b691f213c49e91b4 - Missing
    glad/0.1.36@YOUR-PREFIX/public:c48a49ca4467bb2c7f002b84b691f213c49e91b4 - Missing

Installing (downloading, building) binaries...
ERROR: Missing binary: DearImGui/1.86.0@YOUR-PREFIX/public:c48a49ca4467bb2c7f002b84b691f213c49e91b4
ERROR: Missing binary: GLFW/3.3.5@YOUR-PREFIX/public:c48a49ca4467bb2c7f002b84b691f213c49e91b4
ERROR: Missing binary: glad/0.1.36@YOUR-PREFIX/public:c48a49ca4467bb2c7f002b84b691f213c49e91b4

DearImGui/1.86.0@YOUR-PREFIX/public: WARN: Can't find a 'DearImGui/1.86.0@YOUR-PREFIX/public' package for the specified settings, options and dependencies:
- Settings: arch=x86_64, build_type=Release, compiler=clang, compiler.libcxx=libstdc++, compiler.version=13, os=Macos
- Options:
- Dependencies:
- Requirements:
- Package ID: c48a49ca4467bb2c7f002b84b691f213c49e91b4

ERROR: Missing prebuilt package for 'DearImGui/1.86.0@YOUR-PREFIX/public', 'GLFW/3.3.5@YOUR-PREFIX/public', 'glad/0.1.36@YOUR-PREFIX/public'
Use 'conan search DearImGui/1.86.0@YOUR-PREFIX/public --table=table.html -r=remote' and open the table.html file to see available packages
Or try to build locally from sources with '--build=DearImGui --build=GLFW --build=glad'

More Info at 'https://docs.conan.io/en/latest/faq/troubleshooting.html#error-missing-prebuilt-package'

This command, for example:

$ conan search DearImGui/1.86.0@YOUR-PREFIX/public --table=table.html -r YOUR-CONAN-REMOTE

will generate a table of all the packages available for Dear ImGui in your Conan remote:

Ideally, I would need to add a build environment on the buildbots that matches mine, build all dependencies with it and publish them to Artifactory too, but it’s bloody impossible to cover all possible configurations that users might have, so instead users (and I) can just adjust their local Conan profiles. In my case it would be ~/.conan/profiles/artifactory with the following contents:

[settings]
    arch=x86_64
    build_type=Release
    compiler=apple-clang
    compiler.libcxx=libc++
    compiler.version=12.0
    os=Macos

And then I can use this profile like this:

$ conan install .. --profile artifactory -r YOUR-CONAN-REMOTE

However, getting ahead of myself a bit, while this helps to convince Conan to install the packages, such profile tinkering will cause CMake to fail on configuration with the following error:

CMake Error at build/conanbuildinfo.cmake:731 (message):
  Detected a mismatch for the compiler version between your conan profile
  settings and CMake:

  Compiler version specified in your conan profile: 12.0

  Compiler version detected in CMake: 13.0

But it’s no worries, as this can be overcome with -DCONAN_DISABLE_CHECK_COMPILER=1.

When the conan install succeeds, packages are installed to the local cache on your machine:

$ tree -L 7 ~/.conan/data/
/Users/YOUR-USERNAME/.conan/data/
├── DearImGui
│   └── 1.86.0
│       └── YOUR-PREFIX
│           ├── public
│           │   ├── dl
│           │   │   ├── export
│           │   │   └── pkg
│           │   │       └── f83037eff23ab3a94190d7f3f7b37a2d6d522241
│           │   ├── export
│           │   │   ├── conanfile.py
│           │   │   └── conanmanifest.txt
│           │   ├── locks
│           │   │   └── f83037eff23ab3a94190d7f3f7b37a2d6d522241
│           │   ├── metadata.json
│           │   ├── metadata.json.lock
│           │   └── package
│           │       └── f83037eff23ab3a94190d7f3f7b37a2d6d522241
│           │           ├── cmake
│           │           ├── conaninfo.txt
│           │           ├── conanmanifest.txt
│           │           ├── include
│           │           └── lib
│           ├── public.count
│           └── public.count.lock
├── GLFW
│   └── 3.3.5
│       └── YOUR-PREFIX
│           ├── public
│           │   ├── dl
│           │   │   ├── export
│           │   │   └── pkg
│           │   │       └── f83037eff23ab3a94190d7f3f7b37a2d6d522241
│           │   ├── export
│           │   │   ├── conanfile.py
│           │   │   └── conanmanifest.txt
│           │   ├── locks
│           │   │   └── f83037eff23ab3a94190d7f3f7b37a2d6d522241
│           │   ├── metadata.json
│           │   ├── metadata.json.lock
│           │   └── package
│           │       └── f83037eff23ab3a94190d7f3f7b37a2d6d522241
│           │           ├── conaninfo.txt
│           │           ├── conanmanifest.txt
│           │           ├── include
│           │           └── lib
│           ├── public.count
│           └── public.count.lock
└── glad
    └── 0.1.36
        └── YOUR-PREFIX
            ├── public
            │   ├── dl
            │   │   ├── export
            │   │   └── pkg
            │   │       └── f83037eff23ab3a94190d7f3f7b37a2d6d522241
            │   ├── export
            │   │   ├── conanfile.py
            │   │   └── conanmanifest.txt
            │   ├── locks
            │   │   └── f83037eff23ab3a94190d7f3f7b37a2d6d522241
            │   ├── metadata.json
            │   ├── metadata.json.lock
            │   └── package
            │       └── f83037eff23ab3a94190d7f3f7b37a2d6d522241
            │           ├── cmake
            │           ├── conaninfo.txt
            │           ├── conanmanifest.txt
            │           ├── include
            │           └── lib
            ├── public.count
            └── public.count.lock

You can also list them like this:

$ conan search
Existing package recipes:

DearImGui/1.86.0@YOUR-PREFIX/public
GLFW/3.3.5@YOUR-PREFIX/public
glad/0.1.36@YOUR-PREFIX/public

And the project build directory now looks like this:

$ ls -L1 .
conan.lock
conanbuildinfo.cmake
conanbuildinfo.txt
conaninfo.txt
graph_info.json

The conanbuildinfo.cmake contains auto-generated CMake instructions for finding all the installed dependencies, and you need to add it to your project’s CMakeLists.txt:

include(${CMAKE_CURRENT_BINARY_DIR}/conanbuildinfo.cmake)
conan_basic_setup()

After that you can call find_package() as usual, for example:

find_package(glfw3 CONFIG REQUIRED)

And this is it, the project is ready to build, and it will only build its own sources and link to pre-built dependencies.

Here are all the commands from the creation of the build folder:

$ mkdir build && cd $_
$ conan install .. --profile artifactory -r YOUR-CONAN-REMOTE
$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX="../install" \
    -DUSING_CONAN=1 -DCONAN_DISABLE_CHECK_COMPILER=1 ..

$ time cmake --build . --target install
[3/4] Install the project...
-- Install configuration: "Release"
-- Installing: /path/to/glfw-imgui/install/bin/glfw-imgui/glfw-imgui
-- Installing: /path/to/glfw-imgui/install/bin/glfw-imgui/JetBrainsMono-ExtraLight.ttf

real    0m1.166s

$ ../install/bin/glfw-imgui/glfw-imgui

Absolutely magnificent.

Dependency graph

Speaking about dependencies of dependencies, here’s another nice feature of Conan - it can generate a dependency graph:

$ conan info ../conanfile.py -g ./graph.html
$ open ./graph.html

Here’s an example of a graph from one of our projects that actually depends on E57Format library:

As you can see, E57Format is a direct dependency of the project and in turn E57Format has Xerces-C++ as its own dependency. You can also click on any node and get some more information about it (Zstandard is clicked on that screenshot).

In our project’s conanfile.py we only list direct dependencies:

requires = [
    "Zstandard/1.5.2@YOUR-PREFIX/public",
    "pdqsort/1.0.0@YOUR-PREFIX/public",
    "JsonCpp/1.9.5@YOUR-PREFIX/public",
    "E57Format/2.2.0@YOUR-PREFIX/public"
    # no Xerces-C/3.2.3@YOUR-PREFIX/public"
]

And Conan then takes care of fetching dependencies of dependencies, how convenient is that.

If you add -j argument to conan info, then you’ll get a JSON representation of the graph, which you can easily parse to generate a report about your dependencies (versions, licenses, etc):

$ conan info ../conanfile.py -j ./dependencies.json

Updates

2022-03-03

If you are your own evil twin (сам себе злой Буратино), then you can just fetch it from original repositories on GitHub, GitLab or wherever. If you really are an enemy to yourself, then you can also store your own project source code in an external repository too.

[ ... ]

just as well users from certain sanctioned countries can be denied using a service;

the entire country can lose internet completely, because authorities shut it down;

…Just one month later all this is unexpectedly much more fucking real for developers from Russia. Even though GitHub stated that they are home for all developers:

this can change any time. And GitHub did block access for developers from certain countries before, so.

Also, one can expect more IT companies and services to follow.

2022-10-30

Some months later we started looking into resolving dependencies with vcpkg package manager.

Declaration of VAR

Conan and resolving dependencies in a C++ project

Why bother

Conan

Packages storage

conan_server

Conan Center

JFrog Artifactory

Dependencies

Source code repositories

Project files

conanfile

CI/CD

Projects and configurations

Building

Dependencies of dependencies

Packing

Resolving dependencies in a project

Dependency graph

Updates

2022-03-03

2022-10-30

Feeds

Tags

Conan and resolving dependencies in a C++ project

Why bother

Conan

Packages storage

conan_server

Conan Center

JFrog Artifactory

Dependencies

Source code repositories

Project files

conanfile

CI/CD

Projects and configurations

Building

Dependencies of dependencies

Packing

Resolving dependencies in a project

Dependency graph

Updates

2022-03-03

2022-10-30

Feeds

Social networks

Tags