ORX code reorganization proposal

edited September 2019 in Projects - Tools

Hello, I have some proposals regarding orx dev-ops that may help to add flexibility in building orx with different configurations and plugins.

  1. Change premake and use CMake instead:
    I know you may dislike CMake (It's really ugly and generates a lot of garbage, syntax sucks) but it's a de facto standard and it's better than having a custom modified old premkake 4 version.
    More important, it fits really well with proposal 2.

  2. Use conan.io as package manager:
    No more need to have a complete custom package management with rebol, downloading zips, etc.
    You simply create a packet for each dependency (if vanilla may be they are already available) and declare on conan recipe that you need such packet.
    Users will use conan directly in their development flow and just add something like "orx/1.10.0@orx/stable" and they will obtain all needed stuff to build the game.
    This gives a you a quite standard way to create and distribute pre-built binaries.
    In the same time allows users to automatically build orx and dependencies in case a pre-build for such exotic configuration does not exists.

  3. Divide orx code base in multiple packages:
    This would allow plugin injection on a vanilla orx version just by overriding a dependency using conan.
    Just to make an example, assume I have a modified orxPlugin_Display that can expose the GLFW window identifier to the orx host application.
    With just one line of configuration I could substitute such orxPlugin_Display at compile time without loosing benefits of being aligned to last orx vanilla version.
    Orx code-base seems already ready to be conan-ized as it really well separated between core and plugins.

Of course it's a lot of stuff to be done but if you're interested I can help you with that as I'm doing this on all projects at my workplace (~50+ repos now converted to packages and inserted in CI).

Comments

  • Hi @ainvar and sorry for the late reply, I had quite a busy week on my side. :)

    First of all, thank you for your proposal. Here's what I think:

    1. I really do not like CMake. In addition to having a terrible syntax, it also adds a dependency to end users, unless this has changed in the past few years (I admit it's been a while since I've last used it). I do not think it's a good idea to switch to CMake, it wouldn't bring us anything substantial: being a "de facto" standard isn't sufficient per se, and your #2 would still work without CMake, from what I understand of their documentation, even if it's not already integrated.

    2. This sounds more interesting to me already. However the workflow isn't clear to me at the moment. How would that work for the end user? The main goal being to remove as much friction as possible and have the simplest experience possible from a user point of view. At the moment, all that users need in order to build orx is a copy of the source code, be it a source clone or an archive dump, call a single setup script (only the first time in the case of a source repository clone), and that's it. They can pick the version they want with a commit granularity, dependencies will always match that version, without any extra work from their point of view.

    3. This is one of the goals we'd like to achieve. A few years ago, @enobayram started working on a git-based approach for that very same purpose. I'm not sold on the separation of the main repository into multiple small ones, simply because it'll incur a highest maintenance cost on my side, especially considering that we mirror the hg source repository to git. It'll also bring dependencies issues are some cross-plugins changes would be not be contained in a single commit anymore. As a side note, you can already select your own plugins quite easily. If you use the runtime-loaded plugin approach, you simply pick the ones you want instead of the default set. If you use the compile-time embedded plugin approach (the default one in orx nowadays), it's already a single line to change in orxPlugin_EmbeddedList.cpp. The most interesting part would not be for plugins but for customizing the engine with extra features, such as Scroll, orxArchive or orxRemote to cite a few.

    To sum it up, I'm not opposed to #2 and #3 as long as there's a real gain from a user point of view and doesn't disproportionately increase the maintenance cost when compared to the gains.

  • Hello @iarwain , no problem we're all busy.
    I'll answer point by point so we keep it easy to follow points:

    1. I agree. CMake really sucks.
      But I don't understand what you mean by "it also adds a dependency to end users". Apart from CMake itself (if you develop it's easy you have it already installed) no more dependencies are required by the end user.

    Being "de facto" standard does not mean that works well, it only means that any new tool maker (like conan guys) will spend more effort in supporting CMake than Premake because nobody on the market uses Premake.
    I really like premake because it outputs really clean solutions and project files in fact I started using it at work but at the end I had to switch to CMake because it was easier to build scripts (full of documentation, examples and pre-made stuff) and in Premake I found really hard (or impossible) to make precompiled headers to work.
    In a single day I was able to convert all (16) projects to use CMake and precompiled headers were working.

    Yes, conan works with Premake and since we do not need precompiled headers it's ok.
    But Premake do not work with conan!

    I mean, in cmake file you can
    include(${CMAKE_BINARY_DIR}/conan.cmake)
    and then call
    conan_cmake_run(CONANFILE conanfile.txt ...
    to run conan from cmake and conan will automagically fill all needed to cmake in order to build a compiling solution.
    Using premake you have to manually run conan before cmake. Of course you can solve with a script to automate all.

    1. The workflow, for the end user is quite simple. Here's an example:

    my_game
    |__ include
    |__ src
    - CMakeLists.txt
    - conanfile.txt

    In your "my_game" application folder you just place a conanfile.txt with the following contents:

    [requires]
    spdlog/1.3.1@bincrafters/stable
    imgui/1.69@bincrafters/stable
    orx/1.9.0@orxproject/stable
    ``
    [generators]
    cmake
    premake

    And you're ok!

    If using cmake you just have to create a build sub_folder, enter the sub_folder and type:
    cmake ..
    You'll ready to start developing as the solution/project/makefile/whatever prepared by cmake already lets you to open your main.cpp and code because all dependencies are ready to be used.
    If you need to add other dependencies you just edit the conanfile.txt and re-run cmake.

    If you have Premake you have to run
    conan install . (if I recall right)
    and then run premake.
    Then you'll be ready.

    1. I understand. It was my doubt before moving from submodules to packages. But if you separate things like this you will have all under control:
    • orc_core (depends on nothing)
    • orx_plugins (depends on core)
    • orx_engine (depends on core and plugins)

    In this way, the package the user will depends on will be "orx_engine" and in the conanfile.txt of orx_engine project there will be the working combination of core/plugins.
    In my_game conanfile.txt I can override a plugin just by explicitly typing it into the txt file.

    The workflow is not so different. You have to think that when you work on a library you're the library maintainer and before running the application you have to publish it into a repo. Then when you move back to the application you're the application developer that consumes the library.
    For small projects sounds overkill, but, from my experience, when the project and the team grows and it saves you from a lot of headaches.

  • edited September 2019

    @ainvar said:
    1. I agree. CMake really sucks.
    But I don't understand what you mean by "it also adds a dependency to end users". Apart from CMake itself (if you develop it's easy you have it already installed) no more dependencies are required by the end user.

    That's it, end users will be required to install CMake. That's an extra implicit dependency. And exactly the opposite of what I want. I do not want to have users to have to install anything more than what is strictly necessary to compile C/C++ code.

    That includes the likes of Python, CMake, Conan, etc. The less implicit dependencies like those there are, the easier it is for users to get orx working.

    I really like premake because it outputs really clean solutions and project files in fact I started using it at work but at the end I had to switch to CMake because it was easier to build scripts (full of documentation, examples and pre-made stuff) and in Premake I found really hard (or impossible) to make precompiled headers to work.

    Mmh, we never had any problems with premake and precompiled headers in the past (on Child of Light, for example). Granted, I wasn't the one taking care of the premake setup on those projects, so maybe it's something I've simply not been exposed to.

    Yes, conan works with Premake and since we do not need precompiled headers it's ok.
    But Premake do not work with conan!

    I mean, in cmake file you can
    include(${CMAKE_BINARY_DIR}/conan.cmake)
    and then call
    conan_cmake_run(CONANFILE conanfile.txt ...
    to run conan from cmake and conan will automagically fill all needed to cmake in order to build a compiling solution.
    Using premake you have to manually run conan before cmake. Of course you can solve with a script to automate all.

    That's not entirely true. Premake is extensible and we could easily add support for Conan to it, by extending the collection of LUA scripts that premake relies on to support targets and actions.

    1. The workflow, for the end user is quite simple. Here's an example:

    my_game
    |__ include
    |__ src
    - CMakeLists.txt
    - conanfile.txt

    In your "my_game" application folder you just place a conanfile.txt with the following contents:

    [requires]
    spdlog/1.3.1@bincrafters/stable
    imgui/1.69@bincrafters/stable
    orx/1.9.0@orxproject/stable
    ``
    [generators]
    cmake
    premake

    And you're ok!

    That's... not something I'd call "simple". Quite the opposite. Compare all the above to init my_game -with imgui spdlog which is similar to what I'd like to get in the end. Based on the version of orx that the user chose (where $ORX is pointing), all the dependencies and revisions will be handled behind the scene, without unnecessary exposure to the user.

    If using cmake you just have to create a build sub_folder, enter the sub_folder and type:
    cmake ..

    Here we go back to having users install CMake as a prerequisite. I do not want this.

    1. I understand. It was my doubt before moving from submodules to packages. But if you separate things like this you will have all under control:
    • orc_core (depends on nothing)
    • orx_plugins (depends on core)
    • orx_engine (depends on core and plugins)

    The issue is not the submodule approach itself, it's that cross-support between hg's subrepositories and git's submodules wasn't working correctly last time I checked. This might have improved, but it still add in complexity and I do not see any advantages in doing so.
    Having extensions (ie. imgui, scroll, etc...) as subs would be interesting, but not for what is developed almost entirely by a single person, and tightly coupled together.

    In this way, the package the user will depends on will be "orx_engine" and in the conanfile.txt of orx_engine project there will be the working combination of core/plugins.
    In my_game conanfile.txt I can override a plugin just by explicitly typing it into the txt file.

    You can already override a plugin as mentioned above, with the exact same simplicity as described here and with the advantage of it being already available and not depending on an external tool that would become a prerequisite.

    The workflow is not so different. You have to think that when you work on a library you're the library maintainer and before running the application you have to publish it into a repo. Then when you move back to the application you're the application developer that consumes the library.

    I'm not opposed to the approach but I am opposed to any additional prerequisites to achieve it.

    For small projects sounds overkill, but, from my experience, when the project and the team grows and it saves you from a lot of headaches.

    I guess it depends on the teams and projects. I can guarantee that would have been too complicated for most gamedev projects I've worked on, with team sizes ranging from a couple of dozens of people to many hundreds ones, where source control remains extremely linear and almost always based on Perforce.

  • Ok, I understood your will.

    If in the future I'll have time I'll release a conan recipe (and hopefully some prebuilt binaries ready to be used) on my bintray account for those who wants to use it.

  • Sounds like a good idea. When it's done, maybe we can have others users try both approach and decide which one suits their needs best. Having additional points of view on that topic will definitely be helpful.

Sign In or Register to comment.