Status report: Community Bonding
Good morning everyone!
I’m checking in today to let you know what I did in my GSoC project these past weeks. This Community Bonding period was really wonderful; although I’ve been more or less involved with the project since 2016, I’ve acquainted myself with the efforts of each of the members, and so far it’s been a wonderful experience.
During these past weeks, I’ve been preparing for the coding period by talking with Boudewijn and Wolthera about the particulars of Krita’s file format and build system. The objectives for the past two meetings were:
- speccing the new layer
- integrating SeExpr into Krita’s dependency build system
- document the process
This post summarizes my efforts on each of these bits. I apologise for the late post; but this week I got to attend (virtually) my first Eurographics and LGM sessions, and with the homework load, it’s been some wild days.
Without further ado, let’s see how deep this rabbit hole goes.
I’m glad to report that SeExpr can be successfully built as a dependency! Currently, the build process points to my own fork of SeExpr. This is for two reasons: first, for GSoC evaluation purposes, my mentors need to be notified of every commit I make (in this case, both in Krita and in the SeExpr library itself). But the other, and most important, is that SeExpr, as designed by Disney, is natively incompatible with our compilation toolchain in Windows. When Disney developers wrote the library, they made a critical assumption, that a Windows compilation would always be made with Visual C++. Also, since their parser is written with Bison and FLEX, they required Windows users to copy bundled, pre-generated files before being able to compile the library.
My fixes were four-fold:
- Detect MinGW, skip the MSVC bits accordingly, and fix all the lint warnings
- Improve Bison and FLEX detection by using
- Regenerate the parser files with up-to-date versions of Bison and FLEX
- And finally, do the copy dance automatically if needed.
With this process, I was able to get successful builds on both Windows and macOS. (There’s a bug that Iván fixed quickly in the
script, but as it was easily worked around, I’ve yet to cherry-pick it into my branch.)
Building AppImages: oh my…
The process for Linux wasn’t so smooth, unfortunately. The build process I demoed in a previous post uses Manjaro’s system libraries, and since it’s rolling release distribution, it guarantees that almost no one will be able to run a demo. Since tiar usually asks users to test using AppImages, I decided to take that route, and attempted to build them… blindly.
I spent a whole weekend fixing the ensuing mess.
The AppLauncher mess
To begin with, Manjaro comes with an AppImage integration called AppLauncher that hooks into any AppImage that tries to run and helps you integrate it with your system. The bad news is that, as a system application, it uses the local version of Qt, 5.14. Why is this important?
Turns out, the build scripts use a packer,
linuxdeployqt. I’d naïvely thought this was bundled with Qt (like
windeployqt), but it’s an AppImage on its own right. And since the build scripts patch
LD_LIBRARY_PATH, in order for it to find our (heavily patched) Qt 5.12 distribution, I got very unhelpful crashes from AppLauncher. I uninstalled it once I recognized the Qt version mismatch, but it kept crying “file or directory not found”.
tl;dr: uninstall AppLauncher and reboot your system before using
My Ryzen 7 box packs quite a punch so, thankfully, the above didn’t waste too much time. The real problem arose, when I noticed that
linuxdeployqt automatically detects and refuses to run if your system is too new. This is to prevent incompatibilities between the AppImage and the version of the
glibc library it needs.
I needed a way to isolate my Krita build and use a least common denominator distribution. Thanks to my best friend’s sysadmin tales, I knew I had a solution in the shape of a LXC container.
tl;dr: I installed LXD (the Snap version),
sudo snap install lxd
added my account to the
lxd group to skip the
sudo usermod -a -G lxd amalia
After the customary reboot, I initialized LXD with all the defaults,
created a new container (Ubuntu 18.04 is still too new for AppImages, but it helped to satisfy my learning needs),
lxc launch ubuntu:18.04 krita -c security.privileged true
~/krita folder as a writable device (which is why I need
lxc config device add krita src disk source=$HOME/krita path=/krita
and relaunched the container:
lxc stop krita lxc start krita
With the container set up, I could start building the dependencies… But wait, what packages do we actually need?
Yes, this step felt like another whack-a-mole of mistakes. The dependencies’ build script is notoriously unhelpful:
# Prerequisites: cmake git build-essential libxcb-keysyms1-dev plus all deps for Qt5
After a whole day of getting complaints from each step, and installing the required packages, I managed to determine exactly which dependencies are needed for Krita. For the build process in general,
sudo apt-get install build-essential cmake libgl1-dev libglu1-mesa-dev
and additionally, for the KDE libraries,
sudo apt-get install libxcb-keysyms1-dev libxcb-res0-dev libxrender-dev libxkbcommon-dev
But there are hidden dependencies that aren’t listed anywhere, perhaps because we bundle them in
3rdparty yet we do not use them in Linux. The GMic plugin needs:
sudo apt-get install libtiff5-dev
exiv2 library needs:
sudo apt-get install zlib1g-dev
And, finally, Qt itself:
sudo apt-get install libssl-dev gettext libfontconfig1-dev
With the dependencies installed, I added the two remaining utilities,
wget -c -nv "https://github.com/probonopd/linuxdeployqt/releases/download/continuous/linuxdeployqt-continuous-x86_64.AppImage" -O /usr/local/bin/linuxdeployqt chmod a+x /usr/local/bin/linuxdeployqt cd /tmp/ wget -c -nv https://nixos.org/releases/patchelf/patchelf-0.9/patchelf-0.9.tar.bz2 tar -xf patchelf-0.9.tar.bz2 cd patchelf-0.9 ./configure -prefix=/usr/local make -j4 install cd ~ rm -rf /tmp/patchelf-*
By this time, I had realised there was an official KDE repo listing everything I needed to set up such a container,
sysadmin/ci-tooling. You can save time and headaches by pulling the
kdeorg/appimage-ubuntu1604 Docker image, or by using Dmitry’s scripts.
Dmitry and Agata asked:
- why I don’t follow the Docker route
- how I’d code and debug Krita under this process
- and whether there was any difference between Docker and LXC.
I’d like to clarify that the main development will be done under Windows, as that’s the most clean environment possible; it doesn’t ship Qt nor any other libraries that could interfere. For macOS, I have my MacBook Pro with Mojave. And for the AppImages, the LXC container resides in a separate SSD with the Manjaro install I run.
Finally, I prefer LXC to Docker because Docker (without e.g.
docker-compose), is oriented to ephemeral containers, that need a lot of command line flags to set themselves up. I’m also too prone to
docker image prune --all, and I wouldn’t want to download a multi-GB image each time I build Krita
. And since the LXC container runs in the background, I can have it always up and pull the demonstration AppImage very easily – with just a copy-paste operation.
The new addition: a ‘SeExpr’ generator plugin
Two weeks ago, Wolthera sent me her MR for the KRA file format documentation. The gist of this is that each KRA file is a Zip container. It has a manifest in
maindoc.xml listing, among other things, the component layers’ properties.
To the above, I’ll add a new layer type identified as
SeExpr. My plan was to store the code in the layer node itself. In heavily simplified terms, it would look like this:
<layer nodetype="SeExpr"> <![CDATA[ $val=voronoi(5*[$u,$v,.5],4,.6,.2); $color=ccurve($val, 0.000, [0.141, 0.059, 0.051], 4, 0.185, [0.302, 0.176, 0.122], 4, 0.301, [0.651, 0.447, 0.165], 4, 0.462, [0.976, 0.976, 0.976], 4); $color ]]> </layer>
However, Boudewijn suggested I should instead follow the existing pattern and move the SeExpr rendering code to a separate file. This should make it easier to repair KRA files if they are corrupted. I have not done any further work on this, especially the UX bits; hopefully next week, I’ll be able to show you something
I have some more work to do as regards dependencies. I noticed Krita doesn’t maintain a list of the supported libraries for each operating system; for instance, the
exiv2 library uses
zlib, yet we don’t ship it in Linux. I’d like to sit down and document the whole list, and the necessary fixes for ensuring a consistent feature set. Next, I’ll be dissecting the SeExpr library, and why I elected to bundle only a limited set of features.
In the meanwhile, please chip in on David’s thread at Krita-Artists.org and let me know your expectations!
Until next time,