KDE PIM in Randa 2017

Randa Meetings is an annual meeting of KDE developers in a small village in Swiss Alps. The Randa Meetings is the most productive event I ever attended (since there's nothing much else to do but hack from morning until night and eat Mario's chocolate :-)) and it's very focused - this year main topic is making KDE more accessible.

Several KDE PIM developers will be present as well - and while we will certainly want to hear other's input regarding accessibility of Kontact, our main goal in Randa will be to port away from KDateTime (the KDE4 way of handling date and time in software) to QDateTime (the Qt way of handling date and time). This does not sound very interesting, but it's a very important step for us, as afterward, we will finally be free of all legacy KDE4 code. It is no simple task, but we are confident we can finish the port during the hackfest. If everything goes smoothly, we might even have time for some more cool improvements and fixes in Kontact ;-)

I will also close the KMail User Survey right before the Randa meetings so that we can go over the results and analyze them. So, if you haven't answered the KMail User Survey yet, please do so now and help spread the word! There are still 3 more weeks left to collect as many answers as possible. After Randa, I will be posting a series of blog posts regarding results of the survey.

And finally, please support the Randa Meetings by contributing to our fundraiser - the hackfest can only happen thanks to your support!

Konqi can't wait to go to Randa again!

You can read reports from my previous adventures in Randa Meetings in 2014 and 2015 here:

KDE PIM on Akademy

Akademy is over and so here's a short summary of what the PIMsters have worked on during the past week.

Wiki Cleanup

Me and Volker sat down and went through all KDE PIM wikipages on community.kde.org, userbase.kde.org and techbase.kde.org. Most of our wiki pages are horribly outdated, so we tried to clean them up, remove pages that are no longer relevant or useful. With fewer pages to take care of and better overview of what all content we have, we should be able to keep them more up-to-date than we did in the past years.

Developer Story

Contributing to KDE PIM is hard and we know that. Getting all the dependencies and environment set up correctly is not trivial, and you can't run stable and development Kontact alongside each other easily.

We decided to address all those issues and make contributing to KDE PIM substantially easier. We are working on a Docker image that has all the dependencies and environment set up, so developers just need to run a single command to build entire KDE PIM. And thanks to the containerization, it's also possible to use the development version of Kontact in parallel with the stable version.

We hope that having ready-made development environment it will be easier for new contributors to get involved with KDE PIM. We will post a more details once the Docker image is ready.

Kontact Homepage

Right now kontact.kde.org and kontact.org just redirect to our page on the Userbase wiki. We decided that we want a simple, but professionally-looking web site to market Kontact as an actual product so that it appears more attractive to new users, especially those who will be coming from Windows in the future and contains comprehensive information for both users and developers.

KMail User Survey

During QtCon 2016 we started working on KMail user survey to get a better idea of what our user base is like, how they use KMail and what their impressions of it are. And now the survey is finally live, so please go and fill it if you haven't done so yet.

Wayland Support

Volker has finished porting Kontact to Wayland, so if you have Qt 5.9, you can now run Kontact natively on Wayland. Our main limitation was Wayland support in QWebEngine, which we use to render emails, but that has been resolved in Qt 5.9.

Merging Exchange Support

Krzysztof Nowicki has been working on Microsoft Exchange support for Kontact for a while now. We now have plans to merge his code into kdepim-runtime repository, so if everything goes right the Exchange support will be available out-of-the-box to all our users starting with the December release of KDE Applications.

Next Sprint

We will be meeting soon again in Randa. Our main plan for the sprint is to continue with removal of KDateTime from our code, and thus making KDE PIM free of kdelibs4support.

 

There's some more that I did not mention here, you can check the full notes for details.

Facebook Events in KOrganizer

Sounds like déjà vu? You are right! We used to have Facebook Event sync in KOrganizer back in KDE 4 days thanks to Martin Klapetek. The Facebook Akonadi resource, unfortunately, did not survive through Facebook API changes and our switch to KF5/Qt5.

I'm using a Facebook event sync app on my Android phone, which is very convenient as I get to see all events I am attending, interested in or just invited to directly in my phone's calendar and I can schedule my other events with those in mind. Now I finally grew tired of having to check my phone or Facebook whenever I wanted to schedule event through KOrganizer and I spent a few evenings writing a brand new Facebook Event resource.

Inspired by the Android app the new resource creates several calendars - for events you are attending, events you are interested in, events you have declined and invitations you have not responded to yet. You can configure if you want to receive reminders for each of those.

Additionally, the resource fetches a list of all your friend's birthdays (at least of those who have their birthday visible to their friends) and puts them into a Birthday calendar. You can also configure reminders for those separately.

Facebook Resource Configuration dialog Facebook events in KOrganizer

The Facebook Sync resource will be available in the next KDE Applications feature release in August.

KMail: "Multiple Merge Candidates" error and how to fix it

If you can't synchronize a folder in KMail and you are seeing "Multiple Merge Candidates" error after the synchronization fails, here's a how to fix the folder to make it synchronize again - basically you force KMail to forget and re-sync the folder again.

 

  1. Open Akonadi Console.
  2. Go to the Browser tab.
  3. Right-click the broken folder and select "Clear Akonadi Cache" - this will remove all emails from the folder in Akonadi. This will NOT delete your emails on the server.
  4. Akonadi Console will freeze for a while, wait until it unfreezes (sorry, it's just a developer tool, we don't have a very good UX there :-)).
  5. Logout and login to make sure all PIM components are restarted.

After login start KMail (or Kontact) and hit "Check mail". KMail will now re-download all emails from the previously broken folder. This may take a while depending on how large the folder is and how fast your internet connection is. After that the synchronization should work as expected.

In the upcoming KDE Applications 16.12.1 release Akonadi will have a fix that fixes the reason why the "Multiple Merge Candidates" error occurs, so hopefully in the future you should not see this error anymore.

Akademy 2016 is over :(

It's actually been over for two days, but I'm still sitting in Berlin and only now got to write something.

As every year, it was great to see all my friends and fellow hackers again. Thanks everyone for being so awesome, I enjoyed every day of QtCon and Akademy with you. Can't wait to meet everyone again next year :-)

In the terms of KDE PIM, this year's Akademy was very productive. We had our KDE PIM BoF session on Monday afternoon, where we spent most of the time discussing KDE PIM User Survey - a plan of mine to get more information about our users and their use cases. The results will help us, the KDE PIM devs, to better understand how our users use our software and thus prioritize our focus. We ended up with an initial set of questions we intend to ask our users and next week I'll meet with some more KDE PIM hackers that could not attend Akademy and we will finalize the set of questions so that we can publish the survey later this month.

We also talked about some other topics on the meeting, like releasing of some of our libraries that Kube wants to use and so on.

You can read the mostly complete meeting notes on the KDE PIM wiki.

Outside of the BoF session we touched the topic of KDE PIM sprints and meetings. We want them to be more focused in the future, i.e. having a specific topic for each meeting that we will all work on together. We hope to do one meeting in Autumn this year to finish porting KCalCore away from KDateTime and KDELibs4Support, then a Spring meeting in Toulouse (which has become our new regular place for Spring sprints), then Randaaaaaaaaaaa (which gives us full 6 days of uninterrupted hacking with only small breaks to eat Mario's chocolate :-)) and then it's Akademy time again!

Oh and I can't forget to mention that the KDE PIM team was awarded the Akademy Award for our work on, well, KDE PIM :-). It was a great feeling to stand on the stage knowing that people appreciate our work.

---

Regarding my PIM work during Akademy, I think this year was pretty good. I did my share of partying during QtCon, so I could spent most of Akademy days hacking from morning until they kicked us out from the venue, and then continuing with some more hacking in the KDAB office until late night. Already before the event I merged a big change that improves the Akonadi change notification system. I managed to polish it during Akademy and fix several crashes and bugs.

Another big change was to our test-suite. It contains among other things integration tests, where we run an actual Akonadi server in an isolated environment (so that it does not touch any real data) and test whether clients interact with it as they are supposed to do. For these integration tests we've been only using the SQLite database until now, but I have now enabled MySQL and PostgreSQL too, so we run each test three times - once for each of the backends. This has revealed several corner-case issues that we weren't aware of until now. The test still run into some issues on the CI on build.kde.org but locally they pass for me (with only one exception). Addressing those issues is on the top of my todo list now.

I also started working on an experimental XML->C++ generator, which would allow me to get rid of some 12,000 lines of hand-written C++ code that implements the communication protocol between Akonadi server and the clients. Instead I will generate the code from a simple XML. So far I managed to get it to generate a code that compiles, but there's still a lot of work ahead to make it generate an optimal and correct code.

I'll spend the next week meeting all my colleagues from KDAB, which I'm really looking forward to. Although I know many of them from KDE, there are lots of people I haven't met yet, so it will be great to attach faces to the names. After that, it's back to Prague and to regular work (and some more Akonadi hacking ;-)).

Oh and if you haven't heard yet, KDE is celebrating 20th birthday. Go check out the timeline of KDE and get the amazing "20 Years of KDE" book!

Akonadi - still alive and rocking

It's been a while since I wrote anything about Akonadi but that does not mean I was slacking all the time ;) The KDE PIM team has ported PIM to KDE Frameworks 5 and Qt 5 and released the first KF5-based version in August 2015 and even before that we already did some major changes under the hood that were not possible in the KDE4 version due to API and ABI freezes of kdepimlibs. The KF5-based version of Akonadi libraries (and all the other KDE PIM libraries for that matter) have no guarantees of stable API yet, so we can bend and twist the libraries to our needs to improve stability and performance. Here's an overview of what has happened (mostly in Akonadi) since we started porting to KDE Frameworks 5. It is slightly more technical than I originally intended to, sorry about that.

Human-readable formats are overrated

As you probably know Akonadi has two parts: the Server (that manages the data and resources) and client libraries (that applications use to access the data managed by the server). The libraries need to talk to the Server somehow. In KDE4 we were using a text-based protocol very similar to IMAP (it started as RFC-compliant IMAP implementation, but over the time we diverged a bit). The problem with text-based protocol and large amount of data is that serializing everything into string representation and then deserializing it again on the other end is not very effective. The performance penalty is negligible when talking to IMAP servers over network because the network latency hides it. It however shows when everything is happening locally. So we switched from a text-based protocol to a binary protocol. That means we take the actual representation of the data in the memory (bit by bit) and write it to the socket. The other side then just takes the binary data and directly interprets them as values (numbers or strings or whatever). This means we spent almost zero time on serialization and we are able to transmit large chunks of data between the server and the applications very, very efficiently.

Let's abuse the new cool stuff we have for things we did not originally designed it for

The communication between clients and server needs to work in two directions. It's not just the clients sending requests to server (and server sending back replice), we also need a mechanism for the server to notify the clients that something has changed (new event in a calendar, email marked as read, etc.). For this we were using DBus signals. The clients could connect to a DBus signal provided by the Akonadi Server and when something changed, the server notified the clients via the signal. However during initial synchronization or during intensive mail checks the amount of the messages sent over DBus by Akonadi was just too high. We were clogging the DBus daemon and the transmission of messages via DBus is not cheap either. But hey, we have an awesome and super-fast binary protocol, why not use that? And so we switched from using DBus for change notifications to sending those notifications through the same mechanism that we use for all other communication with the server. In the future it will also allow us tu even more customize the content of the notification thus further improving performance.

Pfff, who needs database indexes?

We do! Once we switched to the binary protocol we found that we are no longer waiting for the data from database to be serialized and sent over to client, but that we are waiting for the database itself! I sat down and look at EXPLAIN ANALYZE results of our biggest queries. Turns out we were doing some unnecessary JOINs (usually to get data that we already had in in-memory cache inside the Server) that we could get rid of. SQL planners and optimizers are extremely efficient nowadays, but JOINing large tables still takes time, so getting rid of those JOINs made the queries up to twice faster.

One unexpected issue I found was that the database was spending large amount of time on "ORDER BY ... DESC" on our main table (yes, we query results in descending order - this way we can show the newest (= usually most relevant) emails in KMail first, while still retrieving the rest). PostgreSQL users will be happy to know that adding a special descending index sped up the queries massively. MySQL users are out of luck - although MySQL allows to create a descending index on a column, it does not really do anything about it.

Splitting libraries is too mainstream, we merge stuff!

One of the things that I've been looking forward to for a very long time was making the Akonadi server a private part (an implementation detail) of the Akonadi client libraries. In KDE4 versions we had to maintain a backwards compatibility of the Akonadi protocol (the text-based one I mentioned earlier) as it was considered a part of public API. This was extremely limiting and annoying for me as a maintainer, as it was making fixing certain bugs and adding new features unnecessarily hard. Historically Akonadi server was a standalone project and it was expected that 3rd party developers would write their own client libraries in their own toolkits/languages. Unfortunately that never happened and the KDE Akonadi client libraries were the only client libraries out there that were actively developed and used (there were some proof-of-concept GLib/Gtk client libraries, but never used seriously).

So, since KDE Applications 15.08 the Akonadi server has no public API and writing custom client libraries is not officially supported. The only official way to talk to the server is through the KDE Akonadi client libraries, which is now the only public API for Akonadi. This may sound like a bad decision, like closing ourselves down from the world, like if we don't care about anyone else but KDE and Qt. And it's sort of true - we were waiting for almost a decade for someone else to start writing their client libraries, but nobody did. So why bother? On the other hand the only actual and real user of Akonadi - KDE - benefits much more now - for example the binary protocol is optimized so that (de)serializing Qt types (like QString or QDateTime) is very efficient because we can use the format that Qt uses internally. If we were to be "toolkit agnostic", we would have to waste time on converting the data to some more standard representation and nobody would win.

To finally get to the point: today I took the Akonadi client libraries (that lived in kdepimlibs.git) and merged them to akonadi.git repository, where the Akonadi server is - at least locally on my machine, still need to fix some build issues before actually pushing this, but I expect to do it tomorrow. In other words the entire Akonadi Framework now lives in a single self-contained git repository. This brings even more benefits, mostly from maintainer point of view. For instance we can now share more code between the server and the libraries that we previously had to duplicate or expose via some private shared library.

The kdepimlibs.git will still contain some libraries for now that we yet have to figure out what to do with, but I guess that eventually kdepimlibs.git will meet the same fate as kdelibs.git - being locked down and preserved only for historical reference.

The Cheese Dependency

In September last year the KDE PIM team also met in Randa in Swiss Alps. I was totally going to blog about it, but then other things got into way and I kept delaying it further and further until now. So with an awkward 5 months delay: huge thanks and hugs to the entire Randa meetings staff and one more hug to Mario just for being Mario. In Randa we met to discuss where KDE PIM should go next and how to get there. After several days on intensive talking we outlined the path into future - you probably read about it already in some of the blogs about AkonadiNext from Christian and Aaron, so I won't go much into that.

To list some of the visible and practical results - we now use Phabricator to coordinate the work in the PIM team and to better communicate what is happening and who's working on what. There's a nice backlog of tasks waiting to be done, so if you want to help us make PIM better feel free to pick up some task and get to work! Furthermore we looked into cleaning up some of the old code and optimizing some critical code-paths - basically a continuation of an effort that started already during Akademy in A Coruña. Some of the changes were already implemented, some are still pending.

Lord of the PIM: The Return of The KJots

One of the major complaints we heard about the new KF5-based KDE PIM was the disappearance of KJots, our note-taking app. Earlier last year, on a PIM sprint in Toulouse, we decided that we need to reduce the size of the code base to keep it maintainable given the current manpower (or rather lack thereof). KJots was one of the projects we decided to kill. What we did not realize back then was that we will effectively prevent people from accessing their notes, since we don't have any other app for that! I apologize for that to all our users, and to restore the balance in the Force I decided to bring KJots back. Not as a part of the main KDE PIM suite but as a standalone app. I have yet to make a first release that packagers can package, but it already builds and is reasonably usable. I'm not planning on developing the application very actively - I'll keep it breathing, but that's about it. That's all I can afford. If there's anyone who would be interesting in maintaining the application and developing it further (it's a rather small and simple application), feel free to step up! When the app reaches certain quality level, we can start thinking about merging it back to KDE PIM.

Is that all?

Yes. No. Well, it's all for today. There is much much more happening in KDE PIM - Laurent did tons of work on of refactoring and splitting the monolithic kdepim.git repository into smaller, better reusable pieces and now seems to be messing around Akregator, and Sandro is actively working on refactoring the email rendering code and calendaring. But I'll leave it up to them to report on their work :) And of course there's much more planned for the future (as always), but this blog post already got a bit out of hand, I'll report on the rest maybe next time I "accidentally" have an energy drink at 11 PM.

And as always: we need help. Like, lots of it. KDE PIM might look huge and scary and hard to work on, but in fact it's all rainbows and unicorns. Hacking on PIM is fun (and we are fun too sometimes!), so if you like KDE (PIM) and would like to help us, let's talk!

KDE PIM in Randa

The first release of KDE PIM based on KDE Frameworks 5 and Qt 5, which will be part of the KDE Applications 15.08 release, is getting closer and closer. Except for porting the entire suite from Qt 4 to Qt 5 the team also managed to fix many bugs, add a few new features and do some pretty big performance and memory optimizations. And we already have some new improvements and optimizations stacked in the development branch which will be released in December!

The biggest performance improvement is thanks to switching to a faster implementation of the communication protocol used by applications to talk to the Akonadi server. We also extended the protocol and we can now use it to send change notifications from the Akonadi server to clients much more effectively than previously. Additionally we started cleaning up API of our libraries and improving it in a way that allows for safer and more effective use. None of this was possible in the KDE 4 version of KDE PIM, where we promised API and ABI compatibility with previous releases. For now we decided not to give any such promises for several more releases, so that we can tune the API and functionality even more.

During Akademy the KDE PIM team had a very long session where we analyzed where the project currently stands and we created a vision of where we want KDE PIM to be in the future. We know what parts we want to focus on more now and which parts are less relevant to us. KDE PIM is a huge and rather complicated project, unfortunately the development team is very small and so we have to make the hard and painful decision to lay off some of the features and functionality in exchange for improvement in reliability and user experience of the core parts of the product.

In order to make these decisions the team is going to meet again in couple weeks in Randa alongside many other KDE contributors and projects and will spend there a whole week. During the sprint we want to take a close look at all the parts and evaluate what to do with them as well as plan how to proceed towards Akonadi Next - the new version of Akonadi, which has some major changes in architecture and overall design (see the Christian's talk from Akademy about Akonadi Next).

However organizing such sprint is not easy and so we would like to ask for your support by donating to the KDE Sprints Fundraiser. Although the attendees cover some of the costs themselves, there are still expenses like travel and accommodation that need to be covered. This year the Fundraiser has been extended so that the collected money will also be used to support additional KDE sprints throughout the year.

What happened in Toulouse?

... a KDE PIM sprint happened in Toulouse! And what happened during that sprint? Well, read this wholly incomplete report!

Let's start with the most important part: we decided what to do next! On the last PIM sprint in Munich in November when Christian and Aaron introduced their new concept for next version of Akonadi, we decided to refocus all our efforts on working on that which meant switching to maintenance mode of KDE PIM for a very long time and then coming back with a big boom. In Toulouse we discussed this plan again and decided that it will be much better for the project and for the users as well if we continue active development of KDE PIM instead of focusing exclusively on the "next big thing" and take the one-step-at-the-time approach. So what does that mean?

We will aim towards releasing KF5-based KDE PIM in August as part of KDE Applications 15.08. After that we will be working on fixing bugs, improving the current code and adding new features like normally, while at the same time preparing the code base for migration to Akonadi 2 (currently we call it Akonadi Next but I think eventually it will become "2"). I will probably write a separate technical blog post on what those "preparations" mean. In the meantime Christian will be working from the other side on Akonadi 2 and eventually both projects should meet "in the middle", where we simply swap the Akonadi 1 backend with the Akonadi 2 backend and ship next version. So instead of one "big boom" release where we would switch to Qt 5 and Akonadi 2 at the same time we do it step-by-step, causing as little disruption to user experience as possible and allowing for active development of the project. In other words WIN-WIN-WIN situation for users, devs and the KDE PIM project.

I'm currently running the entire KDE PIM from git master (so KF5-based) and I must say that everything works very well so far. There are some regression against the KDE 4 version but nothing we couldn't handle. If you like to use bleeding-edge versions of PIM feel free to update and help us finding (and fixing) regressions (just be careful not to bleed to death ;-)).

Another discussion we had is closely related to the 15.08 release. KDE PIM is a very huge code base, but the active development team is very small. Even with the incredible Laurent Montel on our side it's still not enough to keep actively maintaining all of the KDE PIM (yes, it's THAT huge ;-)). So we had to make a tough decision: some parts of KDE PIM have to die, at least until a new maintainer steps up, and some will move to extragear and will live their own lives there. What we release as part of KDE Applications 15.08 I call KDE PIM Core and it consists of the core PIM applications: KMail, KOrganizer, KAddressbook, Kleopatra, KNotes and Kontact. If your favorite PIM app is not in the list you can volunteer as a maintainer and help us make it part of the core again. We believe that in this case quality is more important than quantity and this is the trade-off that will allow us to make the next release of PIM the best one to date ;-).

Still related to the release is also reorganization of our repos, as we have some more splitting and indeed some merging ahead of us but we'll post an announcement once everything is discussed and agreed upon.

Thanks to Christian's hard work most of the changes that Kolab did in their fork of KDE PIM has been upstreamed during the sprint. There are some very nice optimizations and performance improvements for Akonadi included (among other things), so indeed the next release will be a really shiny one and there's a lot to look forward to.

Vishesh brought up the topic of our bug count situation. We all realize the sad state of our PIM bugs and we talked a bit about re-organizing and cleaning up our bug tracker. The clean up part has already begun as Laurent with Vishesh have mass-closed over 850 old KMail 1 bugs during the sprint to make it at least a little easier to get through the rest. Regarding the re-organization I still have to send a mail about it but a short summary would be that we want to remove bugzilla components and close bugs for the apps we decided to discontinue and maybe do a few more clean up rounds for the existing bugs.

I'm sure I've forgotten something because much more happened during the sprint but let's just say I'm leaving some topics for others to blog about ;-).

Huge thank you to Franck Arrecot and Kevin Ottens for taking care of us and securing the venue for the sprint! All in all it was a great sprint and I'm happy to say that we are back on track to dominate the world of PIM.

The only disappointment of the entire sprint was my failure to acquire a French beer. I managed to try Belgian, Spanish, Mexican and Argentinian beer but they did not serve any French beer anywhere. Either there's no such thing or it must be really bad...:-)

KDE PIM Sprint in Toulouse We had a great dinner with the local KDE people on Saturday. Also a prove that Laurent is a real person :-D

I'm going to Akademy 2014 in Brno!

What am I going to do there? I will give a short talk on Saturday at 11:35 about what we have achieved in Akonadi in the past year and I'll be attending the KDE PIM BoF on Tuesday morning - you are all invited of course to join us in discussions about the future of KDE PIM! Possibly I'll also go to KTp BoF (purely for nostalgic reasons) and Solid BoF (did someone mention KScreen?).

I tried to list some talks I want to attend, but as I was browsing the program I realized I want to attend all of them. Seriously! So many interesting talks and so many things to learn this year! Any progress on the self-duplicating technology, so I could attend two talks at the same time? :-)

I'm really excited about this year Akademy. For us in Brno the Akademy has already begun in a sense with all the planing and preparations, but the real Akademy starts when everyone is here. I'm really looking forward to meet with all my friends from KDE again!

By the way, if you arrive on Friday (or earlier), stop by at the brand new Red Hat office for the pre-registration event. Food, drinks and fun are guaranteed!

I'm going to Akademy 2014

See you all in couple days! *hug*

No Gmail integration in 4.14 after all :(

Hi folks,

I'm sorry to bring bad news, but after trying to fight some last minute bugs in the new Gmail resource today, I realized that pushing the resource into KDE Applications 4.14 was too hurried, and so I decided not to ship it in KDE Applications 4.14. I know many of you are really excited about the Gmail integration, but there are far too many issues that cannot be solved this late in 4.14 cycle. And since this will probably be the last 4.x release, shipping something that does not perform as expected and cannot be fixed properly would only be disappointing and discouraging to users. In my original post I explained that I was working on the Gmail integration to provide user experience as close as possible to native Gmail web interface so that people are not tempted to switch away from KMail to Gmail. But with the current state of the resource, the effect would be exactly the opposite. And if the resource cannot fulfil it's purpose, then there's no point in offering it to users.

Instead I will focus on implementing the new native Gmail API and merging together the existing Google resources to create a single groupware solution that will provide integration will all Google's PIM services - contacts, calendars, tasks and emails.