Akonadi – still alive and rocking

Akonadi
Standard

It’s been a while since I wrote anything about Akonadi but that does not mean I was slacking all the time ;) The KDE PIM team has ported PIM to KDE Frameworks 5 and Qt 5 and released the first KF5-based version in August 2015 and even before that we already did some major changes under the hood that were not possible in the KDE4 version due to API and ABI freezes of kdepimlibs. The KF5-based version of Akonadi libraries (and all the other KDE PIM libraries for that matter) have no guarantees of stable API yet, so we can bend and twist the libraries to our needs to improve stability and performance. Here’s an overview of what has happened (mostly in Akonadi) since we started porting to KDE Frameworks 5. It is slightly more technical than I originally intended to, sorry about that.

Human-readable formats are overrated

As you probably know Akonadi has two parts: the Server (that manages the data and resources) and client libraries (that applications use to access the data managed by the server). The libraries need to talk to the Server somehow. In KDE4 we were using a text-based protocol very similar to IMAP (it started as RFC-compliant IMAP implementation, but over the time we diverged a bit). The problem with text-based protocol and large amount of data is that serializing everything into string representation and then deserializing it again on the other end is not very effective. The performance penalty is negligible when talking to IMAP servers over network because the network latency hides it. It however shows when everything is happening locally. So we switched from a text-based protocol to a binary protocol. That means we take the actual representation of the data in the memory (bit by bit) and write it to the socket. The other side then just takes the binary data and directly interprets them as values (numbers or strings or whatever). This means we spent almost zero time on serialization and we are able to transmit large chunks of data between the server and the applications very, very efficiently.

Let’s abuse the new cool stuff we have for things we did not originally designed it for

The communication between clients and server needs to work in two directions. It’s not just the clients sending requests to server (and server sending back replice), we also need a mechanism for the server to notify the clients that something has changed (new event in a calendar, email marked as read, etc.). For this we were using DBus signals. The clients could connect to a DBus signal provided by the Akonadi Server and when something changed, the server notified the clients via the signal. However during initial synchronization or during intensive mail checks the amount of the messages sent over DBus by Akonadi was just too high. We were clogging the DBus daemon and the transmission of messages via DBus is not cheap either. But hey, we have an awesome and super-fast binary protocol, why not use that? And so we switched from using DBus for change notifications to sending those notifications through the same mechanism that we use for all other communication with the server. In the future it will also allow us tu even more customize the content of the notification thus further improving performance.

Pfff, who needs database indexes?

We do! Once we switched to the binary protocol we found that we are no longer waiting for the data from database to be serialized and sent over to client, but that we are waiting for the database itself! I sat down and look at EXPLAIN ANALYZE results of our biggest queries. Turns out we were doing some unnecessary JOINs (usually to get data that we already had in in-memory cache inside the Server) that we could get rid of. SQL planners and optimizers are extremely efficient nowadays, but JOINing large tables still takes time, so getting rid of those JOINs made the queries up to twice faster.

One unexpected issue I found was that the database was spending large amount of time on “ORDER BY … DESC” on our main table (yes, we query results in descending order – this way we can show the newest (= usually most relevant) emails in KMail first, while still retrieving the rest). PostgreSQL users will be happy to know that adding a special descending index sped up the queries massively. MySQL users are out of luck – although MySQL allows to create a descending index on a column, it does not really do anything about it.

Splitting libraries is too mainstream, we merge stuff!

One of the things that I’ve been looking forward to for a very long time was making the Akonadi server a private part (an implementation detail) of the Akonadi client libraries. In KDE4 versions we had to maintain a backwards compatibility of the Akonadi protocol (the text-based one I mentioned earlier) as it was considered a part of public API. This was extremely limiting and annoying for me as a maintainer, as it was making fixing certain bugs and adding new features unnecessarily hard. Historically Akonadi server was a standalone project and it was expected that 3rd party developers would write their own client libraries in their own toolkits/languages. Unfortunately that never happened and the KDE Akonadi client libraries were the only client libraries out there that were actively developed and used (there were some proof-of-concept GLib/Gtk client libraries, but never used seriously).

So, since KDE Applications 15.08 the Akonadi server has no public API and writing custom client libraries is not officially supported. The only official way to talk to the server is through the KDE Akonadi client libraries, which is now the only public API for Akonadi. This may sound like a bad decision, like closing ourselves down from the world, like if we don’t care about anyone else but KDE and Qt. And it’s sort of true – we were waiting for almost a decade for someone else to start writing their client libraries, but nobody did. So why bother? On the other hand the only actual and real user of Akonadi – KDE – benefits much more now – for example the binary protocol is optimized so that (de)serializing Qt types (like QString or QDateTime) is very efficient because we can use the format that Qt uses internally. If we were to be “toolkit agnostic”, we would have to waste time on converting the data to some more standard representation and nobody would win.

To finally get to the point: today I took the Akonadi client libraries (that lived in kdepimlibs.git) and merged them to akonadi.git repository, where the Akonadi server is – at least locally on my machine, still need to fix some build issues before actually pushing this, but I expect to do it tomorrow. In other words the entire Akonadi Framework now lives in a single self-contained git repository. This brings even more benefits, mostly from maintainer point of view. For instance we can now share more code between the server and the libraries that we previously had to duplicate or expose via some private shared library.

The kdepimlibs.git will still contain some libraries for now that we yet have to figure out what to do with, but I guess that eventually kdepimlibs.git will meet the same fate as kdelibs.git – being locked down and preserved only for historical reference.

The Cheese Dependency

In September last year the KDE PIM team also met in Randa in Swiss Alps. I was totally going to blog about it, but then other things got into way and I kept delaying it further and further until now. So with an awkward 5 months delay: huge thanks and hugs to the entire Randa meetings staff and one more hug to Mario just for being Mario. In Randa we met to discuss where KDE PIM should go next and how to get there. After several days on intensive talking we outlined the path into future – you probably read about it already in some of the blogs about AkonadiNext from Christian and Aaron, so I won’t go much into that.

To list some of the visible and practical results – we now use Phabricator to coordinate the work in the PIM team and to better communicate what is happening and who’s working on what. There’s a nice backlog of tasks waiting to be done, so if you want to help us make PIM better feel free to pick up some task and get to work! Furthermore we looked into cleaning up some of the old code and optimizing some critical code-paths – basically a continuation of an effort that started already during Akademy in A Coruña. Some of the changes were already implemented, some are still pending.

Lord of the PIM: The Return of The KJots

One of the major complaints we heard about the new KF5-based KDE PIM was the disappearance of KJots, our note-taking app. Earlier last year, on a PIM sprint in Toulouse, we decided that we need to reduce the size of the code base to keep it maintainable given the current manpower (or rather lack thereof). KJots was one of the projects we decided to kill. What we did not realize back then was that we will effectively prevent people from accessing their notes, since we don’t have any other app for that! I apologize for that to all our users, and to restore the balance in the Force I decided to bring KJots back. Not as a part of the main KDE PIM suite but as a standalone app. I have yet to make a first release that packagers can package, but it already builds and is reasonably usable. I’m not planning on developing the application very actively – I’ll keep it breathing, but that’s about it. That’s all I can afford. If there’s anyone who would be interesting in maintaining the application and developing it further (it’s a rather small and simple application), feel free to step up! When the app reaches certain quality level, we can start thinking about merging it back to KDE PIM.

Is that all?

Yes. No. Well, it’s all for today. There is much much more happening in KDE PIM – Laurent did tons of work on of refactoring and splitting the monolithic kdepim.git repository into smaller, better reusable pieces and now seems to be messing around Akregator, and Sandro is actively working on refactoring the email rendering code and calendaring. But I’ll leave it up to them to report on their work :) And of course there’s much more planned for the future (as always), but this blog post already got a bit out of hand, I’ll report on the rest maybe next time I “accidentally” have an energy drink at 11 PM.

And as always: we need help. Like, lots of it. KDE PIM might look huge and scary and hard to work on, but in fact it’s all rainbows and unicorns. Hacking on PIM is fun (and we are fun too sometimes!), so if you like KDE (PIM) and would like to help us, let’s talk!

45 thoughts on “Akonadi – still alive and rocking

  1. Jiri Eischmann wrote an article on email clients in Fedora. He made some comments as follows.

    > Lately, though somewhat lost popularity, reflected in the results of the survey.
    > …
    > Users complain primarily to the slowness and high system requirements Akonadi backend.
    > …
    > Disadvantages – Akonadi backend system requirements

    Source:- http://mojefedora.cz/e-mailove-klienty-ve-fedore/

    I disagree with it. Things like slow or high system requirements are subjective. I don’t find it slow at all. Also high system requirements or lots of dependencies should also be true for someone installing Evolution on KDE.

    Same article will be published on Fedora Magazine. How would you respond to allegations that Akonadi is bringing down KDE PIM?

    • RJVB

      This is an often heard critique, not untrue if you compare KDE 3 to KDE 4. From the article above I’d say things are getting better in this aspect.

    • I would lie if I said that everything is perfect in the land of Akonadi. However we are improving a lot, compared to the requirements Akonadi had several years ago, we are doing very very good. I think most of the complaints are just because people like to complain about stuff. Like with PulseAudio or KDE4 – it had a rough start so people were complaining about it all the time. The software has improved over the time, but users did not and they always find something to complain about, no matter how hard you try. Some of it also comes from lack of information or insight into the technology – we had big issues with Nepomuk back in the day so many people decided that indexing is evil and they will not accept that without it we cannot work properly, even though we have replaced the technology completely with something that actually works.

      We use a database server as a backend to Akonadi, which understandably can be unsettling for many people and which imposes some additional resource requirements. Our multi-process architecture also has some memory requirements, on the other hand provides robustness and better performance. Nowadays people seem to buy gigabytes of RAM and then complain that we are actually using them.

      There are still many areas where we can improve both memory and CPU wise and we are well aware of them, but the lack of time and manpower makes it harder to fix these while keeping the entire thing rolling in the right direction.

      • Sorry, the reply sounds harsher than I intended. Don’t get me wrong, I value any feedback from users, even negative one, but only as long as it is constructive and based on facts. Unfortunately most of the feedback I’m hearing is the “Akonadi sucks because I have no idea what I’m talking about”-kind of stuff.

        • 1. I don’t think as a user I should even talk about Akonadi. Why would I even care about it if my email client seems to work fine. That is a technical topic that must be debated among who really understand it. Like you said people constantly complain about things they are not qualified to talk about. That’s also a big problem with Linux media who will start rolling eyes at any mention of say Pulseaudio or Systemd.

          2. Any timeline on when we will see the tech preview or the stable version of KF5 based PIM on Fedora possibly on F23?

          • Indeed, you are right, same as they should not talk about PulseAudio or Systemd. I think we don’t even mention Akonadi anywhere in the UI…I guess people found out about it from top ;-)

            Jan Grulich is working on it, I think we already have 15.12 PIM in rawhide, but I don’t know when we are going to push it to stable Fedora. After all, if we do something wrong (on Fedora side I mean), reverting back to old KDE PIM would be basically impossible, so we better take the careful approach here.

  2. RJVB

    You didn’t mention anything about the implication of all this on applications that depend on kdepimlibs (or used to in their KDE4 version), like DigiKam. Care to develop?

    • I already explained this in some previous blog posts and other places. It is not possible to run KDE4 apps against KF5 Akonadi (and vice-versa). All applications that want to use Akonadi must be ported to KF5.

      There are some KDE4 apps that depend on kdepimlibs, but not on the Akonadi libraries. Those can easily work, as KDE4 and KF5 kdepimlibs are fully coinstallable. The only non-coinstallable part is the Akonadi Server (which we did on purpose), so Akonadi-enabled apps need to be ported to be able to use KF5 Akonadi.

  3. Hi,
    thank You for the work! I wonder when there will be usable version of KDE PIM (Brilliant spftware set!). I’m still sticking with KDE4 as I haven’t found KF5 fully matured. KDE PIM and digiKam are among my crucial apps which just must work. And will Akregator be ported to Akonadi? I also wonder about migration – I have big e-mail history locally…
    I’m looking forward new version!

    • Hi,

      we already consider KF5-based PIM to be production-ready, we already did two major releases of it (also note that the KDE4 version is no longer supported – no more bugfixes).

      We really want to port Akregator to Akonadi, there is even a branch with a full port and there is not much missing from being able to switch over. Unfortunatelly now the branch wasn’t touched in years and is still KDE4, so some effort would need to go to rebasing the branch to current master and then just doing the final sprint to get it rolling.

      No migration of emails or other Akonadi data is needed, KF5 Akonadi works on top of the same data as the KDE4 version.

  4. Hi,
    first of all i would like to say, that i love to use Kontact and also the Akonadi idea of a unified backend. I use it for my daily work and find the workflow and many small power user usability settings very attractive.
    THX 4 providing me with such a nice piece of free software.

    However on the other side, I cannot say that akonadi is nearly free of show stopper bugs. There are a few things that are quite annoying and some new show stoppers are introduced with each release. I did not feel, that the move to KF5 version of Kontact and Akonadi improved this situation (from a user perspective).

    E.g.
    – dav resource loosing setting regularly: https://bugs.kde.org/show_bug.cgi?id=341998
    -> this also causes per contact settings to fail (e.g. encryption settings, html settings, etc)
    – dav resource is unable to delete entries since KF5 and falls into unrecoverable error state after doing so. Causing new entries to be never synchronized with the server. This happens silently when using Kontact and therefore, I am regularly loose calendar entries.: https://bugs.kde.org/show_bug.cgi?id=355441
    – Categories get multiplied (i.e. several entries of the same category) since KF5: https://bugs.kde.org/show_bug.cgi?id=357236
    – imap resource regularly crashes since 15.12.: https://bugs.kde.org/show_bug.cgi?id=357528

    This are all bugs i am facing just now on a single machine. I am aware, that some issues are not core Akonadi, but mainly plugins. However, this are plugins used heavily by many people.
    => To conclude: There is still a lot of work to do, to get a non-beta state version of KDE pim stack.

  5. lhugs

    For quite some time, KMail2 has felt like a stable, fast and powerful mail client for my needs (about 8000 mails in three IMAP accounts). For the size of your team, your doing quite a good job, I think!

    PS: It’s annoying I have to re-type this message because I messed up the captcha ;)

  6. Shaheed Haque

    Please please please if you use binary protocols make sure that you have (a) taken endianness into account and (b) added a version number so that protocol/syntax changes can be handled. We are storing imporntant data and we really have to make sure those basica precautions are in place.

    • Endianness should not be a problem as the server and clients are always running on the same machine, so we don’t need to bother with conversion. Protocol is of course versioned, but just for internal consistency, since we now guarantee that users will always have the same version of server and client libraries (because we ship them together now).

  7. Bugzy

    Three quick questions:
    –Since you are not planning on developing KJots very actively, do you have any thoughts about talking to the folks working on Renku [1]? I am currently assuming that KJots and Renku will be using and accessing the data.
    –Is there any in for on what this major change means for syncevolution + Akonadi [2][3]?
    –(off topic but related) does “Laurent…now seems to be messing around Akregator” include any plans for syncing with ownCloud News type of stuff?

    [1] https://zanshin.kde.org/2016/01/01/zanshin-0.3.0/
    [2] https://syncevolution.org/wiki/kde-akonadi
    [3] http://downloads.syncevolution.org/syncevolution/kde/i386-rpm/

    • Re Renku: I just only learned about Renku. Once a KF5 version of Renku is released, we can send KJots away for good (unless someone decides to maintain it).

      Re syncevolution – as far as I know syncevo uses the KDE Akonadi client libraries, so they just need port it to use the new KF5-based libraries (the API did not change very much, so it’s mostly a build-system change).

      Re Akregator: I don’t think so, sorry. Hopefully the Akonadi port of Akregator gets finished eventually and it will be easier for everyone to add support for new services to Akregator.

  8. Foo

    Very often, KMail just stops syncing mails for no reasons and I have to do “akonadictl restart” to get it to work again, this makes me very sad. Is this something that I can hope to see fixed? Would it help if I deleted all the akonadi settings, or switched database, or performed some pagan ritual?

  9. markc

    I have a core i7 laptop with 16Gbs of ram and I can’t use Akonadi because it uses too much ram and I already have MySQL (100Mb+ ram) running for other purposes. I could tune akonadi to use the same system MySQL or try the SQLite backend (rarely seems to work) but all the various akonadi processes + MySQL take up to 2Gb of RES (real) ram on my system. If Kmail worked really well and handled all of my accounts (more than a dozen with 20k messages) then I could sacrifice the ram and resources but way too many times I find Kmail is just stuck and not responding to new messages and I have to restart akonadi itself. Hopefully this is all to do with KDE4 PIM and your post is outlining a glorious new future for akonadi. I hope so but in the mean time I simply have to use Thunderbird for mail and completely remove everything akonadi related from my Kubuntu system.

    The main benefit for me of using Thunderbird is that when my machine starts swapping I can simply quit and restart TB and free up a lot of ram and I do not have to worry about yet another MySQL daemon sucking resources. I love KDE and miss Akregator but I need to get on with work and thank goodness for Thunderbird and it’s limited RSS news reading capabilities.

    Please oh please make sure the SQLite backend works as well as possible so I don’t need to waste 100Mb+ on an otherwise useless MySQL database when this new KF5 akonadi goes mainstream.

    • That’s weird :( I’m currently at 300MB RES with Akonadi (server + resources) and another 300 MB RES takes PostgreSQL. I’m running KMail with some 300 000 emails and it performs rather well. Would be great if you could give it another shot with KF5-based PIM when you have a chance.

      Oh, and it’s also possible to configure Akonadi to use an already existing instance of MySQL, so you don’t have to run two servers.

      SQLite has its quirks, I don’t know what version you were trying last, but we did some minor improvements regarding concurrency in our fork of the Qt SQLite driver at one point, so maybe that could solve some of the issues for you. The biggest problem we have with SQLite is really the concurrent read/write access from multiple threads, which SQLite does not always handle well :(

        • I don’t see a point of that paragraph at all. So he disabled Akonadi and saved 250MB of RAM. Congratulations, you saved enough memory to open two more tabs in Firefox. And the comparision of akonadi_birthday_resource to pulseaudio is just ridiculous. If personally find it just amateur and embarrassing that “KWrite” takes more RAM than NetworkManager. Also my akonadi_birthday_resource is 2MB, while pulseaudio daemon is 5MB. So much memory wasted here! So either one of us is wrong, or the memory consumption is just a completely silly metrics to begin with. Maybe Ken’s resource just finished syncing and not all memory was reclaimed yet. Maybe Ken’s resource was JUST syncing, which took more memory. Or he has lots of contacts with birthdays, and they are cached somewhere temporarily.

      • Damian

        Maybe you could compare your result with Thunderbird with some 300 000 emails?

        One more problem – Akonadi does not well scale down.

        What with people with 1-2 email accounts, and 10-20k emails? Compare resource consumption with other solutions.

        “we already consider KF5-based PIM to be production-ready”

        Yesterday I installed PIM 5.12.1 with KMail 5.1.1. I set up one gmail imap account. Synchronization stooped on 28%…

        • Yep, scaling down is a bit of a problem, in that case switching to SQLite is a possible solution.

          Sorry to hear you had problems with the sync. Have you tried it again? I’m not sure what could go wrong.

          • DAMIAN

            Yes. I tried. Without luck. Removing account, add it again caused double accounts in kmail and spining disk from akonadi.

            Previously I tried this a month ago, and also, if I remember correctly, akonadi not done full synchronization.

            Akonadi in my distro using default SQLite for backend.

            All akonadi backends have some sort of problems.

            I’m must wait for akonadi-next. Please only not years.

  10. Solerman Kaplon

    If mysql sucks for akonadi, maybe there should be a ready-made setup in kde to use postgress instead

    • PostgreSQL has one major issue: one has to manually run pg_upgrade after major database update. We can’t really expect users to go and fiddle with command line after they updated their database, many won’t even know what’s going on.

      There’s a TODO in Phabricator to see if we can somehow detect that we need to run the tool and run it automatically when Akonadi is started, but that’s of low priority atm.

      All in all, I’ve been running with PostgreSQL for over nearly two years now and I’m very happy with the performance.

  11. Marc

    This will be a rather long comment, so sorry. I’ve casually arrived to your post and felt I had to reply, although it’s not only about Akonadi.

    First of all, and for the record, I’m (was) a long time KDE user, since v1.1 (redhat 6.x etc…). That’s around 15 years. The only thing I struggled with at some point in every major version has been kdepim, specially kmail. And when you try to use it “in production”, for actual business your income depends on, stability and that things just work are important. And please don’t miss this: I understand this is free software, you have absolutely no obligations towards the users and that if I don’t like something or feel it can be improved I can try to help coding or stop using it.

    So, in some of this post’s comments you say things like “I think most of the complaints are just because people like to complain about stuff”, “The software has improved over the time, but users did not and they always find something to complain about” and “Some of it also comes from lack of information or insight into the technology”. My answer to this, as a long time developer myself, is pretty clear: if it works flawlessly, no one complains. Seriously, not a single person. Maybe someone will say “I’d like this in another color” or even “It’s ugly”, but they won’t *complain* about how it works.

    After all this years with KDE, I’ve loved it, specially Konqueror and it’s flexibility (KIO slaves are probably the best idea KDE as a project had). But 2 months ago I decided to stop using it and switch to another DE. The main reason? kdepim. The other less important reason is the abandonement of Konqueror which I still was using.

    The transition from KDE3 to KDE4 (I waited until 4.4 if I remember right) and the failure that was version 4.7 update for Kmail, with that never-working migration tool that made a lot of people lose their emails, were the first things that made me lose confidence in Kmail and Kontact in general. I lost a local folder that contained more than 1.000 (not-that-important-hence-not-backed-up) archived messages because it somehow got corrupted by the migration tool. Some early KDE4 nasty bugs were really annoying too and made working a tad less enjoyable but hey, I loved KDE and was my DE of choice, I knew it would get fixed and everything would be ok. It eventually happened and the last KDE4 versions were fairly good, I was even expecting stability and feature parity with KDE3 sooner than later. Then KDE5 was announced and I asked myself “Again?”.

    I waited until 5.2 and, after reading about version 4 going unsupported and some research on stability and other people’s updates successes/failures, I went for it. Visually great and smoother, faster and more reponsive than KDE4. I was happy. After using it for a while though, I started getting memory leaks from the shell. Then, I got hit by this bug which is very annoying. I could live with all that, but in a 4GB RAM system where you want to do work using an IDE, it makes your life really not fun.

    Then I started Kontact, and quickly got hit by this bug (a 2 and a half years old bug is marked as duplicate), that basically doesn’t allow you to delete messages from IMAP accounts when using a Courier IMAP server. Then I tried to write a message and got hit by this other one, making it impossible to compose new messages. Apart from this, I also noticed a large spike in RAM usage. I checked and the akonadi imap processes now used 20MB of RAM each, when in KDE4 they were using arround 5MB (same with dispatcher/notifier). I have 12 accounts so (in a 4GB RAM system) akonadi went from 140MB RAM usage to 380MB (imap accounts + mail dispatcher/newmailnotifier + mysql instance [70mb to 100mb]).

    So I had to start using Thunderbird. Thunderbird uses exactly 110MB of RAM, including all the IMAP accounts and a tad load of RSS feeds that I migrated from Akregator. No need for an external database, all IMAP accounts work flawlessly and I can actually compose new messages. It’s not integrated into KDE and is definitely uglier than Kmail but hey, it works and won’t break after an update I can’t avoid.

    After some weeks, seeing the bugs affecting me weren’t being resolved and considering I use a non-KDE IDE for work and of the two KDE programs I mainly used, Kontact (Kmail + Akonadi specifically) and Konqueror, one is a resource hog that doesn’t even work and the other has been abandoned, I just decided to, after 15 years, ditch KDE.

    I hope I managed to not complain, and just explain how and why KDE lost a long time user.

    And as you can see in other comments, I might not be the last one. Also, if you look at the bugs I linked through the text, despite some being if not critical, at least major in my opinion, there’s absolutely no sign of work nor even interest in them. Again, I understand it’s free (as in freedom) software and what it means.

    To end this wall of text, I’d like to add to one of your statements: “I think most of the complaints are just because people like to complain about stuff they care about“.

  12. Ferry

    I have come a long way myself, moving from windows to linux to be able to use Evolution, then moving to Kontact as Evolution had at a point to many bugs to be usable and lacked proper support for carddav. I know that thunderbird works well under windows and linux, but what I really used a lot in Evo at the time was automatically adding contacts to our LDAP server when replying e-mails and the virtual folders to organize all the incoming mail. Until that broke.

    In Kontact I found things to be full featured but slow and not very reliable. In my opinion, using several servers to organize the data stores is more reliable in theory, but only when implemented well. When everything is in one process, you can be sure (in each piece of code) that all the other code is running and has not died. In client/server architectures either can die, and so needs implementation as if each server is on a different machine at opposite ends of the world connected over the Internet.

    I don’t really believe converting data to text and back is causing the slowness, but round trips to the database server might. But as a relatively big user, I find most of the time things work fast enough, until something (bug) goes wrong. Restarting Akonadi mostly resolves the problem, sometimes logging off and in will. That’s relatively hard compared to just killing Thunderbird and starting it again.

    What I feel is the client/server architecture is a good thing, but we are missing the payoff:
    – imap is already client/server
    – carddav and caldav too (in our case using Owncloud)
    – however each Akonadi user/desktop keep its own database, cache, indexes, and still when you search for an old e-mail, you will never find it (in our case, we have a shared imap with 20000+ messages from over the last 10 years, causing 20GB cache each user/desktop, and 1000+ contacts in Ownclouds Carddav, this is with only 4 persons).

    Of course the imap/carddav/caldav servers are lacking features (like indexing) which Akonadi adds. If you want to be replacing MS Exchange, I believe the data store should be centralized and so should be the indexing etc.

    From this POV, Akonadi should be running on the central server where the sql server resides and the imap/caldavcardav servers too.

    That way, the Kontact client would be light weight, would not need to keep local caches and would find searched e-mail by querying the remote server. Actually Akonadi could then skip storing the e-mails in a imap server and act as imap server itself to non Kontact clients (like Thunderbird), a lot like as it has done up to KDE4. This all sounds like a small step from where we are now, in reality I fear quite a lot needs to be done.

    • The problem with single-process architecture is that when a part of the code crashes, your entire application crashes. If you support 3rd party implementations (like we do), it becomes even more dangerous. And of course if you have a local cache or data store, the last thing you want is data corruption in case of some random part of the code crashes. By isolating those parts into individual processes, the functionality of the entire thing is not affected even when one of the resources crashes. Especially the server, which actually works with the database and local storage, is not endangered.

      You would be surprised how much overhead the (de)serialization was :) Yes, database queries are our problem too, we are not always 100% effective there, but as I wrote in the blog, we are trying to improve that too.

      One of the main points of Akonadi is to give you access to all your data even when you are offline. Yes, IMAP, CalDAV etc. are server-client architecture already, but that’s useless if you can’t connect to the server :) That’s why you have local Akonadi server that you can always talk to, and then we do the magic of talking to the actual server behind your back (when you are online).

      If you don’t benefit offline access (running Akonadi on workstation that is connected to the internet 24/7), you could turn off the “Download messages for offline use” in your IMAP resource, so that Akonadi only stores local metadata, and fetches the bodies on demand (when you open the emails). You can even set cache expiration for each folder (or just the top-folder from which all folders inherit), and Akonadi will discard the cached bodies after the set period, reducing the amount of data stored locally. Of course that breaks that search (we can’t index what’s not locally available)….this only works for email atm., where we store the headers and the body separately, for contacts or events we always store the entire body, so we can’t really expire them.

  13. Dude, you and the other Akonadimals, respect that you keep going strong. Really.

    I’ll wait until it is available for Tumbleweed before I move to KMail5 but I do look forward to it.

    In general, though, I most look forward to Akonadi Next because I believe it’s a far saner design ;-)

    Keep up the good work and – I’d appreciate a status update on Akonadi Next!

  14. Javier

    Theres is a note taking app already: Knotes. It’s pretty spartan -doesn’t allow a folder (book) structure for classifying the individual notes, like KJots did- and its widget is ugly as a sack full of witches, but it works, works very well. Besides, we have that Zanshin thing, which I haven’t tried but claims to be an apt app for note taking; and the there’s Calligra Braindump as well, an app nobody knows what serves for, but that also claims to be a valid note taking application; and finally there are blogs and comments saying since several months ago, that Basket is also preparing a rewrite for KF5.
    Well, it would be awesome to have such a great and diverse offer if they were finished, but when still there’s no complete note taking app -image/audio/video/freehand drawing embedding is basic nowadays- comparable to Evernote, OneNote & al. couldn’t you guys talk to each other to see if it would be possible not to duplicate efforts in such a scarce manpower ambient? Excuse me if I’m being naive due to my ignorance about coding software, believe me, I say all these thngs with the utmost respect and gratitude for your work and efforts, but do, Kjots, Knotes, Basket, etc, developers really live so isolated from other fellow KDE and FLOSS developers that don’t know that other people are working to make apps that serve for the same, or don’t they even consider the possibility of joining projects and make finished and complete software at last?

    I know issues like this have been asked many times, but for incredible it may seem, developers never seem to give an answer really convincing. Again, excuse muy ignorance, but is there any kind of “state of the art” prospection team in KDE’s “bureaucracy”? A couple of persons who look for similar projects, get info about how advanced are each projects and talk to the different teams to propose them to work together in a bigger, better, and, over all things, more likely to be finished nd well mantained projects?

    And let me finish with a suggestion: Akonadi takes a considerable amount of RAM, some few hundreds of MBs, nothing untolerable for my modest 3 GB laptop but still a nice amount. But the certain thing is that I don’t really need Akonadi running all the time. I don’t chat too much except via Whatsapp, which I use on the desktop via the web app on a web browser, nor I check/write mail more than 3 or 4 times a day. So:
    – Could you, guys, write a visible and quick to access “switch” to turn off Akonadi instead having to Alt + F2 to launch the Akonadi server control tool to stop it and then closing said tool?
    – Could also write a simple tool to disable Akonadi’s autostart, since many users don’t need to read our mail or news inmediately every time we boot our computers. Said config tool could be in System Settings control app.
    – Could you add an option somewhere to set a time in minutes for Akonadi to automatically stop after no Akonadi dependent app is running, so, if we open Kmail, use it for a while, and close it, 5, 10, 15, whatever, minutes after Akonadi server exits by itself.

    I’m sure is possible to tweak some config file to do this; almost everything is possible in Linux/KDE/FLOSS, and that’s why we love it, but I’m talking about easy to see and access tools, something even my granny could learn to do in 2 minutes.

    Cheers, and, again, thanks for your great work.

    • As you said, KNotes is not a vaiable replacement for KJots. Zanshin (or Renko) will hopefully provide full-features replacement for KJots eventually.

      Akonadi is auto-started when it’s needed, i.e. first Akonadi-enabled application that is started will start Akonadi. At some point I’d like to see auto-shutdown and on-demand start of individual resources, so that we are indeed only running the components that are currently needed.

      • Javier

        «Akonadi is auto-started when it’s needed»

        Oh, weird; every time I start session on Plasma Akonadi is started. I’ll check my settings, perhaps I have some Akonadi related app autostarting by default. Thanks for clarificating.

  15. Tino Pintonato

    I am really upset to know and see, last update of arch was blue, that kjots is pulled out.
    Now I want to rebuild by myself the application.
    But : where to take the latest source ?
    I am using kdevelop

  16. Ivan Mincik

    Dear Daniel,
    I am very happy to read this post. I was KDE user since 2.0 and I was forced to move away from KDE after Kmail migration to Akonadi. I really hope your recent work will make things much better.
    Thank you.

  17. Ivan Mincik

    I have one more question. Is Postgresql backend for Akonadi finally considered as production ready?

    Thanks

    • Production ready in terms of stability and performance: yes – I’ve been using PSQL for couple years now myself.

      The only problem with PSQL is that it requires manual intervention from user after every major upgrade of the server binary to migrate the database files. I’d like to automate it eventually, so that Akonadi can do the migration on start if necessary, but that’s a low priority atm.

      • Ivan Mincik

        Thank you for info, I have no problem with manual PgSQL upgrade. I will give a with Kubuntu 16.04.

Comments are closed.