KMail User Survey Results, Part 1

Standard

Back in August, we ran a survey to get input from our users and get a better understanding of how they use KMail. First, let me start by thanking everyone who took their time to fill in the survey. We collected over 3000 responses which is much more than we expected. Thank you very much! We got some interesting numbers and data from the survey, which I’ll analyze later, but to my big surprise, the most interesting part was the comments that many of you left at the end of the survey. We got over 1000 comments which provided us with a consistent feedback from the userbase. In this and the next blog posts, I want to address the common themes, complaints, and remarks that appeared in the comments, address the concerns raised and present some action plans that we are going to take to address those.

What’s going to happen to KMail when Kube is released

Nothing. Kube is not going to replace KMail. Both projects will co-exist and be continuously developed and improved. Each project targets a slightly different audience – KMail focuses on power users with several accounts and high volume email traffic, while Kube focuses on less advanced users. Kube will not, and is not attempting to, provide many of the advanced features that KMail has, instead, it will help KDE to provide a friendly PIM solution for all users.

“Akonadi should die”

We hear this a lot: “Akonadi sucks”, “Akonadi should be replaced”, etc. While we understand the frustration of many users who are having huge troubles with Akonadi, we are committed to it and we are convinced it is the right way to go. Akonadi got a lot of bad reputation in the early days, but we worked very hard to fix the bugs and improve the performance over the years and we will continue to do so in the future. I took some more detailed notes, so I have an idea what people dislike about it the most and will try to focus towards solving that. There also seems to be a lot of misconception as to what Akonadi really does and it is often blamed for things it’s not directly responsible for. I might write a detailed blog post about that sometime in the future.

Some of you also mentioned Sink: Sink (formerly known as Akonadi-next, but no longer has anything to do with the current Akonadi) is the backend used by Kube. At this moment Sink is not mature enough and lacks the capabilities and features we need in order to be viable as a replacement for Akonadi. I personally like the concept and some of the ideas behind  Sink and I will try to get some of them into Akonadi and I hope that in the future we will be able to co-operate with Sink more closely.

Search is broken/useless

I was honestly surprised with the amount of complaints about the search feature of KMail – generally it boils down to searching not being reliable and returning wrong or no results. I took a good look into our indexing and searchimg code and indeed found numerous issues with the way we index data and query them. I already created a plan how to solve this and make search fast, reliable and giving more precise results. The work on this is already under way, the fixes to actual indexing and searching code will be available in the December release, with more high-level fixes (indexing speed, better search UI, result presentation etc.) coming next year.

Account management

This issue has been on our radar for some time now and it’s good to know that it’s something you want us to target and fix as well. Adding a new email account to KMail is super hard, you have to basically do three things – create the incoming email account, create an outgoing email account, create the identity, and then you need to re-visit each of the of the accounts to configure them. We have the Account Wizard (Settings -> Add account…, or Tools -> Account Wizard in older KMail versions) which simplifies adding new accounts, but it’s not fully there yet and adjusting a configuration later on is still a painful experience spread across three different dialogs and many mouse clicks.

I took a look at what Thunderbird and Evolution do, and I also looked at K9, an opensource email client for Android, which is very close to KMail in the terms of account configurability (different identities per account, different outgoing accounts per identity etc.) and how their UI works. All in all, we have a lot of inspiration and examples to follow and build on, and we will work with the KDE VDG and our UX experts to overhaul the account management in KMail, making it much smoother and nicer experience.

Feature Discoverability

KMail has myriads of features, but many of them are well hidden from our users, almost as if we did not want anyone to use them. I’ve often seen comments like “It would be nice if KMail could do X” or “Until this survey, I did not even know KMail could do Y”. So, we need to fix that too. There are several ways how to do that, I think the best one is a combination of all: we need to improve our UI and make features easily discoverable and ready at hand so that they are easy to use. Secondly, we need to bring our documentation up to date. Couple years ago Scarlett did a great job at updating our documentation, but there’s still so much that is not covered. Finally, we need a good and easily accessible feature overview, ideally on a web page or wiki. We would welcome any help with writing and updating the documentation – this does not require any programming skills, just some free time and will to help :-) Get in touch with us on the kde-pim@kde.org mailing list!

 

More coming soon in part 2…

KDE PIM on Akademy

Standard

Akademy is over and so here’s a short summary of what the PIMsters have worked on during the past week.

Wiki Cleanup

Me and Volker sat down and went through all KDE PIM wikipages on community.kde.org, userbase.kde.org and techbase.kde.org. Most of our wiki pages are horribly outdated, so we tried to clean them up, remove pages that are no longer relevant or useful. With fewer pages to take care of and better overview of what all content we have, we should be able to keep them more up-to-date than we did in the past years.

Developer Story

Contributing to KDE PIM is hard and we know that. Getting all the dependencies and environment set up correctly is not trivial, and you can’t run stable and development Kontact alongside each other easily.

We decided to address all those issues and make contributing to KDE PIM substantially easier. We are working on a Docker image that has all the dependencies and environment set up, so developers just need to run a single command to build entire KDE PIM. And thanks to the containerization, it’s also possible to use the development version of Kontact in parallel with the stable version.

We hope that having ready-made development environment it will be easier for new contributors to get involved with KDE PIM. We will post a more details once the Docker image is ready.

Kontact Homepage

Right now kontact.kde.org and kontact.org just redirect to our page on the Userbase wiki. We decided that we want a simple, but professionally-looking web site to market Kontact as an actual product so that it appears more attractive to new users, especially those who will be coming from Windows in the future and contains comprehensive information for both users and developers.

KMail User Survey

During QtCon 2016 we started working on KMail user survey to get a better idea of what our user base is like, how they use KMail and what their impressions of it are. And now the survey is finally live, so please go and fill it if you haven’t done so yet.

Wayland Support

Volker has finished porting Kontact to Wayland, so if you have Qt 5.9, you can now run Kontact natively on Wayland. Our main limitation was Wayland support in QWebEngine, which we use to render emails, but that has been resolved in Qt 5.9.

Merging Exchange Support

Krzysztof Nowicki has been working on Microsoft Exchange support for Kontact for a while now. We now have plans to merge his code into kdepim-runtime repository, so if everything goes right the Exchange support will be available out-of-the-box to all our users starting with the December release of KDE Applications.

Next Sprint

We will be meeting soon again in Randa. Our main plan for the sprint is to continue with removal of KDateTime from our code, and thus making KDE PIM free of kdelibs4support.

 

There’s some more that I did not mention here, you can check the full notes for details.

KMail: “Multiple Merge Candidates” error and how to fix it

Standard

If you can’t synchronize a folder in KMail and you are seeing “Multiple Merge Candidates” error after the synchronization fails, here’s a how to fix the folder to make it synchronize again – basically you force KMail to forget and re-sync the folder again.

 

  1. Open Akonadi Console.
  2. Go to the Browser tab.
  3. Right-click the broken folder and select “Clear Akonadi Cache” – this will remove all emails from the folder in Akonadi. This will NOT delete your emails on the server.
  4. Akonadi Console will freeze for a while, wait until it unfreezes (sorry, it’s just a developer tool, we don’t have a very good UX there :-)).
  5. Logout and login to make sure all PIM components are restarted.

 

After login start KMail (or Kontact) and hit “Check mail“. KMail will now re-download all emails from the previously broken folder. This may take a while depending on how large the folder is and how fast your internet connection is. After that the synchronization should work as expected.

 

In the upcoming KDE Applications 16.12.1 release Akonadi will have a fix that fixes the reason why the “Multiple Merge Candidates” error occurs, so hopefully in the future you should not see this error anymore.

KDE PIM Sprint Report

Standard

The KDE PIM Sprint is over (unfortunately…I could do this every day :-)), so now it’s time for some recap of what has been done. I’ll try to cover the Akonadi side, and leave the rest up to others to cover their projects ;-)

Hacking time! (Photo by Martin Klapetek)

Hacking time! (Photo by Martin Klapetek)

Akonadi Server optimizations

We finished and reviewed Volker’s old branch with a big optimization of the database schema. On my computer it reduces size of the file with the largest table by 30% and it speeds up all queries on that database, because the WHERE condition now has to perform only integer comparision, instead of string.

This however means, that we have to migrate user’s database on start. During the migration it is not  be possible to use any Akonadi-based applications.  We improved the code so that the migration takes about 10 minutes on my computer (used to take 20 and more). I personally think that it’s acceptable “downtime” for a one-time migration, so after I finish testing the migration code on other backends, I’ll merge the branch to master and we’ll ship it with KDE 4.13.

There's always time for a beer!

There’s always time for a beer!

Server-side Search

When using online IMAP, only headers are in Akonadi, the body is downloaded on-demand when the message is opened in KMail. This means that Nepomuk can’t index these emails and thus can’t include them in search results. To fix this case, we want to make use of IMAP’s SEARCH functionality. We simply ask Nepomuk to search it’s database of indexed emails, at the same time we send IMAP server the same search query and then we just merge results and show them to users. Most of the infrastructure in Akonadi Server has been in place for a long time now, so I’ll just undust it, adopt it to our current needs and we should be good to go ;-)

Using Akonadi to store tags

Working on tags!I already mentioned this in a previous report: we want to cache tags in the Akonadi database and write them to storage backends if they support it (for instance as additional flags to emails on IMAP server, as CATEGORIES into events in iCal, etc.). Thanks to it it will be possible to share tags between multiple computers, yay! We just need to modify the Nepomuk libraries, so that when you ask Nepomuk for all data tagged with “Holiday”, Nepomuk can search it’s own database and also query Akonadi. Another benefit will be that filtering emails in KMail by tags will be much much faster, because the relation will be stored locally in Akonadi and we won’t have to talk to Nepomuk, which is very slow (mostly because of Virtuoso).

Storing conversations and threads in Akonadi

In his comment under my last blog, Till Adam said:

[…] KMail has the second best threading in the world, I think, second only to mutt because that is faster. […]

Why can’t KMail just have the very best threading in the world? Because right now it has to fetch the entire folder from Akonadi in order to be able to perform Subject comparision when building threads. That’s both very slow and CPU-intensive operation. So we thought: let’s store information about relations between emails in Akonadi, and when KMail asks for content of a folder, we give back only first few conversations just to fill the screen, and then fetch remaining conversations on demand when user scrolls down. This should make opening even massive folders superfast and should save a lot of memory, too.

Akonadi and KDE Frameworks

Hopefully David's code is more stable than his towers

Hopefully David’s code is more stable than his towers :)

The most-awaited discussion of the entire sprint was about KDE PIM and KDE Frameworks. When should we start? What has to be done? What can we use this opportunity for? From Akonadi point of view I want to do several things: remove deprecated API, change some API so that we use consistent naming and separate UI and non-UI stuff. Volker Krause suggested that we could move the client libraries into Akonadi repository with Akonadi server, so thatwe could share some of the code (protocol parsers for example), which I like, so we’ll go for that, too.

A bit unrelated, but still: the Akonadi server already compiles with Qt 5 for a while, so possibly during this development cycle we might switch to supporting only Qt 5 (and making use of all the C++11 awesomeness). There’s a little library that kdepimlibs link against, so we just build both Qt 4 and Qt 5 versions of it. Akonadi depends only on QtCore and QtDBus, so we only need distros to ship qt5-qtbase, which we believe most of them do by now.

Gmail resource

Vishesh and Àlex (Photo by Lukáš Tinkl)

Vishesh and Àlex (Photo by Lukáš Tinkl)

I’ve been promising this for ages, now I finally discussed this with others, got some input and can start hacking :-) Let’s see if I can do something before Christmas ;-) Gmail resource would store all your mails in one folder and would create virtual folders for each label and just link emails from the “All mails” folder into respective labels. This way the emails will share flags (read/unread), and you will even be able to manage the labels by linking or unlinking emails from label folders in KMail.

Here I’d like to thank everyone for coming to Brno – if was a lot of fun and great pleasure to see all of you again, and also thank Red Hat for letting us use the office. Looking forward to see you all again on FOSDEM, next sprint, Akademy or anywhere else :-)

Email Threading in KMail: Your Help is Needed!

Standard

Hi KMail users,

we have a little favor to ask from you :-) On the KDE PIM Sprint we discussed how to improve email threading in KMail by using Akonadi to store the information, so that KMail does not have to compute it every time. This would make opening a folder almost instant, all threads would be reconstructed immediately and it would massively improve CPU and memory consumption (so it’s totally something worth helping us with ;-)) More details on what else we discussed and implemented will follow in another blog post tomorrow.

To implement the threading caching, we need to know, whether in these days it still makes sense to support threading by Subject. It’s used as a fallback when grouping by standardized email headers (In-Reply-To, References) are missing, which used to be a case with buggy email clients years ago, but hopefully is better now, so we could drop it, which would massively simplify the algorithms.

So we would like you to disable threading by Subject, observe how much it breaks your threading (and potentially your workflow) and report back to us. To disable it, go to View ->Message List ->Aggregation -> Configure. There go to Groups & Threading tab and in Threading combobox select Perfect and by Reference. If the combo boxes are disabled, you have to click Clone Aggregation to clone the default settings, and use the clone.

View->Message List->Aggregation->Configure...

View->Message List->Aggregation->Configure…

Aggregation Configuration

Aggregation Configuration

If removing threading by subject would break threading and workflow for too many users, we will keep the settings and we will try to figure out another way to do it.

So please configure your KMails, and let us know in comments below this post, on IRC, kde-pim mailing list or through any other communication means (just please try to avoid using smoke signals and pigeons ;-))

Thank you for your help!