Unified Mailboxes in KMail

Standard

Today KMail has gained a new cool feature that has been repeatedly requested in the User survey last year as well as on forums and social networks: Unified mailboxes.

Unified mailboxes offer not only a unified inbox – a single “Inbox” folder showing emails from inboxes of all your accounts, it also provides unified sent and drafts folders by default. But we did not stop there: you can create completely custom unified mailboxes consisting of any folders you choose. You can even customize the default ones (for example exclude an Inbox from a particular account).

Some obligatory screenshots:

The feature will be present in the December release of KDE Applications.

 

Do you want to help us bring more cool features like this to Kontact?

Then take a look at some of the junior jobs that we have! They are simple mostly programming tasks that don’t require any knowledge of Akonadi or all the complexities of Kontact. Feel free to pick any task from the list and reach out to us! We’ll be happy to guide you. Read more here…

KDE PIM Junior Jobs are opened!

Kontact Icon
Standard

Do you want to help us improve your favorite PIM suite but you were always scared by its size and complexity? Well, fear no more! We have collected a bunch of simple and isolated tasks in various parts of the PIM suite that require none or just very basic understanding of how the entire Kontact and Akonadi machinery works. We have documented them and we are prepared to guide you and help you to accomplish the tasks. Those are small simple tasks, but they will make many users (and PIM developers) very very happy.

I’m in! What do I need to know?

Just some C++ and maybe a bit of QML if you want to go for the QML tasks. We don’t expect you to know anything about Akonadi or how the entire PIM thing works as most of the tasks are pretty self-contained (although you can read up on the basic concepts and architecture if you are interested).

Cool, what do you offer?

We have tasks to improve the look of KAddressbook’s contact list, contact view, and contact editor. If you prefer working on KOrganizer, you can help us to make the event view look more modern. We would also like to improve the Account Wizard experience by porting it to QML and improving Gmail/Google Calendar and Contacts integration. Of course, the key part of Kontact is KMail and even there we have a few places that can be improved: we would like to improve the IMAP quota warning and add support for Autocrypt. And finally, you can also make life easier for other KDE PIM developers by improving our debugging tool, the Akonadi Console: we want to be able to save the output into JSON and load it again, alphabetically sort some of the lists, make working with the DB console a bit more comfortable and be able to restart Akonadi agents whenever we want to.

There’s also a very cool effort ongoing to allow integration between Kontact and MyCroft, the opensource voice assistant. For this, we need help improving a command line tool that’s used as a bridge between MyCroft and Kontact.

If you don’t know any programming but you would still like to help, we have some non-programming tasks as well! Sure! We are working on a new website for Kontact and we could use help with both design and writing content for it! We also need help improving our user documentation, cleaning and updating our wikis on community.kde.org and userbase.kde.org or cleaning up our bug tracker. If you want to help with any of that, get in touch with us on the kde-pim mailing list!

You can find the full list of junior jobs on Phabricator.

Haven’t found anything interesting? Don’t worry, we will keep adding more over the time, so just check the list every now and then. Or do you have your own idea how to improve KDE PIM and you just don’t know where to begin? Get in touch with us and we will help you!

Now how do I get started?

1) Get in touch with us 

To make sure several people won’t try to solve the same thing, it is the best to get in touch with the PIM community first so we can look at the single topics in more details. Some of the descriptions of the tasks are intentionally a bit vague as there are multiple ways how to approach or solve the problem. It’s always better to talk about the options first so that no time is wasted on approaches that won’t work.

2) Get your development environment set up

The KDE PIM community wiki contains articles on how to develop KDE PIM inside a Docker container. Alternatively, as most of the changes are pretty isolated, you should be able to compile just a single component from source against your distribution packages (you will just need to install some development packages first).

3) Pick a task

Pick one of the tasks linked above, or just look at all the junior job in Phabricator. They span different topics, different components and are of different complexity and size. If you find a particular task that you would like to work on, assign it to yourself and get working! If someone else already has the task assigned, you can ask if they maybe want some help, or just look for another task.

4) Get to work!

Fire up your favorite IDE and start working! If you need any help with the task – from finding the right repository and code, through getting the program compiled to being stuck on a bug or something not working – just ask us! You can ask in the Phabricator task or send an email to the kde-pim mailing list and some of the PIM devs will help you.

Also, don’t feel limited by the description of the tasks – feel free to do only part of the task, or do even more than what’s in the task description. If you think you have a better idea how to solve something, let us know in the Phabricator task.

I have the code, what’s next?

Awesome! Now it’s time to upload the code for review. You can use the arcanist command line tool, or you can just generate a diff and upload it manually via the web interface. Don’t worry if you don’t know whom to assign for review, Phabricator sends the notification the entire PIM team automatically.

Plasma Pass

Standard

You may have heard about pass, the standard Unix password manager. I learned about it from Milian Wolf some months ago and I really liked it for its simplicity, respect for privacy and multiplatform support. And so over the past months, I started to slowly change my passwords to randomly generated ones stored in pass.

To get a password from pass, you simply type pass -c SomePath/SomeService into console and pass will copy the password straight to your clipboard. Super simple. Slightly less comfortable when you are dealing with websites though. Luckily there’s a wonderful browser extension called browserpass that can fill online login forms with a single click and has automatic password matching based on the current domain.

But sometimes even I am simply too lazy to open Yakuake and type in a command, so I started looking for some GUI. There’s qtpass, but that’s not exactly what I was looking for. And so I dusted off my QML knowledge and wrote Plasma Pass: a systray Plasma applet to quickly find your password and copy it into the clipboard with a single mouse click. The applet also takes care of removing the password from the X11 clipboard as well as Klipper after 45 seconds so it won’t leak accidentally through your clipboard history.

The source code is currently available in my scratch repo: https://cgit.kde.org/scratch/dvratil/plasma-pass.git/

And now back to fixing Akonadi ;-)

My KDE PIM Update

Standard

This blog post is long overdue, but now that I’m back home from the KDE PIM Sprint in Toulouse, which took place last weekend, there’s some more news to report.

Akonadi Improvements

On the sprint, I finally finished and merged a new improvement in Akonadi called Notification Payloads. I will not go into the technical details here, the most important thing is that this new improvement will notably reduce the CPU and disk load in Akonadi, especially during intensive operations like email sync. It should also help with the long-standing issue regarding errors and email duplication when using POP3 and local mail filters. Finally, this new feature opens doors to further improvements and optimizations like server-side change recording (technicalities here) and ultimately being able to
shut down Akonadi Resources when they are not needed and start them on-demand, thus saving some more resources.

As I was touching the internal notification system in Akonadi I also improved the relevant debugging tools in Akonadi Console, our developer and debugging tool for Akonadi. Based on input from Sandro I also added Logs view. Thanks to that it’s now possible to see debug output from all running Akonadi applications straight in the Akonadi Console without the need to restart Akonadi or the application from the terminal to see the debug output. This will make it easier for users to provide us with relevant information to help us debug and solve their Akonadi issues.

Kontact Improvements

This was just a minor change, but it finally solved my long-standing issue with Kontact and Breeze: the side-pane icons to choose between different Kontact modules were colorful – the only non-monochromatic part of Kontact which was so obviously not fitting into the rest of the UI. With a tiny change, the icons are now also monochromatic, making the Kontact window look more uniform.

 

Native Gmail authentication for IMAP and SMTP

For a while now the IMAP resource supports logging into Gmail accounts using the so-called OAuth method, where you provide your credentials into the Gmail login window which also supports two-factor authentication. The IMAP Resources was forcing the OAuth method with Gmail for everyone, but this requirement has now been relaxed. Although the IMAP resource will choose this method by default it’s possible now to also choose the traditional authentication methods like with any other email provider.

Secondly, the OAuth support has finally landed also into our SMTP module which is used for sending emails, so if you select this method in your Outgoing account configuration with Gmail, you no longer need to use “App-specific passwords” from Gmail.

Syndication Cleanup

The Syndication library is used to retrieve and parse RSS and ATOM feeds and is used among others by Akregator. We have now cleaned up the library and removed some redundant dependencies so that we will eventually be able to move it into KDE Frameworks so that even more applications can benefit from it.

Going to Windows

Thanks to a huge effort from Hannah we are now able to build Akonadi and other parts of the KDE PIM stack on Windows. While we are still a long way away from having Kontact properly running on Windows, we managed to get Akonadi to work on Windows with some other programs. Windows is a huge platform and Kontact with all its features and functionality could be a good competition to established PIM solutions there and a huge potential to grow our user and developer base. While we still focus primarily on Linux, we are slowly looking forward to extending our reach to Windows.

Bugfixes

A lot of them. Big thanks to David Faure who spent a big part of the weekend debugging his IMAP resource to figure out why it keeps getting stuck on occasions. He fixed several issues in the IMAP resource so that it properly reconnects after server connection is lost or times out (due to poor internet connectivity usually) and also found and fixed some issues in Akonadi syncing code.

Future

What’s next then? We will continue to work towards a stable release for Windows,
and hopefully soon finish the rewrite of the indexing and search infrastructure
in KDE PIM to make it faster, reliable and more useful again. There’s also a lot
of smaller tasks and improvements to look into during the year.

NextCloud, Multiple versions of PHP via SCL on CentOS 7

Standard

There’s nothing new in this blog post, it’s mostly a documentation for future myself once stuff breaks and I’ll have to figure out what I did to make it work :-)

This is post is about running NextCloud 10 on PHP 5.6 from Software Collections (SCL) on CentOS 7 (where PHP 5.4 is available by default) while keeping the rest of the web apps running on the system-wide PHP 5.4.

NextCloud 10 itself runs on PHP 5.4 just fine, but the News app requires at least PHP 5.5.  At the same time, I can’t switch over to PHP 5.6 completely since some other apps that I’m running depend on binary PHP plugins, which are not available in PHP 5.6 Software Collections. I’d have to build them myself, and Ithe SCL’s ‘m too lazy to maintain them later.

Step 1 – Install SCL and PHP 5.6

Follow the official instructions here to setup Software Collections on CentOS 7 and to install rh-php56 SCL: https://www.softwarecollections.org/en/scls/rhscl/rh-php56/

You will also need to install additional PHP 5.6 modules:

sudo dnf install rh-php56-php-fpm rh-php56-php-intl rh-php56-php-ldap rh-php56-php-pgsql

(install rh-php56-php-mysqlnd instead of pgsql if you use NextCloud with MySQL)

INTL, LDAP and PGSQL modules are needed by NextCloud, FPM is a FastCGI Processing Manager that we will use to execute NextCloud’s PHP scripts with PHP 5.6 instead of the default system-wide PHP

Step 2 – Configure PHP 5.6

The php.ini for SCL PHP 5.6 is located in /etc/opt/rh/rh-php56/:

Make sure to configure timezone:

date.timezone = "whatever your have in /etc/php.ini"

Setup include_path to fallback to PHP modules from system-wide PHP. This is probably not very kosher, but since the system modules are PHP 5.4, there should be no problems when using them with PHP 5.6:

include_path = .:/opt/rh/rh-php56/root/usr/share/pear:/opt/rh/rh-php56/root/usr/share/php:/usr/share/pear:/usr/share/php

Finally, update open_basedir to include all the paths and NextCloud:

open_basedir = .:/tmp:/usr/share/pear:/usr/share:/usr/share/nextcloud:/etc/nextcloud:/var/lib/nextcloud

You may also want to adjust error handling and log reporting to not use the PHP defaults.

Step 3 – Set up FPM

You can use the stock FPM configuration, which works just fine. You may want to configure logging in /etc/opt/rh/rh-php56/php-fpm.d/cloud.conf to make debugging problems easier.

Start the PHP 5.6 FPM:

scl enable rh-php56 -- systemctl start rh-php56-php-fpm
scl enable rh-php56 -- systemctl enable rh-php56-php-fpm

Step 4 – Configure Apache to forward to FPM

In your VirtualHost configuration for NextCloud, add the following to the <VirtualHost> section:

<FilesMatch \.php$>
  SetHandler "proxy:fcgi://127.0.0.1:9000"
</FilesMatch>

(Make sure mod_proxy and mod_proxy_fcgi are enabled and loaded)

This will let all .php files in the NextCloud virtual host to be executed by PHP 5.6 through the FPM.

Conclusion

That should be it, NextCloud will now run on PHP 5.6, while all remaining virtual hosts will continue using the system-wide PHP 5.4.

It is possible to redirect any other VirtualHosts to the FPM. It’s also possible to do the same for other PHP SCLs, it’s just necessary to configure FPM from each SCL to listen on a different port (FPM listens on port 9000 by default), and then run all of them in parallel, and just forward respective VirtualHosts to respective FPMs depending on which PHP version you want to use for each particular VirtualHost.

References

KMail User Survey Results, Part 1

Standard

Back in August, we ran a survey to get input from our users and get a better understanding of how they use KMail. First, let me start by thanking everyone who took their time to fill in the survey. We collected over 3000 responses which is much more than we expected. Thank you very much! We got some interesting numbers and data from the survey, which I’ll analyze later, but to my big surprise, the most interesting part was the comments that many of you left at the end of the survey. We got over 1000 comments which provided us with a consistent feedback from the userbase. In this and the next blog posts, I want to address the common themes, complaints, and remarks that appeared in the comments, address the concerns raised and present some action plans that we are going to take to address those.

What’s going to happen to KMail when Kube is released

Nothing. Kube is not going to replace KMail. Both projects will co-exist and be continuously developed and improved. Each project targets a slightly different audience – KMail focuses on power users with several accounts and high volume email traffic, while Kube focuses on less advanced users. Kube will not, and is not attempting to, provide many of the advanced features that KMail has, instead, it will help KDE to provide a friendly PIM solution for all users.

“Akonadi should die”

We hear this a lot: “Akonadi sucks”, “Akonadi should be replaced”, etc. While we understand the frustration of many users who are having huge troubles with Akonadi, we are committed to it and we are convinced it is the right way to go. Akonadi got a lot of bad reputation in the early days, but we worked very hard to fix the bugs and improve the performance over the years and we will continue to do so in the future. I took some more detailed notes, so I have an idea what people dislike about it the most and will try to focus towards solving that. There also seems to be a lot of misconception as to what Akonadi really does and it is often blamed for things it’s not directly responsible for. I might write a detailed blog post about that sometime in the future.

Some of you also mentioned Sink: Sink (formerly known as Akonadi-next, but no longer has anything to do with the current Akonadi) is the backend used by Kube. At this moment Sink is not mature enough and lacks the capabilities and features we need in order to be viable as a replacement for Akonadi. I personally like the concept and some of the ideas behind  Sink and I will try to get some of them into Akonadi and I hope that in the future we will be able to co-operate with Sink more closely.

Search is broken/useless

I was honestly surprised with the amount of complaints about the search feature of KMail – generally it boils down to searching not being reliable and returning wrong or no results. I took a good look into our indexing and searchimg code and indeed found numerous issues with the way we index data and query them. I already created a plan how to solve this and make search fast, reliable and giving more precise results. The work on this is already under way, the fixes to actual indexing and searching code will be available in the December release, with more high-level fixes (indexing speed, better search UI, result presentation etc.) coming next year.

Account management

This issue has been on our radar for some time now and it’s good to know that it’s something you want us to target and fix as well. Adding a new email account to KMail is super hard, you have to basically do three things – create the incoming email account, create an outgoing email account, create the identity, and then you need to re-visit each of the of the accounts to configure them. We have the Account Wizard (Settings -> Add account…, or Tools -> Account Wizard in older KMail versions) which simplifies adding new accounts, but it’s not fully there yet and adjusting a configuration later on is still a painful experience spread across three different dialogs and many mouse clicks.

I took a look at what Thunderbird and Evolution do, and I also looked at K9, an opensource email client for Android, which is very close to KMail in the terms of account configurability (different identities per account, different outgoing accounts per identity etc.) and how their UI works. All in all, we have a lot of inspiration and examples to follow and build on, and we will work with the KDE VDG and our UX experts to overhaul the account management in KMail, making it much smoother and nicer experience.

Feature Discoverability

KMail has myriads of features, but many of them are well hidden from our users, almost as if we did not want anyone to use them. I’ve often seen comments like “It would be nice if KMail could do X” or “Until this survey, I did not even know KMail could do Y”. So, we need to fix that too. There are several ways how to do that, I think the best one is a combination of all: we need to improve our UI and make features easily discoverable and ready at hand so that they are easy to use. Secondly, we need to bring our documentation up to date. Couple years ago Scarlett did a great job at updating our documentation, but there’s still so much that is not covered. Finally, we need a good and easily accessible feature overview, ideally on a web page or wiki. We would welcome any help with writing and updating the documentation – this does not require any programming skills, just some free time and will to help :-) Get in touch with us on the kde-pim@kde.org mailing list!

 

More coming soon in part 2…

Randa Report Part 2

Standard

Let me start by annoying you with some pictures:

<3 Randa

The misty mountains below which the coders dwell.

We totally had snow in September. Well, not in Randa, but in the valley next to it, but still. SNOW!

And now for the serious part: in my last blog post, I talked about achieving our main goal for this year’s Randa meetings – we successfully ported the entire Kontact away from the obsoleted KDateTime class. Since we finished this on Thursday, there was still enough time left to start working on something new and exciting.

Volker and Frederik went on to work on a KWin plugin to simulate various kinds of color blindness which will help developers to see how visually impaired users see their software, I did a bit more code clean-up after the porting and a bit of code-review.

On Friday morning Volker and I discussed search in KDE PIM. Broken and unreliable search was one of the most often mentioned issues in the KMail User Survey, which we ran for the past month and a half, so I decided to tackle the problem and make our indexing and searching infrastructure fast and reliable.

The final plan consists of several phases – starting with reorganizing our current code to put related pieces of code (like email indexing and querying code) into a single place to make the code easier to maintain. This phase is already progressing and I hope to finish it within the next week. The second phase will involve moving the code that is responsible for indexing data into the backend Resources – whenever a backend retrieves an email from an IMAP server or an event from Google Calendar it will also index it and will send the index data alongside the raw data to Akonadi. Akonadi will then take care for just writing the data into the Xapian database. This will speed up indexing, reduce the IO load and will ensure that all data are reliably indexed and stored before they are presented in Kontact. The third phase will involve changing Kontact to query the index database directly, instead of asking Akonadi to do it for us. This will speed up the search and provide results faster. The final phase will focus on which data we are actually indexing. As they say – less is sometimes more – so having fewer, but better-defined data will allow us to provide better and more exact search results to the user.

Once this is settled, we can make applications to depend on the search index – for example KOrganizer will be able to query the index to only get events from e.g. December 2017 instead of fetching all events from the calendar and then figuring out if they should be displayed or not, making calendars of even the busiest people to load instantaneously.

All in all, it was an extremely productive hackfest for the PIM team and I’d again like to thank Mario, Christian and the rest of the Randa team for organizing the hackfest. You guys rock!

And remember, you can still donate to the Randa fundraiser to make future productive sprints possible!

Randa Report: The Fall of KDateTime

Standard

The main goal for me and Volker for this year Randa Meeting was to port KCalCore, our calendaring library, away from KDateTime. KDateTime/KTimeZone are classes for handling date, time and time zones which were used back in the KDE4 (and earlier) days when Qt’s QDateTime wasn’t good enough for us, mainly due to missing time zone support.

In Qt5 QDateTime finally gained all the necessary features we need which made it possible for us to leave KDateTime behind. But we couldn’t just go and replace “K” with “Q” everywhere – there are many subtle (and not-so-subtle) differences in API, behavior and some missing features of QDateTime that required us to carefully consider each step. The fact that we started working on this over 2 years ago shows how challenging task this was.

We did not expect to finish the port here, but once we dived into it, things went fairly well and I’m glad to say that after four days of intensive hacking, surrounded by Swiss Alps and mountains of chocolate, we succeeded. KCalCore is now free of KDateTime and KTimeZone and that in turn made (after some minor adjustments) the rest of KDE PIM free of KDELibs4Support. That’s a huge milestone for us.

Many thanks to John Layt for laying the initial ground for the porting and to Mario and Christian for the steady stream of chocolate and sweets to soothe our nerves :-)

If you want to help us to continue improving Kontact and other parts of KDE, please donate to the Randa fundraiser so that we can continue organizing similar productive sprints in the future. Thank you!

Developing KDE PIM with Docker

Standard

Getting started with contributing to KDE PIM can be hard – we have nearly 60 repositories with complicated dependencies – just getting that right can discourage many people from even trying. And then there’s, of course, the risk factor of running development build alongside your production Kontact, endangering your precious emails.

To address all these issues I have created a Docker image. It’s based on the KDE Neon Developer edition and it has all the dependencies pre-installed and pre-configured and comes with a set of handy shell scripts to make your life easier. It also has the environment set up properly so that you can run the development build of Kontact inside of the container – completely isolated from your production installation.

Interested now? Follow the instructions how to build the Docker image and how to run the container on our KDE PIM Docker wiki page.

The section regarding GPU drivers is still incomplete, if you have any knowledge regarding running OpenGL-enabled applications inside a Docker container with Nouveau or AMD/ATI drivers, let us know, please!

You can probably use our Docker image to develop other KDE applications in there as well, or you can take a look at Jos’ blog about Developing KDE with Docker and create your own Docker image to work on your favorite KDE application.

Happy hacking!

KDE PIM in Randa 2017

Standard

Randa Meetings is an annual meeting of KDE developers in a small village in Swiss Alps. The Randa Meetings is the most productive event I ever attended (since there’s nothing much else to do but hack from morning until night and eat Mario’s chocolate :-)) and it’s very focused – this year main topic is making KDE more accessible.

Several KDE PIM developers will be present as well – and while we will certainly want to hear other’s input regarding accessibility of Kontact, our main goal in Randa will be to port away from KDateTime (the KDE4 way of handling date and time in software) to QDateTime (the Qt way of handling date and time). This does not sound very interesting, but it’s a very important step for us, as afterward, we will finally be free of all legacy KDE4 code. It is no simple task, but we are confident we can finish the port during the hackfest. If everything goes smoothly, we might even have time for some more cool improvements and fixes in Kontact ;-)

I will also close the KMail User Survey right before the Randa meetings so that we can go over the results and analyze them. So, if you haven’t answered the KMail User Survey yet, please do so now and help spread the word! There are still 3 more weeks left to collect as many answers as possible. After Randa, I will be posting a series of blog posts regarding results of the survey.

And finally, please support the Randa Meetings by contributing to our fundraiser – the hackfest can only happen thanks to your support!

Konqi can't wait to go to Randa again!

You can read reports from my previous adventures in Randa Meetings in 2014 and 2015 here: