cyrus imap and the amazing bdb library issue and broken lmtp on gentoo

I'm running Gentoo on my server. It's served me well, but every once in a while I really whack my head into some dumb stuff. This is the tale of how I tried to update my Cyrus IMAP server from version 2.3.13 to 2.3.14. I ended up running into a problem with the Berkeley DB versions and then something I can only describe as a broken lmtpd server. I ultimately persevered.

The story began a while back, when I first tried the update and found it failed miserably, dropping emails and generally not doing much useful. So I kept on with the version that was just a little outdated. Then I finally decided, I would just fix it. So I ran the emerge -u cyrus-imapd and it installed quickly. It also broke quickly. The errors I was seeing were related to different versions for Berkeley DB:

Oct 14 20:50:34 black imap[22844]: incorrect version of Berkeley db: compiled against 4.7.25, linked against 4.6.21
Oct 14 20:50:34 black imap[22844]: Fatal error: wrong db version

Clearly not what I wanted. Well, since it was broken I needed to fix it. I tracked the problem down to the ./configure script which was only looking at versions up to 4.6, then compiled against but linked to the newer version. I managed to triage this by unpacking the source and starting the compile to give me the configure command line

ebuild /usr/portage/net-mail/cyrus-imapd/cyrus-imapd-2.3.14.ebuild unpack
ebuild /usr/portage/net-mail/cyrus-imapd/cyrus-imapd-2.3.14.ebuild compile

Then I had the configure command line and added the --with-bdb=4.7

cd /var/tmp/portage/net-mail/cyrus-imapd-2.3.14/work/cyrus-imapd-2.3.14
./configure --prefix=/usr --build=i686-pc-linux-gnu --host=i686-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --enable-murder --enable-listext --enable-netscapehack --with-service-path=/usr/lib/cyrus --with-cyrus-user=cyrus --with-cyrus-group=mail --with-com_err=yes --without-perl --with-openssl --with-ucdsnmp --with-libwrap --enable-gssapi --enable-krb5afspts --disable-idled --disable-nntp --disable-replication --with-krb=/usr --with-krbdes=no --with-bdb=db-4.7
make install

Then those first errors went away, but email was still not being delivered.

Oct 15 08:03:44 black lmtpunix[4661]: accepted connection
Oct 15 08:03:44 black lmtpunix[4661]: lmtp connection preauth'd as postman
Oct 15 08:03:44 black lmtpunix[4661]: verify_user(user.matthias) failed: Mailbox does not exist

Well, I new that my mailbox existed, since the IMAP server was showing it to me. I ended up doing a lot of log trailing and testing. I finally decided it had to be the lmtpd service. I validated this by grabbing the latest version of cyrus-imapd (2.3.15). I compiled the new source using the above configure command and copied the new lmtpd in place of the installed lmtpd.

That fixed my delivery issue. So I created a custom ebuild file for 2.3.15 and now I'm using it.

Let the email flow.