Why did Gentoo sink into obscurity?
Someone asked this over in /r/linux. I try not to partake of that subreddit too much, but the question did intrigue me to some degree. I used Gentoo on my home computer, before and after 2005, and then on and off until a few years ago, and was somewhat active in a few local groups (but not in the larger Gentoo community). I'm no oracle, but there are a couple of thoughts that I hoped would be be worth sharing.
First, I think it's fair to point out that "sink into obscurity" is not quite the way to put it. Gentoo has a very large userbase and, more importantly, a great developer base. Some of its important members have retired in time, but that happens to any 15 year-old team, and a team that included people like Daniel Robbins and Tavis Ormandy certainly has a lot going for it.
Gentoo probably has more users than it had back in 2005, if only due to the growth of the number of Linux users in general. It's not at the forefront anymore, perhaps, but it's also true that the community of Linux users isn't the one in 2005 anymore, either. Many of its members don't really need what Gentoo has to offer. This is not a judgment on their skill or merit, it's a reflection of the greater diversity of needs that Linux users have -- and that Linux fulfills. People are using Linux for many other reasons than back in 2005, and many of them naturally have no interest in Gentoo. It's only natural, and not bad at all.
This does mean that Gentoo lost some of its edge, perhaps. But I think it's not only a reflection of the more diverse base of Linux users, it's also a reflection of the direction of the Linux space in general. Gentoo fills a very necessary space in the landscape of Linux distributions, but I do think that space is getting narrower and narrower.
Arch had little to do with it. One of the things I hear on this subject is that Gentoo's usage fell down because of Arch. There's some overlap between the two but, by and large, Arch gives you only two of Gentoo's advantages -- rolling packages and the chance of boasting about how you built your system from scratch. The most important features of Gentoo -- a great deal of choice in the base system and the ability to customize all installed packages through USE flags, both systemwide and per-package -- are not available in Arch, or in most other distributions.
I do remember when Arch made it to the spotlight and, indeed, it convinced a lot of former Gentoo users to switch. Many of these users disliked fiddling with USE flags and day-long compile cycles (2006-ish, remember?), and Arch offered shelter from that, while still giving users a light system (unlike the recently christened Fedora or the increasingly bloating Ubuntu) with recent package version (unlike Debian). It didn't offer features like fine-grained package customization (except, of course, for those built through AUR), but many Linux users don't need these features today.
They are also difficult to manage. In 2005, the web of dependencies was a lot thinner and the complexity of a Linux system was far more isolated and manageable by a community, rather than a team of paid developers employed by a large company. At the time, largely due to their (for my taste) better artwork and simplicity, I mostly used GTK applications, most of them part of Gnome, but did not use Gnome itself. They dragged a bunch of Gnome dependencies, most of them non-intrusive, and (owing to things like QtCurve and QGtkStyle) getting a desktop that looked uniform and nice was pretty easy.
This is no longer the case. My last attempt to get a coherent-ish Gentoo desktop system was a week-long uphill battle with bugs in systemd, udev and GTK that ended with a half-working system which looked like hell and somehow had three times as many packages, despite being used for pretty much the same things. The previous attempt, of getting a working system without systemd and as little Gnome as possible resulted in a monstrous, intricate set of USE flags and a system built by avoiding and working around more applications and technologies than I can think. I found myself using applications I hadn't used since I first started using Linux, like mutt.
I could, of course, just leave most defaults in their place and embrace all the latest innovations in the Linux landscape. But in that case, it seems a little silly to configure and compile everything myself. The fine folks who develop Arch, Fedora or Debian are already doing it a lot better than I could ever hope to do it. They possess more skill and a lot more patience to deal with these things than I do. Be kind to them; take it from the guy who has to look at systemd's source code virtually every day -- most of us, myself included, would quickly be driven to alcoholism and depression if we had to do it in our spare time, too.
What keeps users so loyal? There are still things that Gentoo does better than anything else. Cross-compiling and cross-platform development in general is one of them. I already do enough of that at work, so I try not to do it at home, but were it not for that, Gentoo would be my first choice. Their overlays system is exceptional. And, owing to the large collection of portage packages, it's one of the best, if not the best platform to explore modern computing, at least as far as Linux and the various Unix shards left in it are concerned.
In the sense of popularity, Gentoo has, at least to some degree, sank into obscurity, there's no question about it. But in the greater landscape of modern computing, it's not just "still relevant", it's probably one of the most vigorous islands of the free software and community spirit behind it, at a time when this spirit is floundering, especially among Linux developers.