I agree with Lawbag on this one, although I think it was the combination of PDF publishing and the OGL that did it.
Let's leave quality aside for a moment, although I think that's really what did in the d20 market.
Before open game licenses, if there was customer demand for a product on, say, ships and sailing in D&D, there was only one place to get it: TSR. Prospective writers had to submit their proposals, TSR would select the stuff they liked, assign an editor and a product lead, and you'd eventually have a product. Good quality or bad, playtested or no, there was one book. All the potential market for such a book went into one place. All subsequent books only had to be compatible with that one book.
With PDF publishing and the OGL, any schlub could produce such a book. I own at least eight third-party 3.5 books on ships and the sea (and three just on airships). None of them are compatible with each other mechanically. And then there's Stormwrack, which came along at the end. Regardless of quality, the market's been diluted by a glut of products on the same topic. Each publisher is getting a fraction of the profit a single publisher would have, and thus it's harder for any of them to produce a quality book, as talent costs money.
Worse, old third-party products no longer vanish to moldy basements and eBay when the initial print run sells out; they hang around in PDF form forever, competing not just with the current official Ships and Sailing book and each other, but with any future Ships and Sailing book that might get published.