Jeff Jarvis, noting that two podcasters are hanging up their mike's, comments on the difficulties:

Step back from the keyboard before you start writing that made-up trend story. I sense that people ( Winer aside ) don’t flame out on blogs the way they might on podcasts and I think the reason for that is that podscasts are both more of a production and more of a performance. It’s harder. That’s also why fewer will start podcasts -- and why I haven’t. It’s easier to blather through a keyboard than a microphone.

It is more difficult. I've only done 4, and I can tell you - they take far more time and effort than posting. First, there's the actual time to have the interview (or conversation, etc). That's the easy part. Then comes the hard part - production work. It's at this point that you start to understand why radio stations (and TV, and movies) have sound people. Doing noise reduction, mixing, and lag correction isn't an insurmountable task, but it's not trivial, either.

Not everyone has the time to spend on it. Blogging takes very little time - with audio, you have to set aside an hour or three before you can get the results posted.