This is a really good, and thorough, response- and one that probably can't be settled on a blog.
I'll try to go back through this in order, but up front I want to address the transparency issue since it came up a few times. I agree with consumer education, and I think that I would extend that sentiment to the costs associated with carbon. So, of course education to ratepayers is incredibly important- but it needs to actually be a balanced education that includes information about the externalities of carbon emissions (from air quality to climate change) and the associated impacts and costs for them and their children.
One more note on the costs question, I think that there needs to be serious consideration of rate restructuring to allow utilities to make these kinds of forward looking investments without undercutting their rate base. So compensation for a "network utility" role would allow and encourage more DG investments, as well as encouraging rates that reflect the actual values of different services. For instance, investments in solar PV that offset the need to construct new peaking gas plants can be valued not just on capital costs, but as incubating consumers from fluctuating fuel costs. This is a nuance I think gets ignored, because our current rate structure is designed to accomodate centralized generation and fossil fuels.
Throughout the rest of your reply there are a couple of really good points- and like I said above- difficult to answer in a post or above my pay grade and technical expertise. However, there are a few things I can address.
First,putting aside the ACESA numbers (which I don't actually care much about) the capacity factors of renewables are currently in the 30% range when you average them together. But as you break down those different inputs, you see that any reasonable portfolio would have to include mostly offshore wind, a little onshore wind, about the same amount CSP, and then even less solar PV. Plus Hydro and Geothermal is actually enough to supply baseload when considering those other factors (storage, transmission, DR). The capacity factors on each of those technologies, when you disaggregate them, give you a much better idea of how to manage a portfolio of multiple technologies to play to each others strengths. For instance, wind blows mostly at night, solar PV and CSP mostly during peak hours, and so on.
There are risks associated with moving to a more distributed system, and there are a lot of smart folks who I work with who agree and remind me about it regularly (and conversations like this, of course). But, at the end of the day, my feeling is that I don't believe in an entirely distributed system- just a more distributed one. For instance, I don't think every home will be on its own little microgrid. I think more buildings will become net zero, through efficiency, DG, and energy management systems, and they will have a different relationship with the grid than they used to. Also, I think some technologies, solar PV and EV's being two, are reflections of this transition.I wrote some of the high-level vision stuff regarding this here http://www.americanprogress.org/issues/green/report/2012/08/27/33827/the-networked-energy-web/.
I agree with you somewhat about demand response, but I can't help but feel that scenario is more of a perfect storm than a new normal. IF storage isn't enough, IF high voltage transmission lines cant pull RE from thousands of miles away, IF hydro and geothermal aren't enough, IF coal and natural gas in small quantities, all aren't enough then yeah, maybe you cant turn on your toaster. Although that's also assuming building energy management systems dont prioritize appliances to 'ration' energy in the building (i.e. washing machine doesn't need power right now, toaster does. TV doesn't, HVAC does). I think its a failure of imagination, not a pipe dream.
I don't understand what you mean about the new capacity (or about where we disagree on that question). I agree that existing capacity should be utilized before building new capacity. Project economics and capacity needs should drive those investment chocies. Definitely. Also not entirely sure about the argument in the second to last paragraph. Do you mean renewables are new, innovative, and unproven?
Good dialogue on this. I think that a low-carbon path is a neccesity, so we have to find the answers to this questions and get moving promptly with a comprehensive, holistic energy policy. By what I have looked at, the absolute largest amount of supply natural gas can play in that picture with the US meeting its 2030 climate goals is 35%. I'd be interested to see anything that gives it more of a stake.