Eat that GW believers!

[Josh Cryer]

and historical adjustments:

http://chiefio.wordpress.com/2010/01/13 ... an-cooked/

This was quite interesting. I wondered, "How come GHCN raw doesn't coincide with the raw NOAA temperature record?" (As is being claimed.)

Well, I spent the last twenty minutes going over the data sets, and it seems quite clear, now.

First, he links this data set as his "raw" data for Central Park (note that this is in Fahrenheit): http://www.erh.noaa.gov/okx/climate/rec ... ltemp.html

Then, he links the GISS data site for "raw" data: http://data.giss.nasa.gov/gistemp/station_data/

The relevant station ID number is 725030010 (plop it in to the search field).

Your options for the data set are:

1) raw GHCN data + USHCN corrections
2) after combining sources at same location
3) after homogenity adjustment

He wants you to click 1, here.

If you do this, load the relevant text file, and compare it to the original link (the first link I have given), and you compare the temperatures, they, indeed, are way way off the chart.

So, being scientifically inclined, I tried options 2 and options 3. It turns out the original "raw data" is in fact using homogenized data!

Want proof? You can view these graphs by going to the page and following the steps I explained to you.

raw GHCN data + USHCN corrections


after homogenity adjustment


Now, to further illustrate the point, I will take the year 2008 from the GISS homogenized temperatures, and I will take the year from the original "raw" link he provides, and show you how their temperatures are identical (within the margin of error because the "original" link is converted from C to F and I will have to convert it back again). I am using 2008 because 2009 is incomplete in the GISS text file.

Note the last number is the annual average.

Code: Select all

ORIGINAL (in F)
2008  36.5 35.8 42.6 54.9 60.1 74.0 78.4 73.8 68.8 55.1 45.8 38.1  55.3
ORIGINAL (in C with extra decimal)
2008  2.5 2.11 5.88 12.72 15.61 23.33 25.77 23.22 20.44 12.83 7.66 3.38  12.94

GISS / GHCN (in C)
2008 2.5 2.1 6.0 12.8 15.6 23.4 25.8 23.2 20.4 12.9 7.7 3.4   2.5 11.5 24.1 13.7  12.93
GISS / GHCN (in C with seasonal avgs removed)
2008 2.5 2.1 6.0 12.8 15.6 23.4 25.8 23.2 20.4 12.9 7.7 3.4 12.93

ORIGINAL OVER GHCN (with decimal removed)
ORIG 2008 2.5 2.1 5.8 12.7 15.6 23.3 25.7 23.2 20.4 12.8 7.6 3.3  12.94
GHCN 2008 2.5 2.1 6.0 12.8 15.6 23.4 25.8 23.2 20.4 12.9 7.7 3.4  12.93

I looked at the data, and attempted to discover the truth. A claim was made, that claim came with copious amounts of links to "bolster" the claim. In the end, data that was called "raw" was in fact homogenized, and the "conspiracy" about "data manipulation," is easily debunked (it's not even a conspiracy because D’Aleo doesn't even know what he's doing when he looks at the data). But instead of thinking critically, we will still believe nonsense.

Being scientifically minded I intuited something was wrong within seconds of seeing the data. But because I am scientifically minded I did not immediately write it off as true or false, I had to see where the comparison failed or succeeded. Usually I don't go to these lengths to explain why I see something and how I see it (I had to type in each temperature into google to convert them), I simply do a random test (picking several numbers out of the homogonized data and several out of the "raw" original data, plus, I can intuit C to F fairly easily and it was obvious that the "raw" data set was in fact GHCN/GISS homogenized), and come to my conclusion. However, I wanted to make it clear here, MSimon, not all of the arguments you read are credible. They are nonsense disguised as critical thinking. I wish you had just a bit of scientific urge in you to understand how things work rather than taking for granted what people say.

I'm afraid you would not have believed me if I simply replied, "MSimon, he is graphing raw but corrected data with homogenized data, and claiming the homogenized data is raw."

I could go in to the other reasons why the rest of this stuff is wrong, but *explaining* things is very difficult if you don't have a scientific inclination to actually do the research yourself.

[MSimon]

Josh,

Leave a comment there. (Or I will)

Smith is a good guy. If he has made an error he will fix it or explain his methods.

Update: Comment posted.

[MSimon]

And Josh,

I did the research. I handed it off to one of my assistants.

Think of me as management.

My field of expertise is in other areas. So I have to trust my staff to sort it out. Aren't you glad I brought it to your attention and that it is open source?

[MSimon]

So you are telling me that homogenization warms the past?

Or do the adjustments cool the past and then homogenization warms it?

Well it will get sorted out.

From Chiefio:

It looks to me like we will need to go all the way back to “first sources” to have any hope of finding out what is really going on in the temperature history of the planet. GHCN “Unadjusted” clearly is too adjusted to be suitable to the task.

So exactly what “input data” are the right ones? You get to chose based on what ‘adjustments’ and ‘corrections’ you would like to have. And they are different from each other, often by several degrees. From this we are supposed to be excited about fractional degrees of change? There is much more than that in the adjustments…

So my take at this point: the graph is mislabeled. It still proves Chief's point if not mine.

[Josh Cryer]

Josh Cryer
Your comment is awaiting moderation.

Guys D’Aleo’s “raw” data set from Central Park matches homogenized GHCN / GISS data, so this “cooking” of data is actually someone completely messing up the analysis. I explain it here: viewtopic.php?p=32329#32329

on January 16, 2010 at 2:23 am

2 hours *before* your post.

MSimon, the "raw + USHCN" is not completely raw, you must go to the NCDC to find that data. It is mostly raw, but it does have corrections.

homogenization warms the past because as I learned when they do the station TOD stuff they adjust upward the past temperatures rather than adjusting down current temperatures (this is how it should be done). Either way of *course* you guys are going to disagree with the homogenization efforts.

So let's ignore that for now. D'Alio's "raw" data is in fact exactly the same as the GISS / GHCN homogenized data. Exactly. If you convert GHCN to Fahrenheit and do simple rounding rules it comes out identical for all values that I tested (better than converting from F to C like I did).

He cannot claim that that data is raw. And I am having an extraordinarily hard time believing this was an accidental mistake, because I saw it in seconds and I am not super mathematically inclined. And have not done this sort of analysis in almost a decade.

edit: it's probably more do to UHI normalization than TOD.

[Josh Cryer]

Rather than converting the "raw" original to C I decided to convert the GHCN to F, since that shows less discontinuity (due to rounding errors).

Code: Select all

GHCN ORIGINAL (from C to F, with one decimal point)
2008  36.5 35.78 42.8 55.04  60.08 74.12 78.44 73.76 68.72 55.22 45.86 38.12  55.27
ROUNDED
GHCN 2008  36.5 35.8 42.8 55.0 60.1 74.1 78.4 73.8 68.7 55.2 45.9 38.1  55.3
ORIG 2008  36.5 35.8 42.6 54.9 60.1 74.0 78.4 73.8 68.8 55.1 45.8 38.1  55.3

They are essentially the same exact data set.

BTW, are you calling me your assistant?

I'd hate for my assistant to know more than me! In the military the guys who know more are the guys in charge. It works the same way with military contractors, in my experience (even dumb construction guys). I'm second in charge of our crew. Only downside is that if something bad happens it's blamed on me, but that is really an incentive to get things right.

[TallDave]

homogenization warms the past because as I learned when they do the station TOD stuff they adjust upward the past temperatures rather than adjusting down current temperatures (this is how it should be done). Either way of *course* you guys are going to disagree with the homogenization efforts.

Apparently we're supposed to believe TOD is so much bigger than UHI that the surplus accounts for a significant portion of the GW signal. Well, okay, maybe that's even true. But corrections that give people the answers they want should always be viewed very suspiciously, especially in a context where you can't replicate an experiment.

It's not even a foregone conclusion TOD adjustments should be going the direction they are. I trust Hansen's judgement about as far as I can throw a coal plant he's been arrested at.

If you think Mann and Hansen and Jones don't have an agenda here, you just aren't being intellectually honest with yourself. Maybe some of their conclusions are reasonable, but they've been caught with their fingers on the scale of a weak theory, and their policy prescriptions are lunacy. Not a situation in which being credulous is a good idea.

[Josh Cryer]

My edit suggested UHI was more responsible for this trend. Usually you can see the TOD adjustment in the raw vs homogenized data (there's a point where the raw data and the TOD data diverge), but it's not clear here. More than 80% of stations underwent a TOD change from daytime to nighttime, though (because the boxes were heating up and it made the temperatures higher), so if there is *anything* you do to the data, you have to do that. But I think UHI is just as important, especially with a station like this one. I can even concede that I don't like the station-station homogenization efforts (which btw were not done on Darwin), but they are backed by sound methods, and IceCube validated NOAA over Antarctica, so they have to be doing something right.

I will look up the (raw, unedited, unfiltered, from some crazy haired meteorologist who might even be anti-AGW) station history on the NCDC when the school VPN isn't being as hammered with requests as it right now.

BTW, you guys should be happy for this homogenization effect, it almost flat-lines the warming trend (which was undoubtedly almost all UHI). Now I'd be on your side if the actual homogenized data looked like the raw, while the raw looked liked the homogenized! Which, btw, is exactly what D'Alio wants you to believe! Imagine my concern when I thought he was actually right!

Major WTF moment there.

[MSimon]

My edit suggested UHI was more responsible for this trend. Usually you can see the TOD adjustment in the raw vs homogenized data (there's a point where the raw data and the TOD data diverge), but it's not clear here. More than 80% of stations underwent a TOD change from daytime to nighttime, though (because the boxes were heating up and it made the temperatures higher), so if there is *anything* you do to the data, you have to do that. But I think UHI is just as important, especially with a station like this one. I can even concede that I don't like the station-station homogenization efforts (which btw were not done on Darwin), but they are backed by sound methods, and IceCube validated NOAA over Antarctica, so they have to be doing something right.

I will look up the (raw, unedited, unfiltered, from some crazy haired meteorologist who might even be anti-AGW) station history on the NCDC when the school VPN isn't being as hammered with requests as it right now.

BTW, you guys should be happy for this homogenization effect, it almost flat-lines the warming trend (which was undoubtedly almost all UHI). Now I'd be on your side if the actual homogenized data looked like the raw, while the raw looked liked the homogenized! Which, btw, is exactly what D'Alio wants you to believe! Imagine my concern when I thought he was actually right!

Major WTF moment there.

The problem is that you can't do blanket adjustments for UHI. Every station has to be examined and adjusted individually.

http://gallery.surfacestations.org/main ... temId=1535



http://surfacestations.org/

This image is too big. Open it in another tab:
http://surfacestations.org/images/Watts_fig23.png

[MSimon]

I'd hate for my assistant to know more than me!

I have medical assistants who know more than me. I have mathematics assistants who know more than me.

It is not a matter of do they know more. It is a matter of can I do sanity checks on their work. Does that preclude error? Of course not. That is what we have open source for. Or in the case of aerospace armies of checkers and a procedure called a design review. And even in aerospace we conform somewhat to the open source model. Any engineer who wants can attend a review.

I remember one that was a project far removed from anything I was working on. But I had heard rumors. So I went. We had the designer in tears (figuratively) by the time we were done. I guarantee you if he stayed in the field he would be much more thoughtful the next time.

[MSimon]

And re:

viewtopic.php?p=32430#32430

America is supposed to have the BEST network in the world.

[Josh Cryer]

The problem is that you can't do blanket adjustments for UHI. Every station has to be examined and adjusted individually.

How do you propose to do that? Give climate scientists ten times the funding to hand apply the algorithms they've developed for each station?

They make the algorithms to reduce the labor requirements, but the steps that they'd take for each station would be similar. NCDC has the data, and it can be parsed and worked with analytically with software.

But here's the data for that station:

ROSEBURG RAW



ROSEBURG HOMOGENIZED



Yet again UHI plays a huge role in normalizing the temperatures for that station (reducing any overall perceived warming trend).

If they're really cooking the data, why not make it one degree higher? Just for shits and giggles. Maybe it's because their data is peer reviewed and people could easily find holes in it if they didn't follow the scientific methodology.

BTW, open source is good for weeding out people who are looking at the data wrong or doing the data wrong, and that is primarily why I like it.

Transparency doesn't allow people to manipulate me. Though they will use it to manipulate others, as in the D'Alio example.

[MSimon]

MSimon said:
The problem is that you can't do blanket adjustments for UHI. Every station has to be examined and adjusted individually.

How do you propose to do that? Give climate scientists ten times the funding to hand apply the algorithms they've developed for each station?

Shut down the modeling and put those folks to work on data. Because bad data = bad models.

But I do like your admission that the data is shoddy. i.e. my inexpert sanity check worked.

[Josh Cryer]

Shut down the modeling and put those folks to work on data. Because bad data = bad models.

Why? They don't specialize in models, they specialize in data analysis. They build algorithms to improve that data analysis. Are you saying that the scientists who build the models should stop their day job and go work for the data analysis people, hand applying algorithms that were developed through the peer review process and capable of being run on computers?

I expect that that would introduce more errors not less.

Seriously MSimon, think about it. You're saying for Gavin (GISS) to go work for Peterson (GHCN) in order to do a runt job analyzing data that interns could do. It's silly. If you want this done hire 10 students and delete the homogenization programs ('cause they'd be sure to copy them off of Peterson's hard drive and make the programs do the work for them).

But I do like your admission that the data is shoddy. i.e. my inexpert sanity check worked.

I believe you can find throughout this thread that the data is imperfect and needs adjustments and analysis. The data itself looks like noise if you go from raw NCDC data. Looking at RAW you'll find stations going up in temperature, down in temperature, not changing, etc. That is all due to the way they operated over time. The algorithms figure out how to improve upon them.

And that is part and partial due to the fact that they improved the data over time. The state of the temperature record is better than it was 50 years ago, and 10 years from now it will be better than any time before it (CLARREO).

[MSimon]

OK. Josh. You convinced me.

Fire the modelers and use the money saved to hire people to inspect the stations, install instruments take measurements, and use that to fix the data.