Are we taking supercomputing code seriously?

[kcdodd]

It also assumes code produced by a software engineer could fair any better. I have rarely used any library, built by "software professionals", which was free of problems of usability, extensibility, and reliability. What you are really talking about is adding time and money to research. Some research warrants it, some does not. Some equipment and code is reusable and some is not. The level of abstraction he suggests may or may not increase flexibility, assuming such flexibility would even be useful, but it may also decrease efficiency, one thing which is almost always useful. Reliability is always an issue of any project of any reasonable size. Again, more time and money to get that. Test test test. Experimentalists are used to testing stuff, so whats the diff.

Well OK. Test code vs production code.

So how does one explain the "Harry Read Me" file from Climategate?

Once you get sloppy it hardly ever improves. And the test code winds up encapsulated in the production suite. Quality code takes time. Why waste time when there are new discoveries to be made?

Now every bit of code I write may not go through the FAA grinder. But I like to write it all as if that was the intent.

Standards.

Once again I fail to understand the meaning of what you are saying.

[happyjack27]

a good programmer should be able to produce more readable and flexible code that runs orders of magnitude faster and has higher accuracy than a non-programmer, and in considerably less time.

a good programmer knows things about computational complexity classes, program optimization, good coding standards, common code libraries, etc. should not only know but have mastered.

and if it's code to be run on a supercomputer, presumably you'd want it to be efficient (i.e. fast). while modern compilers can do a lot of low-level optimizations, they can't do the big complexity-time-class type optimizations and the like that make orders of magnitude efficiency improvements.

add in that on the emergence of data-parrallel hardware such as gpgpus for scientific programming (which offers still more orders of magnitude speedup), which is a whole new level of programming...

it becomes blatently clear that non non-expert programmer should be programming scientific applications for supercomputers. maybe a scientistic can write some of the basic code, sure. get all the physics right. but if you want to do a production simulation run that's going to take at least a month of continuous computing time. well, give it to a real programmer and he'll write you software that'll do it in a day. and the answers will actually be correct.

[kcdodd]

I think we are talking to cross purposes. It's not like this is either or. The scientists code the science. The engineers code stuff that isn't science. I doubt software engineers generally know very much about how different physical problems can be solved numerically, like how to turn an n^6 problem into an n^2 problem etc. A scientist can spend their whole life learning just that. Just as a physicist generally does not know how to manage data on 100 cpus in parallel.

[WizWom]

I think we are talking to cross purposes. It's not like this is either or. The scientists code the science. The engineers code stuff that isn't science. I doubt software engineers generally know very much about how different physical problems can be solved numerically, like how to turn an n^6 problem into an n^2 problem etc. A scientist can spend their whole life learning just that. Just as a physicist generally does not know how to manage data on 100 cpus in parallel.

That's why you consult with a Comp Sci Wonk who's read Knuth and is up on the Literature.
Division of labor. If its complicated enough to have a deep solution, its complicated enough to hire the guy who knows that.

[TallDave]

Well, I'll be working with MCNP-X pretty soon, and I'll have an answer as to the elegance of it.

That looks interesting. Might have to dust off my Fortran.

I suspect over the next 20 years the lines between experts and programmers will become increasingly blurry as experts are increasingly utilized to create expert systems, or increasingly specialized portions of them, rather than applying their knowledge directly. Humanity has reached a point where knowledge complexity is starting to overwhelm our puny memory registers, and we increasingly need more scalable substrates to get useful conclusions out of masses of expert knowledge.

At some further point expert system creation itself will be largely a product of expert systems, and the words "strong AI" will start to be whispered in the corridors.

[happyjack27]

I doubt software engineers generally know very much about how different physical problems can be solved numerically, like how to turn an n^6 problem into an n^2 problem etc.

on the contrary, that's exactly the kind of thing that a real programmer gets off on. which is kinda my point.

[93143]

happyjack27, as a doctoral-candidate computational fluid dynamicist, I disagree with your position.

[happyjack27]

well as a professional computer programmer i agree with my position.

[WizWom]

I see your B.S. and raise you 20 years of work experience

[ladajo]

I see you with my Sinclair...

[93143]

Work experience in what? This isn't an argument from authority here; I'm just pointing out the existence of the field.

My point is, the reason computational fluid dynamics is a vast and highly active field, and has been almost since computers were invented, is because it isn't enough to just "get the physics right" and let someone whose last job was programming a bus stop announcer machine produce a final version. The physics and numerics are tightly coupled, and you need to know both, especially if you're doing something new.

[WizWom]

Work experience in what? This isn't an argument from authority here; I'm just pointing out the existence of the field.

Exactly. By "B.S." I mean Bull product, not a degree. I've done Business and Engineering software, written CAD systems and Web Applications, even done a cellular automata or two and played with recursive systems in real-world applications - and I KNOW I'm not up to doing a fluid dynamics or nucleonics chain application without taking classes to know what's going on, then analyzing the heck out of it to make it work. Knowing what you know and what you don't know is an important part of the job.

Now, if I worked closely with a physicist who knew the physics, I could get the job done - because I know how to translate mathematics into code efficiently and compactly. And it would work properly. But I would NOT be able to find the optimization tricks in the algorithm that a proper compsci wonk would. I am, at the core, a hacker.

[93143]

By "B.S." I mean Bull product, not a degree.

I assumed that. What I'm not quite sure about is whose post you were responding to.

and I KNOW I'm not up to doing a fluid dynamics or nucleonics chain application without taking classes to know what's going on, then analyzing the heck out of it to make it work.

Tell me about it. My M.Sc. was in a different branch of CFD, and it wasn't until several months after the start of my Ph.D. that I finally knew enough to start messing with the group's research code (which is, BTW, a supercomputer code; it uses MPI, and its scalability properties are quite good)...

I guess what I'm saying is that sure, I agree, physicists who don't program seriously shouldn't necessarily be writing supercomputer codes. But the solution is not necessarily to get help from programmers who don't know the physics. Just programming the physics in question is often a field in itself...