A groundbreaking article released earlier today by Advanced Aquarist magazine points to a little discussed, but troubling fact of life in our hobby: manufacturers are free to make whatever claims they want about their products and there is no real way for consumers to know if they are telling the truth or not. In “Experimental Comparison of Measured Flow Output of Aquarium Propeller Pumps” by Michael Sanford, William Straka and Sanjay Joshi, rigorous and advanced experimental design and testing is performed on a number of commonly used pumps for water movement in our reef systems. The results are quite startling as some well known and well regarded pieces of equipment fail to live up to their performance claims.
Few would quibble with minor variations from stated specs, but when stalwart and generally reliable names like Tunze fall short by nearly 50% consumers need to take note. I have no intention of turning this into a company bashing post and readers can draw their own conclusions from the article. I merely want to highlight that there is a real need for independent testing in the hobby– something that is not going to come about easily due to the time, expense and expertise needed. One can also make a general call for manufacturers to make the effort to evaluate their products more realistically– again without an independent testing body there is little incentive to do so.
So, what are hobbyists to do? In cases like this where respectable companies with positive track records are at fault and true data is only garnered by sophisticated testing beyond hobbyist means there really isn’t much that can be done beyond holding companies accountable when real data exposes gross inaccuracies. In other cases, approaching your purchases with some measured critical thinking can go a long way in helping keep manufacturers honest. Rich Ross’ excellent Reefs Magazine series on Skeptical Reefkeeping offers hobbyists much good advice in this regard. I’d recommend a look at the section on the Baloney Detection Kit for Reef Products found HERE. Other high quality independent studies can be found in Sanjay Joshi’s Lighting Guide and Ken Feldman’s skimmer study. Our hobby needs to find better ways to encourage and support these kind of efforts.
The problem of the veracity of manufacturer claims is not new, nor is it likely to go away any time soon. Our hats are off to the authors of this new study and to others like them who are willing to put in the time and effort needed to help lend clarity to this murky side of the hobby. If any of you out there have suggestions to help sort through this industry-wide mess, we’d love to hear from you.
I’ve always taken the “specs” written on the back of a bottle/ box with a grain of salt in this hobby. Some claims just seem a little extreme for me but I never would have expected a 50% margin from a company that’s held in such high regard by so many. Is it really the lack of regulation or just a reflection of the companies standards?
*company’s*
I think a lot of people are missing the addendum to the article where Tunze’s reaction is stated to the testing. I think it should also be noted that EcoTech was nearly just as wrong about their flow rates. In my opinion the problem here is not inflation by companies wishing to sell more. The problem is that the industry standard is to calculate a pumps potential output. I have nothing to back this up, but both EcoTech and Tunze may have used similar calculations for their pumps and missed the mark in opposite directions due to the simple fact that a Vortech sits directly on the glass and a Tunze does not. It’s really impossible to say a company is at fault based on their advertised numbers. It is however possible to say that their means of getting to those numbers is wrong.
I’m not saying that they released information to purposely mislead but you would think that a company with a reputation would want to make sure they were getting the most accurate readings available especially when your business is centered on that information.
I would have thought that about all the companies involved who were tested. I’m sure none of them were trying to be dishonest. The way people quantify and measure things changes once in awhile and this may just be one of those turning points.
Great stuff Randy. That Rich Ross guy sure sounds dreamy!
I am not sure how Ecotech was nearly just as wrong – it seems to me that the percentages off from advertised are wildly different. And it seems there must be some kind of difference in a product that performs better than advertised and one that performs worse than advertised. I think a company is completely ‘at fault’ if their advertised numbers are wrong, especially if they did the testing. They did it all, and the TUNZE response shows that they understand that and that they take responsibility. That more than anything makes them a primo company in my estimation.
But that is all really a distraction from Randy’s main point as does pointing out that people are missing the Tunze respons – just because someone said it, just because its written down on a package does not mean its true. A lot of times you have no choice but to trust information like that in our hobby, but at the same time you have to leave room for the idea that its all wrong.
Time for a new slide for my MACNA talk!
The real point is – Should we expect companies to test their products. ? I would expect any company that wishes to establish itself as a premium, reputable company should in fact be testing their products to see if it meets their advertised claims. How much R&D does a company actually put into the product. ? What am I paying a premium for ?
Unfortunately, a hobbiest does not have access or the ability to conduct their own tests and has to rely to a large extent on the information supplied by the manufacturer or its sales arm and trust the information that is provided.
We should, and I really honestly believe they all tested them to the best of their knowledge. I think what this article really highlights is how an independent can turn things upside down long enough to get things changed. I don’t think there’s anything bigger in reef news right now than this series of tests and the responses to it, and I doubt any of these highly reputable companies will be moving backwards from this point.
Sorry Randy, I didn’t mean to say that you were somehow calling out Tunze with your article. I was trying to to preemptively respond to what I had been reading everywhere all day and knew was coming. I completely agree that we shouldn’t always take what’s on the box fro granted.
I don’t feel unjustly represented at all. Enjoying the conversation. Now continue discussing amongst yourselves……
While I have you out in public Sanjay, can I ask what size tank was used? I may have missed it in the article.
How could you be sure you weren’t getting any compounded flow from currents swirling around the tank?
A few weeks back I was considering looking into an instrument to test flow rates with, but after reading through your methodology, forget it!
Just to avoid any compounding flow with currents, a 200G tank was used.
I thought it was a 350-gallon tank. Maybe I misread the article in AA.
No, I misquoted. It is ~350G. 72″LX36″WX30″H
I think if a company is going to make hard and fast claims, they should have to be able to back them up. Of course in this instance, TUNZE could have done that, so it still takes independent testing. We need someone to donate a few million bucks to Reef Busters. 😀
While most seem to expect that manufacture claims has been tested, perhaps it should be just the opposite and we should expect them to not test their products. I think that is where we are now; take all product claims with 200 gallon bucket of salt.
I feel there is a huge conflict of interest that everyone seems to be ignoring. The test was conducted on equipment rented and provided by EcoTech Marine and the measurements were made by an EcoTech Marine intern. I’m not questioning anyone’s integrity here, but for the study to be valid, shouldn’t it be done independently of a pump manufacturer’s involvement? I realize Tunze has confirmed the results of their own pumps and has released a press release indicating what they plan to do next, but based on my experience with both Tunze and VorTech pumps, the results don’t match what I’ve seen.
According to the results, the EcoTech Marine MP10 puts out around 100gph more than the Tunze 6105. I have both pumps running on the same system and I can tell you the Tunze puts out significantly more water flow. The two pumps are on opposite ends of a four foot long tank and the Tunze creates a huge bare spot on the sand despite being pointed upward at an angle. The EcoTech on the other hand, points straight out and there’s not a bare spot on the opposite end from where it is located. Of course, I don’t have a single number to represent what I’m seeing and you could assume that one pump emits a more concentrated flow with the other a broader flow. Despite that, my experience with each pump would lead me to think that the Tunze 6105 has a higher gallon per hour rate than a VorTech MP10, and perhaps the study is either flawed, the testing equipment improperly used, or the data misinterpreted. Again, I’m not trying to make accusations or question the integrity of the tester and article writers.
The paper was focused on measuring flow rate (GPH) out of the pump. Not how the water disperses in a tank. Way too many variables there.
Using hands, eyes, and sand blowing is not a quantitative measure. The methodology has been described in detail to allow anyone to repeat the test. The intention was to present a methodology to determine more accurately (than a bag test, or pipe test etc that may be being used incorrectly) the flow rate of propeller pumps. Whether you choose to ignore it or accept it is up to you.
I’m not saying that I don’t believe the data, but some of it is hard to digest, especially when it claims a VorTech MP10 puts out more gph than a Tunze 6105. I also realize that seeing sand move and feeling flow on our hands is in no way quantitative or even remotely scientific. However, it is the only way a hobbyist without testing equipment can judge the output of their pumps.
Despite all of this, I would like to see the tests performed by an independent group. Someone who isn’t involved with any of the companies whose products are being tested. I would also like to see more information on the tests Tunze performed. Did they use the same equipment? Did they use the same methods? Did they measure the flow rates of other brands of pumps too? Also, what were the conditions of each of the pumps used in the original study. I’m assuming all pumps were brand new, but I didn’t see any of that information in the AA article.
I think there’s too many variables for any single test to ever be enough. Fluid dynamics are far more complicated than most people think. We might as well be trying to quantify the effect a star on the other side of the galaxy has on bacteria here on earth.
The key here is to look at the tests done and base your conclusion on that context. Even in a 350 gallon tank, you will see some change due to back flow.
I have to agree with you that even though it’s clear Vortech was not trying to stack things in their favor, it cannot truly be scientific unless no biased parties are involved.
I think Tony needs to start an independent testing lab and charge all companies to benchmark their pumps for them. then we could easily ignore ratings not coming from there.
I was never trying to make the claim that EcoTech was doing anything wrong. I love their products and I own some of them at this very moment and will purchase from them again. Their customer service is top notch from my experience and the VorTechs themselves are revolutionary.
I agree there are too many variables to consider, as even gph rates aren’t the make or break for a pump in a reef aquarium. It’s more about how the water is moved, compounding currents, how much turbulence is created, and so on.
http://www.advancedaquarist.com/2009/10/review
That’s the information I was looking for Rich…a flow rate measurement of a product conducted by someone other than the person who makes that product. And I’m glad to see Dana’s measurements coincide (to some extent) with those collected by the most recent study. Again, I wasn’t doubting the results of the AA article, but the skeptic in me took over since I’ve seen both pumps in action for many consecutive months.
Very interesting to see results in a smaller tank. It would be interesting to see the same pump tested in multiple tanks of varying size. I’m really surprised this area hasn’t been explored further.