By Sanjay Joshi

In an earlier article I described my battles with the RTN (Rapid Tissue Necrosis) issues that led to the devastation of about 1/3rd of my Acropora. However all was not lost, there were still several colonies that survived and by August 2013 the tank was well on its way to recovery giving me hope that the tank would soon be back to its former glory.

The tank had been running for almost 7 years at this point, and over the course of time I had improved the robustness of my tank to withstand failures from design and “Murphy.” This was achieved through the incorporation of several elements of failsafe design based on my experiences and feedback on my presentations at several reef clubs and conferences on “Thwarting Murphy.” The confidence in my system was high and I had already had several rounds of success with leaving my tank alone for up to two weeks over summer and winter breaks.

A little bit of background about my tank design philosophy. I have always espoused the design philosophy of fail safe. As a long time reef hobbyist, with a lot of failures from mistakes and poor design, I have come to realize:

1) All systems can fail,

2) Will fail at the weakest link,

3) And will fail at the most inopportune time.

In this context, how is one to design systems that can withstand the impact of failure for a limited time? This is the challenge. Through analysis of failures of other tank failures, often posted on bulletin boards, I have designed my systems to incorporate redundancies along with using more robust industrial controllers for heaters and chillers. The system uses an apex controller which is used primarily for monitoring and reporting functions such as sending emails and text alerts in case of system alarms programmed into the controller. I do not use the apex to control the key life support systems to avoid creating a single point of failure. As a backup, I could always log into my controller remotely from my Iphone app and keep additional tabs while I was away. This approach has served me well over the years and I had built up enough faith and confidence in it to leave my system untended for routine tasks for up to 2 weeks. I had done this several times already.

Imagine my horror when I returned from MACNA early September 2013 to find about 1/3rd of my coral dying and/or bleaching, cloudy water and dead floating fish. What happened? A quick check of the control panel showed the tank temperature at 90 (F)! An inventory of the tank inhabitants revealed the complete nature of the gut wrenching losses. About 1/3rd of the Acropora was dead or dying. The fish loss was just as dramatic. The complete fish list prior to this and the initial losses incurred are listed below (in bold):

500G: (55 Fish)

2 – Neocirrhitus armatus, Flame Hawk (1M, 1F)
2 – Oxycirrhites typus, Long Nose Hawk (1M, IF)
2 – Centropyge joculator, Joculator (1M, IF)
2 – Midnight Black Photon Clownfish (1M, IF)
5 – Chrysiptera rollandi, Rolandi Damsels (Spawning Group)
4 – Chrysiptera hemicyanea, Yellow Belly Kupang Damsel (Spawning Group)
3 – Genicanthus bellus (1M, 2F) (Spawning)
5 – Anthias pulcherimus (or randalli?)
4 – Pseudanthias tuka
2 – Assessor flavissimus, Yellow Assessors (1M, IF)
2 – Gramma loreto, Royal Gramma (1M, IF)
2 – Gramma melacara, Black Cap Basslet (1M, IF)
2 – Centropyge aurantius, Golden Angels (1M, IF)
1 – Zebrasoma xanthurum, Purple Tang
1 – Zebrasoma gematum, Gem Tang
1 – Naso elegans, Naso Tang
1 – Paracanthurus hepatus, Hippo Tang
1 – Acanthurus achilles, Achilles Tang
1 – Zebrasoma flaviscens, Yellow Tang
1 – Synchiropus splendidus, Mandarin
1 – Anampses feminus, Femininus Wrasse
1 – Anampses lennardi, Lenardi Wrasse
1 – Macropharyngodon lapillus, Mauritius Leopard Wrasse
1 – Xenojulis margaritaceus, Gold Nugget Wrasse
1 – Chelmon marginalis, Marginalis Butterfly
1 – Pterapogon kauderni, Bangaii Cardinal
1 – Wetmorella nigropinnata, Possum Wrasse
1 – Atrosalarias fuscus, Yellow Sailfin Blenny
8 – Clownfish (4 assorted pairs, in overflow)

I was in utter disbelief – how could this have happened? I had simulated the failure of my chiller and even with the metal halides the temperature never went over 85 (F) on the hottest days. Upon checking the chiller the first thing I noticed was that the garage was really hot from the chiller exhaust. Much hotter than normal. The chiller was running, but obviously something was wrong since it was not cooling the tank. The first thing to do was to cut the power to the chiller. The next course of action was to bring the tank temperature down quickly. The lights were shut off to avoid adding any additional heat. Bringing the temperature down to the low 80s is not an easy task in a 600G system. Eight bags of ice from the grocery store were put in the sump and they melted away rapidly with barely a dent in the temperature. Then eight more bags were added along with all the cool packs that I had stored in my freezer from the shipping of corals and fish. These were put into large zip lock bags and floated in the sump and in the overflow. Another eight bags of ice were also used, and finally the temperature began to gradually come down. Additionally I was able to borrow a ½ HP chiller from a friend to help run the system while I got the old chiller diagnosed and repaired.

During this time, I started investigating the controller and why it failed to send me a message. Querying the logs to see the temperature history, I was very surprised to see that the tank had been hitting almost 90 (F) for the last 5 days.

Clearly, the controller should have sent me text and email messages based on the alarm set to trigger at 82 (F). Why did the controller fail to do so? On testing the function manually it failed to send me a message and this helped me narrow it down to the email server being used to send the emails. Further investigations through checking various online forums and the Neptune website led me to discover that GMX (as recommended in the apex manual) had changed its protocol and was no longer compatible with apex. This had just happened 2 weeks prior and I was not aware of it.

This failure exposed another weak link in the system. Communication Failure. Lessons learned from this can be used to further strengthen the system against failure. I now use the following approach:

1) In addition to the system alarm, program the controller to send an email and text message every day in the morning at 9:00 AM. This provides a simple daily check on communication failures on a daily basis.

2) Before long travels, manually trigger the alarms to ensure that they are working. Additionally program the controller to send me text and email 2X daily (9:00 AM and 9:00 PM).

Could the tank heating problem have been detected any other way? Unfortunately, YES. I could have manually checked on my tank using the apex App. However I was too caught up in enjoying MACNA and trusting my automation to bother checking manually. Manual checking could have alerted me earlier.

The damage was done. All the dying coral skeletons were removed in order to stop any further deterioration in water quality. As you can see from the picture below, large Acropora colonies that survived the RTN attacks earlier in the summer lay wasted by the 5 day cycle of 86-90 (F). These were removed from the tank using hammers and screwdrivers. I consoled myself with the fact that not all was lost, at least my gem tang, the mother of all photons clownfish, and my bellus trio were still alive. Additionally, there were still several large colonies of Acropora that were still alive and could potentially recover. But this hope was short-lived. Almost a week later when I thought I was through with any more deaths, the prized Gem tang, the mother of all photon clownfish and the male bellus died. Now this really hurt. The false hopes were dashed. Devastation as far as fish were concerned was almost complete. I had lost fish that I would not be able to replace easily. This was more painful than the coral losses, especially after having gone through the RTN losses of coral. I wonder if there is a delayed impact of heat on the fish, which led to fish losses a week after the temperatures were back to normal?

Interestingly, all of the LPS, Zoanthids, chalices and gorgonians survived, along with the non Acroporid SPS corals. There were a few fish surprises too. TheAnampses lennardi, supposedly a cooler water fish came through this ordeal without any observable issues. It is still alive almost a year after the incident. This is one fish that I never thought would survive. On the death of the male bellus, the larger female started to change into a male within a couple of days after the male died. Below is a picture of it showing both the male and female coloration. And this was a regularly spawning female!

The chiller repair shop determined that the chiller coil had sprung a leak and the cooling gas had completely leaked out. The chiller was being forced to work, but it was not removing any heat instead it was adding heat to the water. Luckily they were able to repair it, but could not give me any guarantee on the repair unless they replaced the complete coil. Now knowing the result of a gas leak, I took the chance on the repair to tide me over until I executed on a permanent course of action – ELIMINATE THE CHILLER.

Eliminating the chiller from the system would obviously prevent this from ever happening again. I never want a failed chiller to destroy my system again. The prime reason for including a chiller was to combat the heat added to the system by the 3 400W Metal Halides. The tank is located in the basement and is generally cooler than the house, however the metal halides would still raise the tank temperatures to 85 (F) in the summer. Eliminating the chiller meant the lighting system had to be re-evaluated. At this point, I started contemplating the switch to LEDs. LED technology was advancing to the point where a lot of my initial concerns about switching a large 500G aquarium to LED were starting to get addressed (more on this in a subsequent article).

While I was contemplating and evaluating my choices, the remaining corals and fish started to recover and by October the tank was starting to make a turnaround again. At this point I had decided to switch to LEDs, but before doing so I wanted to address the one remaining problem that had plagued my tank for the last 7 years (since the inception) – ACRO EATING FLATWORMS (AEFW). I had lived with these creatures for a long time and I was successful growing out large/huge Acropora colonies in spite of them. But it was an ongoing battle that I had to stay ahead of all the time. It also precluded me from adding new smooth skinned deepwater type Acropora that always fell victim to the AEFW. Having lost almost 2/3 of the Acropora population I thought this would be the perfect opportunity to try to eradicate the AEFW once and for all. Research on AEFW eradication suggested the following strategy and approach:

1) Removal of ALL Acropora colonies from the main display along with scraping off all Acropora tissue from the rocks

2) Dip and treat each coral once a week for a period of 6 weeks to ensure all hatchlings are eradicated, since the treatments do not kill the eggs

3) Keep main tank fallow of all Acropora (for about 6 weeks) in case any eggs are on the rock and the hatchlings can be starved to death.

There are various dips and treatments recommended for AEFW – Reef Pest Solutions, Revive, Bayer Advanced Complete Insect Killer and others. I had the fore-mentioned on hand and decided to stick with these, and treat on a rotating basis until I ran out of what I had.

The colonies of Acropora were removed from the tank and placed in a 200G holding tub after a dip and rinse with Bayer Advanced Complete Insect Killer. The recommended dosage for this is 5ml/cup. These large colonies were to be dipped in a 20G trash can, so the dosage was scaled appropriately based on volume. The holding tub was set up with some dry live rock, 400W metal halide and a skimmer, along with an Iwaki 40RLT and 2 MP40s as recirculation pumps. As you can see in the picture there were still quite a few Acropora that had survived the prior ordeals.

For the first few weeks, the plan seemed to work well. Until disaster struck again. I was away on one of my weekend trips and came home to a large colony dying and polluting the water. The stress caused by this started a chain reaction over the next few days that impacted most of the large colonies. The stress of dipping, relocation in less than optimal conditions, the changes in lighting and flow all took their toll on the large colonies and they started to die off. At first I tried saving frags, but eventually by week 4 I had reached the point of futility and gave up. Managing large amounts of Acropora in a smaller freshly set up system is definitely not very conducive to Acropora health. At this point I just decided that the effort was not worth it and tossed out the remaining Acropora. From now on, every coral that would be placed in my tank would be treated for AEFW and other pests would be treated before going into the tank. Ideally it should be quarantined before going into the tank. My current practice is to follow the protocol outlined below:

1) Acquire corals from individuals with no aquarium pests

2) Every coral is dipped in Bayer Complete Insect killer. It kills AEFW, and red bugs and potentially other little creatures that could be pests

3) For maricultured colonies, the coral is removed from the base, inspected and dip treated

4) All live rock attached to Acropora are removed and discarded

5) If any eggs are found, only clean tips of the coral are saved and dip treated and rest is thrown out.

6) When in doubt throw it out, take no chances.

In a span of 7 months I had gone from a thriving tank full of large colonies and rare fish to a tank devoid of any Acropora and only a few fish. Below is what the tank looked like in October 2013. I had added a few more fish to keep me interested in the tank including a pair of small Bandit Angels. Several non-acroporids were still alive and thriving. The tank was kept devoid of Acropora for 5 weeks.

While destructive events such as these can be quite devastating and challenge the willingness to stay in the hobby, they also provide a learning and teaching experience. It’s one thing to have an accident caused possibly by poor design or incomplete thinking of the types of failure and its consequences, it is an even bigger foolishness if these events are allowed to happen again. It is my hope that by sharing my experiences and course of actions you will learn from this and become a better reef keeper. May you never repeat the mistakes I made.