disk error with iStorage Server/ Win 2008 Failover Cluster

[mkhan]

Hello,

For past three weeks, I have been testing iStorage Server 2.08 and 2.20 on Windows Server 2008 Enterprise edition. I have setup Windows Failover Clustering, but failover sometimes doesn't work when Windows restart. The disks fail. keep give the popup to reformat the drive.

I have "Ultimate license for iStorage Server for each server.

Ok, here are my steps:
I have to Dell PowerEdge 2950's with Windows 2008 Failover Clustering

1. I installed iStorage server on both servers (same addition), added "Hard Disk" and "image File", and "Standard Image File", gave Device parameters, "Anonymous" access, checked ""Report as readonly device...." and "Enable multiple initiators with full access".

2. Have created twelve iSCSI targets on each server (because I am clustering MS SQL 2008 with full failover capability that requires many disks)
3. In Windows, initiated each disk using iSCSI Initiator Properties by:
-adding Target portals under Discovery tab
-from Targets tab, I found the targets, clicked on "Long on", check both checkboxes for Automatically restore... and "Enable multi-path (Yes, I have MPIO feature installed in Windows), clicked on Advance tab and selected the Local Adapter as "Microsoft iSCSI Initiator, picked the source and target IPs (I had to do this because the target logon was taking too long with multiple networks, which are needed for proper redundancy. Clicked ok, ok, and got it logged on.
3. Added the disks on each server under "Server Manager", formatted them and gave specific drive letters and names for each drive.
4. Now, came back to iStorage Server, and added one application for each drive. I repeated this on both servers. The Application specs are: Failover iSCSI SAN Node-- picked Base Target and Mirror Target and picked the option "Create mirror device without synchronization (Manual Initialization). I have tried with "Full Synch as well, but same errors)

5. Installed SQL server Failover, for Engine and Agent on one drive, SSAS, SSIS, Reports on separate disks, etc.

6. Now, everything works great!!
...until I Restart one of the servers, all of sudden some disks come online, while others Fail. In iStorage server everything seems to work fine, there are no errors. However, it does take it's time to come online when Windows restarts. I keep getting the message to reformat the drive, even after I disconnect from each initiators manually and tried reconnecting them. Sometimes the drives would come online after logging on and off from initiator and everything seems to go back to normal. However, some disks continue to fail. Now, I need to also recover the SQL data/log files that were on one of those disks that are giving the reformat error. Am I missing something????

Please assist. I have high hopes from this software and I need to have everything ready for productions by next week. Thank you.

[Charles [PM]]

Hi,

Happy Holidays!

I will contact you for providing further support.

Charles

[mkhan]

Thanks Charles. Happy Holidays...looking forward to your suggestions.

In case you are wondering, I have Windows Server 2008 x64 bit edition on Hardware RAID 10 using Dell Perc 6/i series RAID card. And yes, I have the relevant ports open in the Windows Firewall for both iSCSI TCP port 3260, and iSCSI Service and for iStorage Server and I have the appropriate exceptions for Processes and Apps open in our Kaspersky Enterprise for Servers (antivirus program). I have tried by disabling all security as well. I am using Windows Domain environment with Active Directory. It seems with the new version 2.20 there are more problems, since some of my drives have completely failed and I had to reformat and start from scratch. Then again, some others failed after I applied MS SQL SP2 and the Cluster has not been able to recover those drives. (Note during the MS SQL update, drives went offline and came right back online, but problem started when I had to restart the server.

I have tried by upgrading the iStorage Server to 2.20 from 2.08, things were fine too, until restart of Windows, but since I continued to have the problems, I uninstalled the iStorage Server completely and installed a fresh copy of iStorage server... the problems with drives failing and failing to failover continues to baffle. All the Windows updates have been applied too. I know I am writing a lot but I want to give you all the details I have...

However, I do love the SnapShot feature, an excellent backup recovery tool!!!

Just a few suggestions, why don't you allow multiple server management from single UI? Currently, if you try to manage the remote server for iStorage configurations/verifications, either you have to RDP to that Server or you can connect with the same user interface, but you get logged out of the existing server, so it allows management of only one from the same Pane.

Secondly, how can you tell if the two drives are sync'd? and how do you know which one is out of sync, so that I can use the most current as Base synch target, since the other target will lose it's data after sync is complete? Currenty all I can see is "Enabled/Disabled/Failingover" for target and "Running/Failed" for apps. How do we resync? should be delete the existing app and recreate the new one?

Thanks a bunch...
Mohammad Khan

[Charles [PM]]

Thank you very much for your such detailed description.
Here are my suggestions:
1, “Enable multiple initiators with full access” option must be selected on both targets.
2, “Manual Initialization” can be only used for new created image disk.
3, Please confirm you system disk have enough space to store temporary data when partner server fail.
Application status:
Running, indicate that application is running in a good state.
Fail, indicate that the application point to the partner server was fail.
Failing over, indicate that the partner server just started to work and synchronize data to the server.
We have started to test according to your description, tomorrow I can give you the result, we’d also like to provide a document for configuring iStorage Server HA for SQL Server clustering in this a few days.
Your suggestions:
1, Multiple server management, yes it is very useful and we’d like to add this feature in future version, because in our design, it is just like Windows Computer Management Console, so we need some time to implement.
2, Data Re-Synchronize, when one partner server from failing to working, the application on another server will do a Incrementally synchronization, that only copy modified data. If you want to do a full data synchronization, you can create a new application, but you can cancel it after data synchronize (Although this method is not good enough), thank you very much for such a valuable suggestion, we will continue improving in future version.

[Charles [PM]]

Hi Mohammad Khan,

I can't see any step for configuring MSCS, how did you configure the shared storage? The iSCSI disks should be added to cluster resource, before setup SQL cluster, do you have followed this article to setup a windows clustering?
http://www.kernsafe.com/tech/iStorage-S ... yper-V.pdf

[mkhan]

Hi Charles,

Yes, indeed. I consulted the clustering document you have referenced to, before setting up any clustering, both Windows MSCS and MSSQL clustering. Thanks.

[mkhan]

Hello Charles,

I have re-done everything all over for the 4th time. It seems to work fine for now. I have failed-over sql on both servers successfully, even after server reboots.

However, there is a serious bug with disk full synchronization and I think this might have been THE ISSUE causing havoc on my clusters. I have isolated this issue, I setup a new clean disk on both servers, as normally required, connected with iSCSI initiator from both servers and then added some test data. I created the High Availability Failover application, and fully synch'd it. There were four total iSCSI connections, two from each server (one local and one remote, plus the mirror, as required for failover redundancy.

However, after about 75% of disk sync, which was so far going successfully, I noticed a message on the mirror disk (second server) to format the disk, which I ignored and when disk was about 90% done, both disks lost formatting and showed "un-initialized" message in the Server Manager snap-in. The iStorage Server continued as normal sync without any error message and even completed the sync without any problem. However, the disk remained in the failed state and needed to be initialized and reformatted.

Please look into this chaotic bug. Thank you.

-M. Khan

[mkhan]

Charles, I tested the sync issue again by removing all iSCSI initiator connections, as per your recommendation, and it went fine this time around. I was then able to logon all my iSCSI connections and access the data. However, I am not 100 percent certain if both HA mirrors are synched or not. How do I confirm that through iStorage server?

So, I think, as you mentioned in your support email, sync'ing while connected causes corruptions. So, I would suggest that you prevent sync option in iStorage Server if there are any existing connections to the disk and give a message to the user to first log off and then do the sync by creating a new application. I think that will relieve many corruption related issues.

Thank you.