Slow File Copy On Read From Target: Large Files

[stevea]

Hello, I'm wondering if you can reproduce this problem in the test lab:

Prerequisites: 2 Machines running Windows Server 2008, 2008 R2, Win 7, or Win XP; GB LAN on both ends.
I can reproduce this problem with any of the OS's and using either a PhysicalDisk or Disk Image File target (I've been unsuccessful getting Partition Target to work - don't worry about it here, different question).

1. Create an iSCSI target using Physical Pass-through Disk.
2. Attach to it from initiator on another machine, initialize and format the disk NTFS.
3. Create a 9gb VHD file on the initiator boot drive or some other available disk (not the iSCSI target disk).
4. Copy the large VHD file to the iSCSI target drive. Mine copies in between 1:30 and 2:00 minutes, very nice.
5. Now the problem: Copy the file back from the iSCSI target drive to another available disk or folder on the initiator. Mine crawls at around 1mb/sec. - takes forever. I have gotten this down to a best time of 8:00 minutes +/- by doing all kinds of network tweaking, but no faster.
6. Now a simple comparison test:
7. Disconnect the initiator from the target and create a Windows share on the target machine using the disk containing the copied 9gb VHD (or connect to the administrative '$' share).
8. Connect to the share on the initiator machine and copy the file back using SMB. Mine will copy back and forth either direction in about 2:00 minutes +/- either way when no iSCSI involved, and machine memory caches cleared.

Thanks in advance for your help,
Steve

[Olivia (staff)]

Thanks for your feedbacks, we will begin to start test this issue on next Monday.

[stevea]

Hello: I'm wondering if you have any news on this. I'm ready to implement, but this is a bit of a problem. On a data disk I have set up, copying 860gb of data to the target only took a few hours, but the backups are taking about 18 hours -- not good.

I continue to work on this here, and actually found that disabling jumbo frames evened out the up/down transfers a bit. With leaving all the other TCP and network adapter tweaks in place, I lost speed on the write to target but gained some on the read from target. Now the read is only about 4 - 5 times slower than the write instead of about 10 times. I have Intel Pro/1000 nics in all three machines for the dedicated iSCSI network, and followed Intel white paper suggestions for optimizing the driver settings for iSCSI as well.

Any forward progress would be helpful -- thanks in advance.

[Olivia (staff)]

Hi, the jumbo frames may not the main reason,
we have gone a series of tests, but we have not found the exact reason of this problem, in some computers the copy large files our of iSCSI disk is slow, but the read speed is more fast while iStorage Server running on Windows Server 2003 than running on Windows 2008, a little strange.
We will continue and need about another two days for completing full-test.

[stevea]

Hi, and thank you for working on this problem. Here's a random thought regarding Server 2003 copy speed faster than Server 2008: what if you tried a test on 2008 after disabling UAC (and a reboot) to see if it makes a difference. I had a case last week where one of my cluster servers got access denied errors attempting to open a shared iSCSI cluster disk after it was taken offline and later brought back online. After some amount of unsuccessful troubleshooting, I disabled UAC, and the problem disappeared. After that experience, I disabled UAC on the SAN and both cluster servers -- interestingly, it seems to me that they have noticibly faster user response than before.

[stevea]

Hello: Re: WebGuy's question about cache in this post: viewtopic.php?f=5&t=129, I notice that in the Locals.xml file, there is a "cache=" tag with a value of "0" for each target. Can this setting be used to cache data on the target computer side RAM? My setup also sits there without the free memory being used -- maybe has something to do with the slowness on large file reads.

[Olivia (staff)]

The cache field is used for previous version, currently this field is reserved, and the cache is managed by iStorage Serve automatically. Set a large cache may bring more performance but may cause less reliability of data (e.g. when the power lost, may cause data which haven’t written to hard disk lost).
We have done the test on a various OS, the performance is not good enough for large files transmission, especially running under UAC.
We will improve it in the future version, considering both performance and reliability.