Tuesday, March 17, 2015

VMware Transport Modes: Best practices and troubleshooting

 

Encountered issues restoring via “SAN” failed during the boot up, error symptom is “operating system not found”, tried again with “NBD”, it works.

http://www.symantec.com/business/support/index?page=content&id=tech183072

 

Issue

A VMware Backup Host can access Virtual Machine data from datastores using four different methods – SAN, LAN(NBD), HotAdd, NBDSSL. These methods are referred to as VMware Transport modes. This article talks about these transport modes, the best practices around them and troubleshooting tips for some commonly seen errors related to transport modes in NetBackup and Backup Exec.

Solution

For both Backup and Restore operations, NetBackup and Backup Exec allow choosing any of the four transport modes or a combination of these. If a combination of the transport modes is given, NetBackup and Backup Exec will try all of them one by one until gaining successful access to the data of the Virtual Machine.

Details on each of the transport modes

1. SAN:  The SAN transport mode requires the VMware Backup Host to reside on a physical machine with access to Fibre Channel or iSCSI SAN containing the virtual disks to be accessed. This is an efficient data path because no data needs to be transferred through the production ESX/ESXi host.

In this mode, vStorage APIs obtain information from the vCenter server or ESX/ESXi host about the layout of VMFS LUNs and, using this information, reads data directly from the SAN or iSCSI LUN where the VMDK resides.

Best practices around SAN:

  • For using SAN, make sure that datastore LUNs are accessible to the VMware Backup Host.
  • SAN transport is usually the best choice for backups when running on a physical VMware Backup Host. However, it is disabled inside virtual machines, so use HotAdd instead on a virtual VMware Backup Host.
  • SAN transport is not always the best choice for restores. It offers the best performance on thick disks, but the worst performance on thin disks, because of the way vStorage APIs work. For thin disk restore, LAN(NBD) is faster.
  • For SAN restores, disk size should be a multiple of the underlying VMFS block size, otherwise the write to the last fraction of a disk will fail. For example, if virtual disk has a 1MB block size and the datastore is 16.3MB large, the last 0.3MB will not get written. The workaround in this case would be to use NBD for restores of such Virtual Machines.
  • When using SAN transport or hot-add mode on a Windows Server 2008/2008 R2 VMware Backup Host, make sure to set:
    • SAN policy to onlineAll
    • SAN disk as read-only, except during restores

2. LAN (NBD): In this mode, the ESX/ESXi host reads data from storage and sends it across a network to the VMware Backup Host. As its name implies, this transport mode is not LAN‐free, unlike SAN transport.

LAN transport offers the following advantages:

  • The ESX/ESXi host can use any storage device, including local storage or NAS.
  • The VMware Backup server could be a virtual machine, so you can use a resource pool and scheduling capabilities of VMware vSphere to minimize the performance impact of backup. For example, you can put the VMware Backup Host in a different resource pool than the production ESX/ESXi hosts, with lower priority for backup.
  • If the ESX/ESXi host and VMware Backup Host are on a private network, you can use unencrypted data transfer,which is faster and consumes fewer resources than NBDSSL. If you need to protect sensitive information, you have the option of transferring virtual machine data in an encrypted form using NBDSSL.

Best Practices when using LAN:

  • Since the data in this case is read by ESX/ESXi server from storage and then sent to VMware Backup Host, It is must to have network connectivity between ESX/ESXi server and VMware Backup Host. If the VMware Backup Host has connectivity to vCenter server but not the ESX/ESXi server- snapshots will succeed but vmdk read/write operations will fail.
  • The VMware Backup Host will need the ability to connect to TCP port 902 on ESX/ESXi hosts while using NBD/NBDSSL for backup/restores.
  • VMware uses Network File Copy (NFC) protocol to read VMDK using NBD transport mode. You need one NFC connection for each VMDK file being backed up. There is a limit on the number of NFC connections that can be made per ESX/vCenter server. These limits differ in different versions of vSphere - please refer to the NetBackup for VMware Admin Guide(linked below) for these limits. Backup/Restore operations using NBD might hang if this limit is reached.

3. HotAdd: When running VMware Backup Host on a Virtual Machine, vStorage APIs can take advantage of the SCSI Hot-add capability of the ESX/ESXi server to attach the VMDKs of a Virtual Machine being backed up to the VMware Backup Host. This is referred to as HotAdd transport mode.

Running the VMware Backup server on a virtual machine has two advantages: it is easy to move a virtual machine around and it can also back up local storage without using the LAN, although this incurs more overhead on the physical ESX/ESXi host than when using SAN transport mode.

Best practices when using HotAdd:

  • HotAdd works only with virtual machines with SCSI disks and is not supported for backing up virtual machines with IDE disks.
  • A single SCSI controller can have a maximum of 15 disks attached. To run multiple concurrent jobs totally more than 15 disks it is necessary to add more SCSI controllers to the HotAdd host.  The maximum number of 4 SCSI controllers can be added to a HotAdd host, so a total of 60 devices are supported at the maximum.
  • HotAdd requires the VMware Backup Host to have access to datastores where the Virtual Machine being backed up resides. This essentially means:
    • ESX where the VMware backup host is running should have access to datastores where the Virtual Machine being backed up resides. 
    • Both the VMware backup host and Virtual Machine being backed up should be under the same datacenter.
  • HotAdd cannot be used if the VMFS block size of the datastore containing the virtual machine folder for the target virtual machine does not match the VMFS block size of the datastore containing the VMware Backup Host virtual machine. For example, if you back up virtual disk on a datastore with 1MB blocks, the VMware Backup Host must also be on a datastore with 1MB blocks.
  • Restores using HotAdd on a Windows Server 2008 proxy require setting the SAN policy toonlineAll
  • If you are converting a physical machine to a virtual machine with the intention of using hottadd to back up the virtual machine, do not use IDE controllers for any disks that are used during the conversion process.
  • The VMware Backup Host will need the ability to connect to TCP port 902 on ESX/ESXi hosts while using HotAdd for backup/restores.

4. NBDSSL:  NBDSSL is the same as NBD, except that NBDSSL uses SSL to encrypt all data passed over the TCP/IP connection.

Troubleshooting for some common transport mode related failures

Backups/Restores failing with status 6 or status 13 or status 11 with following indication in Activity monitor might indicate that there is some issue with transport modes:-

  • ERR - Error opening the snapshot disks using given transport mode: Status 23 indicates that there was some problem in accessing the vmdk using given transport mode.
    Here are some tips on handling this kind of error:
    • If you are using NBD, make sure the VMware Backup Host has connectivity to ESX server hosting the virtual machine.
    • If you are using SAN, please make sure that the datastore LUNs are accessible to VMware Backup Host.
    • If you are using HotAdd, please make sure that your backup host is Virtual Machine and following conditions are satisfied:
      • The VM should not contain IDE disks.
      • Ensure that there are sufficient SCSI controllers attached on the Backup Host VM.
      • The Backup Host VM has access to datastores where VM being backed up resides.
      • The Backup Host VM and VM being backed up should be under the same datacenter.
      • If the previous backup failed, it might have left some disks of the backup VM attached to Backup Host. These disks need to be manually removed before attempting the next backup.
    • If a non-default port for vCenter is in use, then that port needs to be defined while adding vCenter credentials to NetBackup or Backup Exec.
    • If using NBD/NBDSSL/HotAdd, please make sure the VMware Backup Host is able to communicate to port 902 of ESX server hosting the VM.
  • file read failed indicates that there might be problem in reading the VMDK using the given transport mode. 
  • file write failed indicates that there might be some problem in writing to the VMDK using the given transport mode.
    • If using SAN for restores, please make sure datastore LUNs are accessible to the VMware Backup Host and in an online state.
    • If using HotAdd for restore, please make sure that SAN policy on the Backup Host is set to OnlineAll.
    • If using SAN for restore, make sure that the size of VMDK is multiple of datastore block size.  Otherwise, the write of the last block will fail.  In this case, a workaround would be to use NBD for restore.
    • Please make sure that the you assign necessary privileges to the user configured in NetBackup or Backup Exec to log on to vSphere.

29 comments:

  1. I definitely loved every little bit of it. I have you bookmarked your site to check out the new stuff you post. Europa-Road Kft.

    ReplyDelete
  2. So luck to come across your excellent blog. Your blog brings me a great deal of fun.. Good luck with the site. híres nemzetközi szállítmányozás

    ReplyDelete
  3. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful. Beol

    ReplyDelete
  4. Tips on how to structure essays and have feel more manageable, as well as be sure that you receive in all the relevant points so you never miss out on any marks! trucking

    ReplyDelete
  5. If comparing pavers to concrete, consider that concrete can crack, breakup and stain, becoming unsightly and lowering the value of your home. Pavers, Asphalt Paving Amarillo TX

    ReplyDelete
  6. When you have chosen to move your vehicle you can contact these trucking organizations for your assistance, they have their own site and given you the complementary number and mailing office where you can get in touch with them and ask your questions. https://europa-road.eu/hu/transportation.php

    ReplyDelete
  7. It is a great website.. The Design looks very good.. Keep working like that!. parcels to Pakistan

    ReplyDelete
  8. Tracking and tracing is mentioned in one sentence. But there is a small difference between the two; one is focused on the process in which you need to intervene (tracing) the other is needed to manage the first click here

    ReplyDelete
  9. transportation terminals, just as dumpsites, just to specify a couple. The transportation administrations move products between the offices. europa-road.eu

    ReplyDelete
  10. If you are looking for the cheapest way to ship a car or other type of vehicle overseas, chances are you will want Ro Ro shipping. This tends to be a less expensive option for vehicle shipping than container shipping. how long is shipping from china to uk

    ReplyDelete
  11. This is such a great resource that you are providing and you give it away for free. I love seeing blog that understand the value. Im glad to have found this post as its such an interesting one! I am always on the lookout for quality posts and articles so i suppose im lucky to have found this! I hope you will be adding more in the future... İstanbul evden eve nakliyat

    ReplyDelete
  12. I might want to thank you for the endeavors you have made recorded as a hard copy this article. I am trusting a similar best work from you later on also. Truth be told your exploratory writing capacities has motivated me to begin my very own BlogEngine blog now. Extremely the blogging is spreading its wings quickly. Your review is a fine case of it. discord push to talk not working

    ReplyDelete
  13. Eşya taşıma konusunda üstün dikkat ve gayretlerimiz, hizmeti kalite standartları üzerine çıkarmak için oldukça hassasız. evden eve nakliyat

    ReplyDelete
  14. Bu kapsamda eşya depolarımız ile sunduğumuz hizmet dikkate alındığında İstanbul’un en iyi eşya depolama firmaları arasında en çok ilgi gören firmalar başında gelmekteyiz. istanbul eşya depolama

    ReplyDelete
  15. Bu yazımızda eşya deposu hizmetleri hakkında, geniş kapsamlı bilgiler yer almaktadır. eşya depolama

    ReplyDelete
  16. Evden eve nakliyat konusunda her şeyi rahat bir şekilde elde etmek için sizler de iletişimde yer alabilirsiniz.
    maltepe evden eve nakliyat

    ReplyDelete
  17. Şu an en çok zahmet isteyen taşınma durumlarından bir tanesi olan evden eve nakliyat için hem şehir içi hem de şehirler arası olan hizmetler yer alıyor.
    parça eşya taşıma istanbul

    ReplyDelete
  18. You guardians do an astounding web diary, and have some unfathomable substance. Continue doing extraordinary. white house market link

    ReplyDelete
  19. It was the time of numerous developments that in a general sense changed the idea of transport. Correspondence, telecommunication, the steam motor and the steam transport sped worldwide transport every which way.https://europa-road.eu/hu/foldmunkagep-szallitas-debrecen.php

    ReplyDelete
  20. When traveling, whether for business or for pleasure, you can still enjoy the benefits of feng shui. You might have even noticed that you enjoy your travel more (or less) when you travel to certain places that are in one direction. Maybe you find that you enjoy travel equally well, but more so to some places than others. For some, Asia might hold greater attraction, and for others, it might be Europe. Here are some ideas to enhance your travel feng shui -- including how to tell which is the right "shui" for you! ship tracker

    ReplyDelete
  21. This comment has been removed by the author.

    ReplyDelete
  22. Thanks for the blog loaded with so many information. Stopping by your blog helped me to get what I was looking for. Europa-Road túlméretes szállítás

    ReplyDelete
  23. Specialists go about as mediators between you, the customer, and the transporter or engine transporter, and are indeed the essential wellspring of cargo for most of engine transporters in the business. 中港搬屋

    ReplyDelete
  24. Wonderful blog! I found it while surfing around on Yahoo News. Do you have any suggestions on how to get listed in Yahoo News? I’ve been trying for a while but I never seem to get there! Appreciate it. airport

    ReplyDelete
  25. This is certainly in the process an exceptionally superior ad people very seriously suffered browsing thru. It is actually hardly every single day you'll find associated risk to visit a little something. transportation

    ReplyDelete
  26. Once more, in any LTL shipment, the cargo will be gotten on to the pallet or prepared forklift with one more taking care of unit before season of pickup.Shipping from China to Us

    ReplyDelete
  27. The training of creatures and the innovation of the sled gave way to early creature transport administrations. https://shuttlesky.in/

    ReplyDelete
  28. The organization you select ought to have a group of experts who cooperate for the best result for your association. https://europa-road.eu/hu/fuvarozas.php

    ReplyDelete
  29. scales removals are very easier nowadays. Move your properties from one place to another with Waage Umzüge (Scale removals) just at a cheap price.

    ReplyDelete