Sunday 18 June 2017

Problems with sharing data in China? Use Azure way to fix this

Table of Contents

  • Introduction
  • High-level Requirement
  • Solution
  • Some performance matrix from my environment

Introduction

The main problems while sharing data in china from rest of world is “The Great Firewall of China”
I’m sure that all of us are aware of the word so called” The Great firewall of china”
Let me put this into more technical term. Chinese government has its own network security/firewall so called “The great Firewall of China” to filter all the data traffic coming over the internet.
Well it’s good for cyber security but it causes many problems such as high latency and slowness of network speed, less privacy, limited reliability and so on.
There are many solutions available in the market to share files in China location. But I want to share my experience on using Azure way to do that.
How it works: Azure has one of the Data center in East Asia Hong-Kong.
Why we mentioned Hong-Kong? Read the below write up from Wiki
The special administrative regions (SAR) are one type of provincial-level administrative divisions of China directly under Central People's Government, which enjoys the highest degree of autonomy, and no or less interference by either Central Government or the Chinese Communist Party.
So let’s take an advantage of international law.

High-level Requirement


We had requirement where approximately 1 TB data to be moved to the China location from EU Data 
center (it was Sweden in my case)
Below are the high level business requirement from the customer.
  1. Data should be moved to China DC location in a faster way
  2. Data should travel in secure manner and encrypted way.
  3. Solution should be cost effective.
My team tried to move the data from EU DC to China DC over the VPN/internet. But unfortunately it didn’t work due to the issues mentioned above. It took me about 25 hrs to move 200 gb of data and then I realized that data got corrupted.  My team tried multiple times with the same results.

Solution

Azure has datacentres in west Europe ( Netherlands ) and East Asia (Hong-Kong) and I used them for data movement.
Tools We used:
  1. One Azure storage account in Azure DC in West Europe (Netherlands)
  2. One Azure storage account in Azure DC in East Asia (Hong-Kong)
Azure tool(Azcopy): AzCopy is a Windows command-line utility designed for high-performance copying of data to and from Azure Storage.
This activity was done in 3 phases mentioned below.  
  1. Customer had good connectivity between their EU DC and Azure DC(Netherlands) so in the first phase all data files were moved to Azure west Europe (Netherlands) using existing Express Router/ VPN connection. I used Azure storage account to store all the data in a one place.
  2. Then data files were moved from Azure storage (Azure Europe) to one of the Azure storage account available in Hong-Kong region. (Microsoft has private tunnel for data to travel within Azure DC(s) worldwide so data which is being transferred from any datacentre to any datacentre will never pass-through the public internet. Hence data will be more secured and encrypted format while movement)
  3. In the final phase I download the data from Azure storage (Hong-Kong) to the destination.

Some performance matrix from my environment


Around 1 TB data is moved with 100% accuracy rate and it took about 14 hrs for entire activity

EU DC File server à Azure Europe FS:  3 hrs
Azure Europe storage  à Azure HK storage FS: 7 hrs.
Azure HK FS àBranch site 4 hrs.
From 300 gb files


Hope the solution will be informative for you.

Br,
Digvijay

Thursday 2 February 2017

Windows Azure Pack: Subscription usages info through PowerShell

Introduction:

Windows Azure pack is a great tool for your Private cloud. It brings the ability to run Azure services inside your own environment. It gives ability to use subscription based access and control like Azure public cloud does.
Since WAP uses plan and subscription based concept for private cloud resource. It’s very important to keep a track of usages or each subscription.
 This Article will help you to generate a nice report about the usage of each subscription.  It’ll also give useful information’s such as co-admin details, subscription status, sync status, creation date etc.
So let’s go it into the detail and see how we do that.
 
Prereqs.
Import MgmtSvcAdmin PowerShell module. Follow this url if you don’t have module installed
(https://technet.microsoft.com/en-in/library/dn282110.aspx )
Access to the WAP admin API
Address of WAP Admin API (https://ServerName:30004)
At least 1 plan/subscription configured in WAP for trial purpose

PowerShell commends:

you can download contents from here.  


  $AdminUri = 'https://ServerName  :30004'
   $AuthSiteUri = 'https://ServerName  :30072'
   $ClientRealm = 'http://azureservices/AdminSite  '
  
  $token = Get-MgmtSvcToken -Type Windows -AuthenticationSite $AuthSiteUri -ClientRealm $ClientRealm -DisableCertificateValidation
   $allSubscription = Get-MgmtSvcSubscription -Token $token -AdminUri $AdminUri -DisableCertificateValidation
$a = "<style>"
$a = $a + "BODY{}"
$a = $a + "TABLE{border-collapse: collapse;width: 100%;}"
$a = $a + "TH{border-width: 1px;padding: 0px;border-style: solid;border-color: #ddd;padding: 10px;text-align: left;}"
$a = $a + "TD{border-width: 1px;padding: 0px;border-style: solid;border-color: #ddd;padding: 10px;text-align: left;}"
$a = $a + "table tr:nth-child(odd) {background-color: #f2f2f2;}"
$a = $a + "TABLE tr td table{width: 50%;}"
$a = $a + "TABLE tr td table th{border: 1px solid #ddd;}"
$a = $a + "TABLE tr td table td{border: 1px solid #ddd;}"
$a = $a + "</style>"
 
 
$body = "<H2>Windows Azure Pack Subscription Information</H2><table><colgroup><col/><col/><col/></colgroup><tr><th>Subscription Name</th><th>Subscription ID</th><th>Status</th><th>Created Date</th><th>Admin</th><th>Sync Status</th></tr>"
 
foreach($Subscription in $allSubscription){
    $SubUsage = Get-MgmtSvcSubscriptionUsage -SubscriptionId $Subscription.SubscriptionID -AdminUri $AdminURI -Token $token -DisableCertificateValidation
    $SubUsage.RetrievedSuccessfully
 
    $body = $body + "<tr><td>"+$Subscription.SubscriptionName+"</td><td>"+$Subscription.SubscriptionID+"</td><td>"+$Subscription.State+"</td><td>"+$Subscription.Created+"</td><td>"+$Subscription.CoAdminNames+"</td><td>"+$SubUsage.RetrievedSuccessfully+"</td></tr>"
    $body = $body + "<tr><td colspan=""6"">Usage Information<br><br><table><tr><th>Service Type</th><th>Usage</th><th>Total Limit</th><th>Unit</th></tr>"
 
    $SubUsageList =  $SubUsage.Usages
    foreach($Usage in $SubUsageList){
        $body = $body + "<tr><td>"+$Usage.DisplayName+"</td><td>"+$Usage.CurrentValue+"</td><td>"+$Usage.Limit+"</td><td>"+$Usage.UnitDisplayName+"</td></tr>"
    }
    $body = $body + "</table></td></tr>"
}
$body = $body + "</table>"
 
ConvertTo-HTML -head $a -Body $body | Out-File "$env:USerProfile\desktop\WAP-Sub-Usages.html"
  
 
Invoke-Expression "$env:USerProfile\desktop\WAP-Sub-Usages.html"



Output

I’ll now explain more into technical detail of it.
Addresses and URI of require components.
$AdminUri = 'https://AdminAPIServer  :30004'
$AuthSiteUri = 'https://WindowsAuthSiteServer  :30072'


Note: You may have above components installed in different server in distributed environment.
You can use below command to see the address.
Get-MgmtSvcEndpoint | Format-Table -Property Name,Address


Generate a token:

The Get-MgmtSvcToken creates an identity token. Tokens are used by several of the Windows Azure Pack for Windows Server cmdlets. You can create a token and store it in a variable for use with other cmdlets. Find WAP subscription and take each for taking detailed information.

$token = Get-MgmtSvcToken -Type Windows -AuthenticationSite $AuthSiteUri -ClientRealm $ClientRealm -DisableCertificateValidation


I’m not expert in html code so I took help from my buddy "Tapan Gupta". However I am using html as an output for making it presentable. Hence you need listed codes.

You can modify it as per your requirement.

$a = "<style>"
$a = $a + "BODY{}"
$a = $a + "TABLE{border-collapse: collapse;width: 100%;}"
$a = $a + "TH{border-width: 1px;padding: 0px;border-style: solid;border-color: #ddd;padding: 10px;text-align: left;}"
$a = $a + "TD{border-width: 1px;padding: 0px;border-style: solid;border-color: #ddd;padding: 10px;text-align: left;}"
$a = $a + "table tr:nth-child(odd) {background-color: #f2f2f2;}"
$a = $a + "TABLE tr td table{width: 50%;}"
$a = $a + "TABLE tr td table th{border: 1px solid #ddd;}"
$a = $a + "TABLE tr td table td{border: 1px solid #ddd;}"
$a = $a + "</style>"
 
$body = "<H2>Subscription Information</H2><table><colgroup><col/><col/><col/></colgroup><tr><th>Subscription Name</th><th>Subscription ID</th><th>Status</th><th>Created Date</th><th>Admin</th><th>Sync Status</th></tr>"


Now it’s time for us to take each subscription one by one and have detailed information.
I’m using foreach loop to collect usages information’s.

foreach($Subscription in $allSubscription){
    $SubUsage = Get-MgmtSvcSubscriptionUsage -SubscriptionId $Subscription.SubscriptionID -AdminUri $AdminURI -Token $token -DisableCertificateValidation
    $SubAdmin = ""
    if($Subscription.CoAdminNames.Count -gt 0){
        $SubAdmin = $Subscription.CoAdminNames.Item(0)
    }
 
    $body = $body + "<tr><td>"+$Subscription.SubscriptionName+"</td><td>"+$Subscription.SubscriptionID+"</td><td>"+$Subscription.State+"</td><td>"+$Subscription.Created+"</td><td>"+$SubAdmin+"</td><td>"+$SubUsage.RetrievedSuccessfully+"</td></tr>"
    $body = $body + "<tr><td colspan=""6"">Usage Information<br><br><table><tr><th>Service Type</th><th>Usage</th><th>Total Limit</th><th>Unit</th></tr>"
 
    $SubUsageList =  $SubUsage.Usages
    foreach($Usage in $SubUsageList){
        $body = $body + "<tr><td>"+$Usage.DisplayName+"</td><td>"+$Usage.CurrentValue+"</td><td>"+$Usage.Limit+"</td><td>"+$Usage.UnitDisplayName+"</td></tr>"
    }
    $body = $body + "</table></td></tr>"
}
$body = $body + "</table>"



Final step is to convert output data and feed this informations to the html page which we created earlier.

ConvertTo-HTML -head $a -Body $body | Out-File "$env:USerProfile\desktop\WAP-Sub-Usages.html"
 Invoke-Expression "$env:USerProfile\desktop\WAP-Sub-Usages.html"


This script will list out all subscription and give the informations back in html format.


I hope this post will be informative.


Thanks

Digvijay Singh

Sunday 4 September 2016

SCVMM 2012 R2 & Hyper-V: Storage Migration in Microsoft Private Cloud

SCVMM 2012 R2 & Hyper-V: Storage Migration in Microsoft Private Cloud

This is a white paper that my buddy Aayush Bhatt and I wrote on Storage Migration using System Center 2012 R2 Virtual Machine Manager & Hyper-V on Windows Server 2012 R2 in Microsoft Private Cloud

Table of Contents

  • Applies to
  • Introduction
  • Storage Migration Plan
  • Storage LUN Migration Process
  • Stand-alone Virtual Machine Storage Migration
  • Clustered/ Shared drive Virtual Machine Storage Migration
  • Boot LUN Migration Process
  • Quorum Disk Migration Process

Applies to

  1. System Center 2012 R2 Virtual Machine Manager
  2. Hyper-V on Windows Server 2012 R2

Introduction

In order to migrate the physical storage underneath the Fabric and Tenant environments of a Microsoft System Center 2012 Private Cloud, we need to consider several components of the Storage layer i.e. Boot LUNs, Storage LUNs, Quorum disk witness, Cluster Shared Volumes (CSV) etc.

Storage Migration Plan

Following is the high level overview of the Storage Migration plan.
           Procedural Activities
 1Formulation of Migration strategy.
 2Formulation of Project plan with timelines and responsibilities.
 3Formulation of Rollback strategy (each for Boot LUNs, Storage LUNs etc.)
 4Formulation of Performance testing strategy for VMs and Hyper-V hosts.
 5Workshop among all respective teams for process agreement.
 6Informing respective Application team(s).
 7Initiation of a change request in ITSM.

 Technical Activities
 8Updating all the Hyper-V hosts with latest patches.
 9Conversion of new storage to LUNs.
Storage LUN10Attaching LUNs to host clusters.
11Conversion of new storage to CSVs.
12Performing Storage migration of test VMs.
13Performance testing of test VMs after storage migration.
14Testing Live Migration.
15Migration of Tenant and Fabric VMs to the new storage.
16Performance testing.
17Removal of Old Storage
Boot LUN  
18Attaching new Boot LUN's to Hyper-V hosts.
19Shutting down host and migrating boot configuration and Boot LUN.
20Removing old Boot LUN.
21Performance testing of Hyper-V hosts.
Quorum Disk  
22Attaching new quorum disk to Host cluster(s).
23Adding new disk as a cluster quorum disk.
24Removing old quorum disk.
 25Verification and validation of the environment.
26Troubleshooting issues (if any)
27Sign-off of the activity

Storage LUN Migration Process

Storage LUNs are used to save virtual machine hard disk files (.VHD and .VHDX), virtual machine configuration data files and snapshots etc. Typically, storage LUNs are usually converted to Cluster Shared Volumes (CSV), which in turn are shared by Hyper-V hosts in the cluster.
The process that happens behind the scenes while performing VM storage migration using SCVMM is as follows.
  1. A new VHD or VHDX file is created in the specified destination location (storage migration works with both VHD and VHDX).
  2. The VM continues to both read and write to the source VHD, but new write operations are now mirrored to the destination disk.
  3. All data is copied from the source disk to the destination disk in a single-pass copy operation. Writes continue to be mirrored to both disks during this copy operation, and uncopied blocks on the source disk that have been updated through a mirrored write are not recopied.
  4. When the copy operation is finished, the VM switches to using the destination disk.
  5. Once the VM is successfully using the destination disk, the source disk is deleted and the storage migration is finished. If any errors occur, the VM can fall back to using the source disk.

Using Microsoft System Center 2012 R2 Virtual Machine Manager with Hyper-V on Windows Server 2012 R2 provides you the functionality to live migrate the storage of a stand-alone virtual machine without any downtime. In case of Clustered VMs, or any virtual machines which have shared virtual hard disks, a downtime is required. The details of both the scenarios are described below.

Stand-alone Virtual Machine Storage Migration

Stand-alone virtual machines are those VMs which have their own VHDs and are neither clustered, nor have a shared VHD. These machine can be migrated without any downtime using the straight forward migration approach of SCVMM. Its process is as follows:
  1. Perform pre-requisites on new storage.
  2. Clean-up the environment by removing unused virtual machines, virtual hard disks, snap-shots etc.
  3. Migrate the storage of a stand-alone virtual machine in VMM console. Right click and select 'Migrate Storage'.


4. Select the new CSV, where you want to save the VM data, then review the settings and select 
5. Validate the migration process by checking job status in VMM.
6. Initiate performance testing of the migrated VM using the performance metrics and test mentioned in       https://msdn.microsoft.com/en-us/library/cc768535(v=bts.10).aspx
7. Once the storage of all the VMs are migrated, go to ‘Failover Cluster Manager’, select the Cluster Volume in the available disks. Right click and select ‘Take   Offline’. It will put the CSV to offline mode from available.


8. Once the status of a disk shows as offline, right click and select the option as ‘Remove from Cluster Shared Volumes’. This step will remove this CSV.


 Clustered/ Shared drive Virtual Machine Storage Migration

Second scenario includes virtual machines which are either clustered on Hyper-V hosts or have virtual hard disk attached to it which is shared among two or more virtual machines. Migration of such kind of virtual machines requires downtime as the shared drives needs to be removed before migrating the VM storage. Its process is as follows:
  1. Stop Cluster service and shut down virtual machine(s).
  2.  Go to virtual machine properties and remove sharing in the 'Advanced Properties' of the shared virtual hard disks

3. Remove shared virtual hard disks from the properties of the VM   (Note: Kindly make a note of the order in which the shared VHDs were added)


4. Migrate virtual machine storage using the process mentioned above
5. Manually move the shared virtual hard disks to the new storage location.
6. Once the storage of all the nodes (VMs) of the cluster are successfully migrated, re-add the shared VHDs to the VMs by changing its properties.
7. Change the sharing permissions in the shared virtual hard disks.
8. Turn on the virtual machines and make sure the cluster service is running.
9. Check the failover by pausing each node at a time.

Boot LUN Migration Process

 Boot LUNs are attached to the physical Hyper-V hosts and contains booting configuration data of the host. Unlike storage LUNs, these are not shared and each host has one Boot LUN attached to it.
Migrating the Boot LUN of a Hyper-V hosts requires the host to be shut down. In case of a Hyper-V host clustering environment, we can live migrate all the virtual machines from one host to another host(s) before shutting it down, thus preventing any downtime. But if, Hyper-V hosts are not clustered, then the virtual machines running on top of the Hyper-V hosts needs to be shut down in order to perform the activity.
The overview of the Boot LUN migration process is as follows  
  1. Go to SCVMM console, select the desired node in the Hyper-V cluster. Right click and select 'Start Maintenance Mode'. This will stop the cluster service on that host.

2. Now as the maintenance mode is on, all the virtual machines on the Hyper-V host will automatically start live migrating to the other hosts in the cluster.
3. Once all the workloads are moved, the cluster service will show as 'Paused'.


 4. Confirm the status in 'Job History'


5. Shut down the Hyper-V host.
6. Perform Boot LUN migration by transferring all the boot configuration to the new storage.
7. Remove the old storage and attach new Boot LUN to the host.
8. Start the host and stop the maintenance mode. Performing this step will activate the cluster service on the host.


9. Once the cluster service is started, right click on the Hyper-V cluster and select 'Optimize Hosts'. It will optimize the workloads across all the hosts in the cluster. Make sure that the live migration is happening seamlessly.




10.Perform same steps for the remaining hosts in the cluster.

 Quorum Disk Migration Process

  The quorum configuration in a failover cluster determines the number of failures that the cluster can sustain. If an additional failure occurs, the cluster must stop running. A Quorum is a Disk Witness which contains a copy of the cluster configuration.
  The process of migrating Quorum disk is as follows:
  1.  Add new quorum disk to the cluster storage.
  2.  Open 'Failover Cluster Manager', Right click and select 'More Actions' then select 'Configure Cluster Quorum Settings'.

3. Proceed through the wizard by selecting 'Use Default Quorum Configuration'. Verify the details and make sure that the new quorum dis is selected as the Disk Witness.


4. Close the wizard.
5. In 'Failover Cluster Manager', go to Disks under Storage and make sure that the new Quorum disk is reflecting there as 'Online' and 'Disk Witness in Quorum'.




6. Monitor the cluster by checking event logs etc. 


Br,
Digvijay