You are here

MSDN Blogs

Subscribe to MSDN Blogs feed
from ideas to solutions
Updated: 1 hour 53 min ago

Large SQL Server database backup on an Azure VM and archiving

Thu, 02/26/2015 - 14:06
Applicable Versions

SQL Server 2014 and lower versions.

Background

Azure Virtual Machines (VM) allows the capability to install SQL Server and lift/shift an application or data warehouse to the cloud. The largest disk size Azure currently allows is 1tb (or more accurately 1023GB). Larger VMs such as A9 and D14 allows 16 and 32 of these disks to be attached respectively. For a large SQL Server database on an Azure VM, backing up the data can have some challenges. Some challenges for a backup can be scheduled VM reboot, unplanned reboot or VM migration because of faulty OS which are sometimes unavoidable but not very common occurrences and thus out of scope for this blog. We will focus on two particular challenges for backing up large SQL Server database on an Azure VM – size and speed.

Size related challenges

SQL Server allows the option to backup to disk and URL. Currently, URL backup to Azure Storage is limited to 1 file only and also that file cannot exceed more than 1TB. This is a good option for backing up small databases.

Backup to disk would certainly have to be smaller than 1TB per disk unless the disks are configured as a storage pool. In a case where backup will not fit on the disks (16 disks on A9 and 32 disks on D14) we would recommend looking at SQL Shep’s blog. That blog describes how to create DFS file storage in Azure which is not a small undertaking. Thus, planning for backup space has to be one of the factors when moving large SQL Server database in Azure. If the data files fit on 16 disks but the backup will not you may want to think about D14 VMs which allows attaching up to 32x1TB disks.

Another size related challenge comes from the requirement of archiving data to Azure Blob Storage. Currently, each blob cannot exceed 200GB in Azure Storage. Another option of archiving backup is to use Azure File Service. At the time of the writing of this blog, Azure File Service is still under preview. Azure File Service allow you to mount a 5TB drive through SMB protocol as a network drive on an Azure VM. Azure File Service also has the limit where 1 file cannot be greater than 1TB. But multiple Azure Files can be connected to one VM and used to copy/archive backups. These file share(s) can also be mounted on many VMs and can be accessed using a REST APIs.

Speed related challenges

When backing up SQL Server on-premises we usually have large storage pool or SAN connected servers. Backing up using one backup file on-premises is not uncommon and works fine. In Azure VMs we can also use a storage pool to create a backup with one backup file. But since space on VHDs are limited we need to plan for archiving the data onto Azure Blob Storages. Moving one large backup file to Azure Storage can become impossible because of the size restriction. Azure file Service allow storing of larger files but having one large backup file can prove challenging when copying.

Recommended Approach

Our recommendation would be to backup to multiple files for even a storage pool rather than one large file. In this case let’s assume there are ten drives mapped, H through Q. The backup command would look something like the following:

BACKUP DATABASE tpch_10tb
TO DISK = 'H:\backup\tpch_10tb_01.bak',
DISK = 'I:\backup\tpch_10tb_02.bak',
DISK = 'J:\backup\tpch_10tb_03.bak',
DISK = 'K:\backup\tpch_10tb_04.bak',
DISK = 'L:\backup\tpch_10tb_05.bak',
DISK = 'M:\backup\tpch_10tb_06.bak',
DISK = 'N:\backup\tpch_10tb_07.bak',
DISK = 'O:\backup\tpch_10tb_08.bak',
DISK = 'P:\backup\tpch_10tb_09.bak',
DISK = 'Q:\backup\tpch_10tb_10.bak'
WITH COMPRESSION,
STATS = 5;
GO

SQL Server allows compression when creating a backup. In certain customer cases we have seem a 10TB database compressed into 2TB or smaller in backup files. When Transparent Data Encryption (TDE) is used in the database the backup of that database does not compress. As such an important recommendation is to compress the backup to keep the size of the backup smaller unless TDE is used.

The reasons behind having many compressed backup files vs. one large file are the various size limits discussed earlier and the practicality of data movement in the cloud. Let's assume the storage pool or drives are rather full after the backup, we need to move the backup somewhere else to make space for the next backup. In this case Azure Blob Storage is a good candidate to copy the backup and free up the disks. When a storage account is created on azure, the user has option to choose if the disk will be locally redundant (LRS) or geo redundant storage (GRS). We will leave it up to your business decision as to what type of redundancy to choose.

As 200GB is the current limit of a single blob, you have to calculate to keep the files under that limit. If the database has multiple files or filegroups, a quick way to check would be to backup a file or filegroup to see the size and estimate the backup size from there. SQL Server does a good job at keeping the backup files the same size when multiple files are used.

Let’s assume our backup is divided into 10 files that doesn’t exceed 200GB limit for each file. We can now write 10 copy command to parallelize the copy operation to archive the backup to Azure Storage. We will also need to make sure all files are copied correctly or else the backup is useless when it comes time to restore the data for whatever reason.

We have run the following backup scenarios and captured timing:

Database

Size

Database Compression

Timing (hh:mm:ss)

TPCH_10GB

10GB

Uncompressed

00:02:18

TPCH_1TB

1TB

Compressed

02:29:00

TPCH_10TB

10TB

Compressed

21:16:00

In our case, we have observed that when all the tables in a database is compressed it takes a lot less time to backup that database. In all the above tests we used a MAXTRANSFERSIZE of 4MB. Changing BUFFERCOUNT option didn’t make much difference. The backup command looks like the following for 10TB database.

BACKUP DATABASE tpch_10tb
TO DISK = 'H:\backup\tpch_10tb_01.bak',
DISK = 'I:\backup\tpch_10tb_02.bak',
DISK = 'J:\backup\tpch_10tb_03.bak',
DISK = 'K:\backup\tpch_10tb_04.bak',
DISK = 'L:\backup\tpch_10tb_05.bak',
DISK = 'M:\backup\tpch_10tb_06.bak',
DISK = 'N:\backup\tpch_10tb_07.bak',
DISK = 'O:\backup\tpch_10tb_08.bak',
DISK = 'P:\backup\tpch_10tb_09.bak',
DISK = 'Q:\backup\tpch_10tb_10.bak'
WITH COMPRESSION,
MAXTRANSFERSIZE= 4194304,
STATS = 5;
GO

Archiving

Azcopy does a good job at copying data in Azure. The following Azcopy command will copy the backup files from the VM to a storage of your choice.

This first command will copy all the files one after the other:

AzCopy /Source:X:\tpchbackup\ /Dest:https://tpch1tbbackup.blob.core.windows.net/backup/ /Destkey:<destination storage key> /Pattern:*.bak

This command will copy tpch_1tb_01.bak file:

AzCopy /Source:X:\tpchbackup\ /Dest:https://tpch1tbbackup.blob.core.windows.net/backup/ /Destkey:<destination storage key> /Pattern:*_01.bak

In the latter case, we can execute multiple of these commands to finish copying faster. We would recommend adding a date component to the directory structure to make the files unique so the older backup are not overwritten and also many versions of the backup can reside side by side but also can be uniquely identified. In that case the copy command will look as following:

AzCopy /Source:X:\tpchbackup\ /Dest:https://tpch1tbbackup.blob.core.windows.net/backup/20150101 /Destkey:<destination storage key> /Pattern:*.bak

Azure File Service is another option to copy the backups for archival purposes. Use the following commands to mount a network drive to archive the backup:

1. Create a storage account in Windows Azure PowerShell

New-AzureStorageAccount -StorageAccountName "tpch1tbbackup" -Location "West US"

2. Create a storage context

$storageAccountName = "tpch1tbbackup"

$storageKey = Get-AzureStorageKey $storageAccountName | %{$_.Primary}

$context = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageKey

3. Create a share

New-AzureStorageShare -Name backup -Context $context

4. Attach the share to your Azure VM

net use x: \\tpch1tbbackup.file.core.windows.net\backup /u:$storageAccountName $storageKey

5. Xcopy backup to x:\ to offload the backup on the mounted drive

xcopy D:\backup\*.* X:\tpchbackup\*.*

This begs the question why not just use the UNC path of the mounted drive to do the backup. In which the answer is, it's not possible at this time. Notice that the file is mounted using a storage account name and storage key. If you try to backup to the UNC path of the drive directly SQL Server will throw the following error:

Operating system error 1326(The user name or password is incorrect.).

If you are using multiple Azure File shares for archiving backup you should also have some kind of accounting (metadata) that tells you where each backup files of a backup set reside.

Lastly, you should follow the same procedure for backup and restore that you follow for your on-premises database. As an example, if your business procedure is to check the newly created backup using ‘restore verifyonly’, follow that before you move your data for archiving.

Security descriptors, part 1: what they are, in simple words

Thu, 02/26/2015 - 13:23

I've been dealing with the security descriptors recently. I've figured out for myself, what different concept around them mean, and wrote a few helper functions to deal with them in PowerShell. Now I want to do a brain-dump of it all, starting with the concepts, and the PowerShell examples will be forthcoming later.

Overview

The security descriptor is the way to control the access to an object. The object may be whatever you want, and the access may be whatever you want as long as someone defines the meaning of various bits in the mask for this object type (more on this later). You can even make the ACLs for your own objects, as long as you also provide the code that checks this access. But there also are a few object types (files, registry entries, synchronization primitives, ETW log sessions), for which the Windows OS checks the access.

A security descriptor consists of the information about the owner and group of the object, the system ACL (access control list) and discretionary ACL. There also are other small bits and pieces like control flags that carry the extra information but they're not particularly important for the general understanding.

The meaning of the owner and group is entirely up to the particular object type. For the files in the filesystem, I think the owner is the one allowed to change the security descriptor for this file, and is used to compute the quotas, and I'm not sure what can the group do. Maybe the group is also used for some quotas.

The System ACL can be used for auditing: recording when someone accessed something, or even killing the process that touched something that it should not have. I haven't looked at it much, so I won't talk any more about it.

The Discretionary ACL is what controls the access. In the future, when I talk about the ACLs, I'll mean the discretionary ACLs. The system ACLs are similar, just they're used for a different purpose.In general, the ACL in Windows is the ACL-as-usual, nothing particularly special, if you've seen the ACLs before, you know the idea. Basically, it contains a number of Access Control Entries (ACE), that are examined sequentially in the order they're placed in the ACL (so the order matters, more on that later). Each ACE contains the identity of a user or group (principal, in Windows terms), the type of the entry (allowing or denying the access) and the bitmask describing what exact access it allows or denies (I'm used to calling them "permissions" in the Unix way, I like this word, and I'll be using it). The ACL gets read sequentially, each entry is checked whether the principal in it matches the identity of the current user or one of its groups, and if the match is found, the bitmask of permissions in the entry is applied to the bitmap of the permissions the user is asking. If anything asked is found in a Deny entry, the access error is returned. If anything asked is found in an Allow entry, that permission is considered satisfied. If all the asked permissions are satisfied, the access gets granted and the walk through the ACL stops. Otherwise the walk continues.

The ACLs can be inherited. For example, instead of specifying an ACL on each file, its possible to specify an ACL on the directory and have it inherited by all the files in it.

Command-line

The two main commands to know for manipulation of security descriptors on the files are takeown.exe and icacls.exe.

Principals

A principal is a user or group, they're in a common namespace, which is very convenient. A principal generally has a name and a binary identifier. The binary identifier is called SID (Security IDentifier).  Naturally, the identifier gets used in all the internal representations.

In Unix, the user (or group) namespace is flat and the identifier is just a number. Which is historically simple but creates many problems when combining the users from different sources, such as the local users and the users from the network, requiring the rules like "the users with the id less than N are the local users, and greater than that are the network users".

In Windows, the identifiers are more flexible. They're organized into domains (such as the network domains, the local computer domain, and a disturbing number of domains for the pre-defined "Well Known" users), and the SID contains the parts for the domain and the principal as such within the domain. The format is also extensible, and in result the binary format having the variable length.

There is also a human-readable numeric form of SID. It consists of a bunch of numbers separated by a dash and starting with "S-1-". I suppose, "S" stands for "SID". "1" means that this is the version 1 of the SID format, and the rest are the bits and pieces from that variable-length numeric format.

The names of the principals also include the domain part, they're generally written as "domain\principal". The domain named the same as the current computer contains the principals defined locally on the current computer (or "." can be used for the same purpose). For all I can tell, there is no concept of "user on THAT computer", there are no domains denoting the other computers, but only the domains describing the shared networked users. It's also possible to give the name simply as "principal", and when it gets resolved to a SID, the resolved is smart enough to look for it in the available domains and figure out the domain part by itself (of course, if the same principal name is used in multiple domains, the result might be different from what you've meant).

It's also possible to have a SID for which there isn't any translation to a name. It's really an undefined principal but the code that deals only with SIDs doesn't care: it will process it just like any other SID.

There is also the third kind of names for some principals. Some principals are pre-defined in the windows OS to have a hardcoded meaning. They are the "well-known" principals. Many of these well-known principals also have a special two-letter representation of the SID. They do have the normal SID starting with "S-1-" as well, and the normal binary representation, but also a short name that can be used in any place where a "S-1-" string form of the SID can be used, and the SID parser knows how to parse these strings. There are places that require a text form of a SID, not the user name. The idea here is that it's easier to remember and write say "BA" than "S-1-5-32-544" for the group "BUILTIN\Administrators".

Descriptor Formats

The  descriptors can be represented in multiple formats, depending on the needs.

There are two native binary formats, one for the in-memory use, one serialized and usable for the off-line storage. Since the descriptor contains multiple elements (all these principals and ACLs, that further contain ACEs in them that also contain principals), the natural and convenient in-memory format for them consists of a tree connected by pointers. The serialized format is essentially the same but all these bits and pieces are collected in a contiguous chunk of memory, and the pointers are replaced by the relative offsets in that chunk.

If you see the type SECURITY_DESCRIPTOR, it's the header for the in-memory format with pointers, the type SECURITY_DESCRIPTOR_RELATIVE is the serialized version with the relative offsets. The first 4 bytes of these structures are actually common, and contain the version of the format and the control flags. The control flag SE_SELF_RELATIVE means that the data is in the serialized format, so the code that handles it can determine the format automatically.

There is also a compact human-readable format named SDDL (Security Descriptor Definition Language, or some such). It's basically the same information in a string, using the string SID format for the principals. Looks like this:

O:BAG:BAD:(A;;0x201;;;SY)(A;;0x200;;;BA)(A;;0x200;;;BO)(A;;0x200;;;SU)(A;;0x200;;;WR)(A;;0xffff;;;SY)(A;;0xff7f;;;BA)(A;;0xffff;;;S-1-5-80-880578595-1860270145-482643319-2788375705-1540778122)

"O:BA" and "G:BA" mean that the owner and group are both the "BUILTIN\Administrators". "D:" is the start of the Discretionary ACL followed by ACES in parentheses. Each ACE shows its type ("A" for "Allow"), the bitmask of permissions, and the principal's SID in the text form. There are more fields in the ACE for such things as the inheritance flags, and the permissions can also be written symbolically with the two-letter combinations, like:

(A;CI;GRGX;;;BU)

I won't go into the details, they can be found in the SDDL definition. It's somewhat readable but not hugely so, and I'll show the PowerShell functions to print it in the format closer to the one used by icacls.exe.

The .NET provides the classes that encapsulate the descriptors, which can also be used from PowerShell. Now, I've been searching for examples, and people use all kinds of strange ways of dealing with permissions in PowerShell, including the WMI objects. There is no need to go that strange, the .NET classes provide a more straightforward way. Unfortunately, it's not very straightforward, so I'll show in the future installments a few PowerShell functions I wrote to make things easier.

The namespace for all the security stuff is System.Security.AccessControl, and the main two classes are RawSecurityDescriptor and CommonSecurityDescriptor (both inheriting from GenericSecurityDescriptor). The difference between them is that the Common version provides a more high-level interface (though still not very easy to use) but expects the ACLs to be in a canonical form (se e more later); while the Raw class just gives the access to a descriptor and lets you deal with it in any way you want, including the creation or repair of the non-canonical forms.

The canonical form

The canonical form means that the entries in the ACLs are ordered in a certain way: all the Deny entries must go before all the Allow entries.

But why? Because in this order the further ordering of the entries for different principals doesn't matter. Remember, in general the ACEs are processed sequentially, and the order matters. There might be multiple entries applicable for the current user that contain the overlapping permissions masks. If there are two entries, one of them Allow and another one Deny, with the same permission bit set, and the Allow goes before Deny, the access will be granted; if Deny goes before Allow, the access will be denied. These entries won't necessarily be for the same principal, they might be for the different groups to which the current user belongs. The creative ordering of the entries allows to create some complicated ACLs but figuring out what these complicated ACLs do is a pain. the canonical order makes sure that the ACLs are easy to understand.

Things can get even more complicated when the inheritance of the security descriptors is taken into account. In the canonical form, it's simple: just take all the inherited ACEs and all the explicit ACEs, and put them into one ACL, and order them to make all the Deny entries go first. The meaning of that ACL will be the same as of the original set of ACLs. The ordering within the Deny and Allow sections of the ACL won't matter.

(To be continued...)

How Office 365 does SPF checks for customer-to-customer mail

Thu, 02/26/2015 - 13:04

There may be some confusion about how Office 365, or Exchange Online Protection (EOP), does SPF checks on incoming email - especially in the case when Customer A sends email to Customer B and both parties are EOP customers. This applies to the case when the sending email account is from a separate mail server, not from a fully hosted mailbox within EOP.

Setting up connectors

Customers who use EOP for sending outbound email typically do it by setting up an Inbound On-premise Connector (that is, to send outbound email through EOP, you need to set up an inbound on-premise connector). For example, suppose CustomerA.com uses the IP 1.2.3.4 to relay outbound email through EOP, they might set up a connector like the following:

You would enter in your connecting IPs and any associated sender domains. I personally recommend adding actual domains and not putting wildcards like *.com, or at least *.customerA.com, because leaving it as a wildcard makes it more susceptible to phishing attacks.

Mail routing and SPF checks in EOP

When customer A sends an email to customer B through the IP 1.2.3.4, the email is relayed out through EOP, relayed back in to EOP, and then routed to customer B (Customer B may be fully hosted, or have an on-premise mailer, it doesn’t matter which). So, the mail loops out and around, like below:

Customer A needs to publish the following TXT record:

customerA.com. 3600 IN TXT "v=spf1 include:spf.protection.outlook.com ip4:1.2.3.4 -all"

  • Customer A must add include:spf.protection.outlook.com because when the email goes out to the Internet (e.g., Gmail), or to another EOP customer, EOP or the 3rd party on the Internet (e.g., Gmail) looks to see if the connecting IP is in customerA.com’s SPF record. The connecting IP is EOP’s outbound IP.

  • Customer A also must add ip4:<on-premise IPs> into their SPF record because EOP does an SPF check at point (1) above when it scans the email when sending the message outbound. It then does a second SPF check at point (3) when the mail is relayed back into EOP.

    Typically, the on-premise IPs in the SPF record are the same ones that you would put into an inbound on-premise connector.

If you do the above, everything will work properly with regards to SPF.


Attribution of outbound email and SPF checks

EOP’s SPF checking for customer-to-customer mail is more complicated than the above.

If you relay outbound email from 1.2.3.4 and the sending domain (e.g., customerA.com) matches the domains you have set up in the connector, this is called an outbound attributed message and the message is attributed to your organization. When sending email from Customer A to Customer B, then at point (3) EOP will use the IP in EOP’s SPF record:

However, if you relay outbound email from 1.2.3.4 and the sending domain (e.g., random.com) does not match any domains you have set up in the connector, this is called an outbound unattributed message and the message is attributed to a default organization – not your own. When sending email from Customer A to Customer B, then at point (3) EOP will use the original sending IP address and not the connecting IP:

This is a change that was introduced in December 2014. It is done to combat phishing and spoofing.

This is most noticeable in two places:

  1. When you send email from a subdomain that is not in your list of sender domains.

    For example, the inbound on-premise connector lists the domain customerA.com, but you send email from bounce.customerA.com, autoreply.customerA.com, and so forth. These will be unattributed messages and if sending to another EOP customer, the SPF check will use the original sending IP address. However, 3rd parties (e.g., Gmail) will use EOP’s relaying IP address.

  2. When your on-premise mail servers sends bounces, auto-replies, or other messages with an empty 5321.MailFrom, <>.

    An empty SMTP MAIL FROM will always be an unattributed message. See the next section to see how EOP handles this.


How EOP handles messages with an empty SMTP MAIL FROM <>

If a message has an empty mail from, then there is no domain upon which to perform an SPF check. EOP then falls back to the EHLO/HELO string and checks that instead. So, ff the message comes from an external 3rd party such as GoDaddy, EOP will use GoDaddy’s connecting IP address and the EHLO/HELO string used to in the SMTP transaction. Other 3rd party receivers do the same thing to EOP – if you send an email with <> to a 3rd party, the 3rd party uses EOP’s EHLO/HELO string in their SPF check (EOP has SPF records set up on all of its EHLO/HELO strings).

However, if the message with <> comes from another EOP customer, the message is unattributed. Therefore, EOP uses the original connecting IP address and falls back to the original connecting HELO/EHLO string, that is, your server’s HELO/EHLO string. Therefore, you should ensure that you have SPF records set up on your server’s HELO/EHLO string, too.


This behavior change is coming in Feb or March 2015.

Summary

To summarize everything, here’s what customers need to do:

  1. If connecting email from an on-premise mail server (which you will if you have an on-premise environment or are a hybrid customer), every IP in an inbound on-premise connector should also be in the SPF record for each domain you send mail from. The SPF record must also have include:spf.protection.outlook.com.

  2. The HELO/EHLO string of the on-premise mail server should also have an SPF record but only needs to include the IPs in the inbound on-premise connector. It is not necessary to have include:spf.protection.outlook.com. But to make things easier, you could simply copy/paste the SPF record from the domains you send mail from.

If you don’t do the above, there’s a good chance that you, or the senders you send to, will get false positives.

Related articles

Sales and Licensing Excellence Webcasts Featuring Real-Life Scenarios and Updates

Thu, 02/26/2015 - 12:31

As a product or license seller to customers in the public sector, it’s important for you to be current on the latest pricing updates, to anticipate real-life customer scenarios, and to know how to share about Microsoft business opportunities. Our Sales and Licensing Excellence webcasts for our public sector partners can help you acquire this information and have more effectual customer conversations. The webcasts range from 100- to 300-level and cover a variety of Microsoft products and technologies. In our live webcasts, you’ll also have the opportunity to ask questions to our experts.

You can view the current schedule and access recorded sessions anytime at http://aka.ms/lspwebcasts. We’ll also share them in the Training Recommendations Yammer group, include the listings on the Hot Sheet training schedule, and post updates on Twitter.

Schedule of Public Sector On Demand Webcasts:
SQL for Government Licensing Review and Best Practices (Level 200)
Microsoft Dynamics CRM Online Government (Level 100)
Microsoft Azure Government (Level 100-200)
Microsoft Azure Government Cloud Solutions Webcast and Executive Keynote (Level 100) 
Azure Government Licensing (Level 200-300)

New! Take the Sales Excellence Challenge

In addition to the sales and licensing updates you receive on our webcast series, preparing for and taking Sales Specialist assessments can round out your knowledge and improve your ability to talk with customers about Microsoft technologies.

Partners who both attend one of the webcasts posted on this blog post (live or on-demand) and pass an eligible Sales Specialist assessment between February 1 and April 30 will be entered* into a drawing for one of three Surface Pro 3 tablets. You can view offer details and links to the assessments after February 1 at http://aka.ms/saleschallenge.

*No purchase necessary. Open only to legal residents of the 50 U.S. + DC 18+ with an active MPN ID. Game ends 4/30/15. Official rules.

Submit WPC Awards Now

Thu, 02/26/2015 - 12:30

The 2015 Partner of the Year Awards program is your opportunity to compete for awards that showcase the solutions you’ve built on Microsoft technologies to benefit our mutual customers.  Receiving a Partner of the Year Award positions your company for new business opportunities, generates positive press coverage and can lead to even greater market recognition.  Here are the Microsoft Dynamics specific awards but feel free to see what other awards are applicable for your organization:  Cloud Customer Relationship Management; Customer Relationship Management (CRM); Enterprise Resource Planning (ERP) and Microsoft Dynamics – Industry.  Reminder:  Tool closes on April 14, 2015 at 11:59 PM PST. 

https://mspartner.microsoft.com/en/us/Pages/WPC/awards.aspx

Microsoft Dynamics AX and Microsoft Dynamics CRM Certification Needed NOW

Thu, 02/26/2015 - 12:30

Confirm your organization is ready for March 1, 2015 by completing the latest certification exams. 

You can view that information by going into PartnerSource Business Center – Exam Results. 

https://mbs.microsoft.com/partnersource/northamerica/readiness-training/readiness-training-news/MSDexamreqchanges

Microsoft Dynamics 2015 Partner Recognition Program

Thu, 02/26/2015 - 12:29

Make sure your organization reads the criteria for the FY15 President’s Club and Inner Circle criteria.  It would be great to see your organization listed. 

https://mbs.microsoft.com/partnersource/northamerica/partner-essentials/recognition/MSDY_FY15_Partner_Recognition

Microsoft Dynamics CRM 2015 Virtual Foundations Course - April 2015

Thu, 02/26/2015 - 12:28

Join us to ramp up on the new features and functionality of Microsoft Dynamics CRM 2015. This virtual instructor-led course will focuses on both product knowledge and certification as key objectives.

Key components will include self-paced lecture sessions and instructor led demos with Q&A sessions.

Course will consist of self-directed weekly learning with corresponding weekly instructor-led 3.5 hour sessions with class hours of 6:30pm – 10:00pm CST.

Students taking this course will need to have access to the new Dynamics Learning Portal.  To learn more about DLP, please see here.

Course Agenda:

Session 1
Introduction and Getting Started with Sales Management (80546)
Session 2
Customer Service in Microsoft Dynamics CRM (80545)
Session 3
Microsoft Dynamics CRM Customization and Configuration (80542)
Session 4
Microsoft Dynamics CRM Customization and Configuration (80542)
Session 5
Microsoft Dynamics CRM Customization and Configuration (80542)
Session 6
Microsoft Dynamics CRM Installation & Deployment (80539)
Session 7
Consolidated Test Prep for Certification Exams
Target Audience: Competitive, Industry, Retool, or Young Professionals with NO Microsoft Dynamics CRM experience. 

Every Thursday - $1750
April 23rd to June 4th
6:30pm-10:00pm CST
REGISTER NOW

Microsoft Dynamics CRM Spring Readiness Blitz!

Thu, 02/26/2015 - 12:27

Social listening! Customer care! Marketing! It’s all coming in the Microsoft Dynamics CRM Spring release. Please join us as we overview the upcoming releases and show how you and your customers can help create amazing customer experiences with Microsoft Dynamics CRM.

We are offering both a sales and a technical track to help you get the content applicable for your role.  These three-hour online events will include market positioning, licensing, overviews and demos of Microsoft Dynamics CRM.

Incorporating both recorded content and live chat-based Q&A with Microsoft Subject Matter Experts, the Microsoft Dynamics CRM Spring Readiness Blitz will be offered at two different times to provide attendance options convenient for your schedule.

Don’t miss this opportunity to learn more about the Microsoft Dynamics CRM Spring release!

Recommended Audience

Microsoft Dynamics CRM partner marketers, sales specialists, pre-sales specialists, functional consultants, technical consultants, and developers.

Sales Track:
Option 1 – March 5 | 7:00 AM – 10:00 AM PST
Option 2 – March 5 | 7:00 PM – 10:00 PM PST

Technical Track:
Option 1 – March 5 | 7:00 AM – 10:00 AM PST
Option 2 – March 5 | 7:00 PM – 10:00 PM PST

Register

Using Stream Analytics with Event Hubs

Thu, 02/26/2015 - 12:08

This post will show you how to use the new Stream Analytics service with Azure Event Hubs to process streaming event data in near real time.

Background

Stream Analytics is a new service that enables near real time complex event processing over streaming data.  Combining Stream Analytics with Azure Event Hubs enables near real time processing of millions of events per second.  This enables you to do things such as augment stream data with reference data and output to storage (or even output to another Azure Event Hub for additional processing). 

This post is a demonstration using the walkthrough document Get started using Azure Stream Analytics

The scenario we will use in this post builds upon previous posts that I have written so far:

Event Source Collect (this post) Filter / Analyze / Aggregate Event Sink (this post)

The data each device sends to the Event Hub is a simple JSON message. 

Metric Event
  1. {'DeviceId':2,'Temperature':72}

We are simulating potentially millions of devices sending telemetry data indicating the device ID and a temperature.  The data is sent to an Event Hub, and processed in parallel by 3 different processes:

This is incredibly powerful.  Instead of chaining processes together, making one process wait on the completion of a previous process, we can scale our solution by having many independent processors work with the data in parallel.

This post is focuses on using Stream Analytics to process streaming event data from Azure Event Hubs at scale.  Where previous posts showed a lot of code, this post will show how to process the Event Hub data with zero code.

Setting Things Up

Before we start using Stream Analytics, we will need an Event Hub, something that sends data to the Event Hub, and we need a database to store the data in.  Creating the Event Hub is very easy using the Azure Management Portal.

Provide a name and region and a namespace (creating one if you haven’t already).

Provide a partition count and message retention period (8 is the default, 16 is recommended, 32 max).

Now we need a database to store the data in. 

Provide the database name, region, and server.

Provide login details.

Once created, we can manage the database.  I choose to use Visual Studio’s integrated capabilities.

We are prompted to set a firewall rule to allow the local computer to access it.

Provide login credentials, then right-click on the database node and choose New Query.

Our query will create two tables.

Code Snippet
  1. CREATE TABLE [dbo].[PassthroughReadings] (
  2.     [DeviceId]      BIGINT NULL,
  3.         [Temperature] BIGINTNULL
  4. );
  5.  
  6. GO
  7. CREATE CLUSTERED INDEX [PassthroughReadings]
  8.     ON [dbo].[PassthroughReadings]([DeviceId] ASC);
  9. GO
  10.  
  11. CREATE TABLE [dbo].[AvgReadings] (
  12.     [WinStartTime]   DATETIME2 (6) NULL,
  13.     [WinEndTime]     DATETIME2 (6) NULL,
  14.     [DeviceId]      BIGINT NULL,
  15.         [AvgTemperature] FLOAT (53)NULL,
  16.     [EventCount] BIGINT null
  17. );
  18.  
  19. GO
  20. CREATE CLUSTERED INDEX [AvgReadings]
  21.     ON [dbo].[AvgReadings]([DeviceId] ASC);

We now have two tables that are ready to receive output from our Stream Analytics jobs.

Using Stream Analytics

The Stream Analytics service is still in preview at the time of this writing.  You’ll need to sign up for the preview at the Preview Features page.

Once provisioned, create a new Stream Analytics job named DeviceTemperature.

Once created, the Quick Start page lists the operations in order:

We start by adding an input.  The input for this job will be a stream of data from Event Hubs.

The next page asks for an input alias (this is important as we’ll use this name in our SQL query in just a moment).  You also provide the information for the existing Event Hub.  Note that I have used an existing policy called “Manage” because our Stream Analytics job requires Manage permission for the Event Hub. Creating a policy for the Event Hub is simple, although not shown here.

Finally, our data is serialized as UTF-8 encoded JSON.

Stream Analytics will then make sure it can connect to the source.

The next step in the checklist is to create a query.  We will create a simple passthrough query to select from our input.  It’s important to note that the entity we select from, EventHubInput, is the name of the input alias used in the previous step.

Finally, we can create an output that sends data to our new SQL Database table, PassthroughReadings.

Once the job is created, click Start.  You are prompted when to start the job output.

After a few minutes, your job is now in the Processing state.  Start up the device simulator (a sample is shown in the post Device Simulator for Event Hubs).  I simulate 1000 devices, 10 messages, no wait time, and to run only once.

Now go to the SQL database table “PassthroughReadings” and see that there are 10 rows in there now!

Doing Some Analytics

While passing data through to persistent storage is interesting, it would be far more interesting to do something with the data that would be otherwise difficult.  Let’s calculate the average temperature over a 5 second span.  Instead of editing the existing job, let’s create a new job called DeviceTemperatureTumbling.

Again we use the Event Hub as input, naming it EventHubInput.

This time our query will be a bit more sophisticated.  Stream Analytics introduces extensions that make it easy to process time series data, something that would be quite difficult in a relational scheme.

Code Snippet
  1. SELECT
  2.     DateAdd(second,-5,System.TimeStamp) as WinStartTime,
  3.     system.TimeStamp as WinEndTime,
  4.     DeviceId,
  5.     Avg(Temperature) as AvgTemperature,
  6.     Count(*) as EventCount
  7. FROM EventHubInput
  8. GROUP BY TumblingWindow(second, 5), DeviceId

Our output this time is the AvgReadings table.

We start the new job, and then run our device simulator again to send messages to the Event Hub.  This time, instead of running just once, we’ll simulate 20 devices, send 10 messages at a time, run 1000 times, with no pause between iterations.

sender devicereadings 20 10 16 0 1000

If we look at the SQL output for the PassthroughReadings table, we can see that there is a lot more data now per device, and we have multiple readings per device.  The PassthroughReadings table doesn’t have any notion of time, it’s just raw output data.  The AvgReadings table, however, has been augmented with the system time and a tumbling average of temperature within a 5 second window. 

OK, that’s very cool… we can get an average temperature for a 5-second window.  Let’s see what that looks like when we point Excel at it.

Self-Service BI

OK, now that we have data in Azure SQL Database, we can use familiar tools for self-service business intelligence.  In Excel, go to the Data tab and choose From Other Sources / SQL Server.

Provide the server name (something.database.windows.net) and your login credentials that you used when you created the database.

Choose the database and AvgReadings table:

I choose to create a Pivot Chart.

I use the recommended charts, and choose a line with points.

Or we might choose a radar chart.

The very cool thing about this is that once the data is in Event Hubs, we are able to process it, augment it, analyze it, and store in long-term storage, even report on the data… all this with zero code.

For More Information

Get started using Azure Stream Analytics

Use JMS and Azure Event Hubs with Eclipse

Device Simulator for Event Hubs

Scaling Azure Event Hubs Processing with Worker Roles

Different ways to connect Azure to on-premise datacenter

Thu, 02/26/2015 - 11:47

I found this great video (hat tip to Brian H. Prince and Keith Mayer) that quickly summarizes different ways to connect Azure to an on-premise datacenter.  The video is only about 10 minutes long and gives a great high-level view of the different options: http://channel9.msdn.com/Series/Explain/HybridCloud101 

Here’s a quick summary of the options:

1. BizTalk hybrid connections: this is a simple, super-easy option to implement that allows Azure to communicate with on-premise without needing to punch holes in your firewall.  You simply create a new “hybrid connection” in Azure, which creates a package that you install on your on-premise box. 

2. Azure AD application proxy: Azure AD can manage both cloud and on-premise applications.  This is great to use for websites with authentication/authorization.

3. Azure Service Bus Relay: this option is for web services only; the web service with your custom code initiates a connection to Azure so they can communicate.  You get automatic load-balancing as well as no opening firewall ports, etc.

4. Azure Service Bus Queue: a queue on Azure waits for http requests.  Use this when you need queues (brokered messaging). 

5. Point-to-site and site-to-site VPN: in these scenarios, you get a full virtual private network (VPN), more than just listening to a port.

6. Express Route: most complex to set up. Create your own pipe directly to Azure that doesn’t go over the internet.

Environmental Data and Services in the Cloud

Thu, 02/26/2015 - 11:16

There are several publicly available environmental data sources that you can use to build apps, hosted on Microsoft Azure. You can use a £125/$200 Azure free trial that gives you access to just about all services that you may need. Sign up at http://azure.microsoft.com/en-us/pricing/free-trial/

For example:

  • UK Met Office open weather data on Azure Marketplace, including JSON API - http://datamarket.azure.com/dataset/datagovuk/metofficeweatheropendata 
  • Gaugemap flood information and APIs from Shoothill via - http://www.gaugemap.co.uk/ 
    • Gauge Widget – easily embed any gauge easily
      onto another website (Gauge-Widgets automatically update with the latest
      info every hour. For example -  http://kingsbridgecourt.com/home/2015/02/22/ 
    • Borehole data – Shoothill added live borehole (i.e. groundwater level) data to GaugeMap very recently. BTW 50% of UK drinking water comes from the ground (not rivers) and in London, it is 80%. It is also responsible for groundwater flooding (as opposed to river / sea flooding).
    • River flow data – Shoothill has added live river flow data to GaugeMap and this is very relevant to the insurance industry
  • FetchClimate from the Computational Science Lab of Microsoft Research provides ready access to complex geographical information including, but not limited to, climatological information. On accessing the  FetchClimate Azure web service, you simply need to perform four steps to find what you are looking for:
      • Draw the location on the Earth via points or grids (Where?)
      • Specify the data of interest (What?)
      • Set the timeframe, including future predictions, and a combination of averages over—or steps through—years, days, and hours (When?)
      • Fetch and view your results.
    • FetchClimate will choose the best data set for your query, and perform all the necessary regridding in space and time. It will return a best guess, uncertainty, and provenance for your query and display the results on the map for visual exploration. Alternatively, the FetchClimate service can be used directly via a simple API, from within programs written in any .NET language, Python, R or Matlab. See http://research.microsoft.com/en-us/projects/fetchclimate/
    • An example app that uses FetchClimate is called Climatology, available on the Windows Phone and Android Stores.
Predictive models using Azure Machine Learning

It is now possible to use machine learning without being an expert. The Azure Machine Learning service makes it easy to build predictive models, trained on data, and publish them as web services with a REST API. It is ideal for building apps that can bring together data from different sources to produce productive and meaningful predictions. You can use one of the many state-of-the-art machine learning techniques included, or your own in R or Python. More details are at -
http://azure.microsoft.com/en-us/services/machine-learning/

Building mobile apps with the Cloud

With Azure Mobile Services, it’s easy to rapidly build engaging cross-platform and native apps for iOS, Android, Windows or Mac, store app data in the cloud or on-premises, authenticate users, send push notifications, as well as add your custom backend logic in C# or Node.js. See http://azure.microsoft.com/en-us/services/mobile-services/

  • Mobile Services makes configuring user authentication via Facebook, Google, Microsoft, or Twitter accounts as simple as a few clicks. Once users  authenticate, not only can you personalize their experience based on social APIs, but also increase engagement and sharing.
  • Azure can help you easily build apps which interact with customers as they are using your app, so you can deliver fast experiences to delight your users. Mobile Services supports both SignalR and Socket.IO for managing realtime communication with devices.
  • Use offline data sync to build responsive apps. Create robust apps that remain useful when there are network issues, so users can create and modify data even when they are offline. Improve app responsiveness by caching server data locally on the device. With Mobile Services you can easily provide a native sync experience across your iOS, Android, and Windows apps.

For visualisation and dynamic dashboard reports for data and maps, the PowerBI service and Excel 2013 plugin provides easy-to-use capabilities with many applications for environmental apps and services. See http://www.microsoft.com/en-us/powerbi/default.aspx

You can find out how to use Azure with Linux and open-source software at http://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-opensource/

For hackathons you can use a £125 Azure free trial that gives you access to just about all services that you may need. Sign up at http://azure.microsoft.com/en- us/pricing/free-trial/   

For help you can see the documentation at http://azure.microsoft.com/en-us/documentation/

We hope Microsoft Azure can help you build great apps that make use of environmental data.

Join the Windows Azure for Research community and conversation on LinkedIn & Twitter. Use #azureresearch and start the discussion. http://linkd.in/18Z5OL4  and https://twitter.com/azure4research

 

 

Easy diagnostics for your Azure website

Thu, 02/26/2015 - 10:56

I was going a bit of work on an Azure website I have been building.  There is a bit of PHP code that I am using that a DePaul colleague shared with me.  I wanted to easily get access to my weblogs, other diagnostics as well as some debug type information that I was writing to the console. 

For Azure websites there are a couple of easy steps to do to get this information here are are couple of ways to get this information. 

First you’ll need to turn some of these diagnostic tools on.

Turning on diagnostics
  1. First login into the management portal at https://manage.windowsazure.com
  2. Select website and then select the website you want diagnostics for (see below)

3.   Click on the to Configure menu

4. Turn on application diagnostics

5. Turn on site diagnostics

Now for the magic

Let’s say your site is name foo.AzureWebsites.net.  Navigate to foo.scm.AzureWebsites.net.  Note the addition of the scm piece to the URL.  This will take you to the site manager for your site.  Login if needed and check out the tools the SCM provides.

Check out details for each of the following:

1. Your websites environment

2. Debug console (easy access to directory structure including logs)

3. Process explorer (see the running processes)

4. Tools (among other things your log stream)

.NET Core Open Source Update - February 26, 2015

Thu, 02/26/2015 - 09:59

It has been a couple weeks since we last reported on the .NET Core open source project. It's been a very fun time, watching more people get involved in the project and to see progress on a daily basis. It's amazing watching my GitHub news feed. I have to scroll through several page views just to get through the last hour (on a weekday) of corefx, coreclr and roslyn repo activity. Today, we're going to focus on the CoreCLR repo.

In addition to the growing community activity, there have been important product improvements that have been committed. Some of the key changes are coming from the community. That's very impressive. Wow and Thanks.

Work-in-progress

We published the CoreCLR source earlier this month, but did not immediately transition our work-in-progress feature work to be open and viewable. We received community encouragement to adopt a more transparent approach. We are following through on that.

People want to approach building a significant feature with the knowledge that Microsoft is not building the same thing. It would be very frustrating to see a Pull Request (PR) closed (not merged) after putting several hours (or more) into it, due to inadequate coordination upfront. That's the anti-pattern.

There are two parts to that. The .NET Core team will:

  • Publish issues, as both self-assign and up-for-grabs.
  • Do work on a publicly visible branch (either coreclr/corefx or a fork).

Here are a few of the issues that we published using both the self-assign and up-for-grabs patterns. The act of publishing issues enables transparent ownership of issues, which is beneficial for everyone.

Doing feature work in the open means that anyone can:

  • Watch changes as they are made, to understand and give feedback on the approach for a given feature, before it is submitted as a PR.
  • Collaborate as a group on the actual feature work.

Here's a good example of one of the first features that was developed in the open. If you read the PR, you'll see an in-depth publicly-visible conversation, with both community and Microsoft folks. I cannot show you the actual branch anymore, since it has been (naturally) deleted.

You can see that this feature was developed in a central branch - unix_issue177 - on the coreclr repo. We will make use of both central branches and personal forks for feature work. We choose the branch location based on the level of discoverability and engagement that makes sense for a given feature.

You might be wondering how to find out about these feature branches before they show up as PRs. Watch the Issues queue. Feature branches should be advertised within the issues they 'implement'.

Most of you are are not paying attention at quite this level of detail. There are a set of folks who are making a significant set of contributions and for whom transparency matters a lot, for both product direction and code changes. We are making a real effort to cater to that need. Keep the feedback coming, so that we can continue to improve the transparency and communication.

Mac OS X Support

One of the biggest additions since we published the CoreCLR repo is the initial implementation of Mac OS X support. The internal Microsoft team has been focussed on Linux support, making Mac OS X support a great community-led project, in order to bring it up in parallel with Linux.

@kangaroo has been leading the charge on Mac OS X. Thanks! Check out these two PRs if you want to look at the changes: Mac OS X Support and Initial Mac OS X Support. There have been lots of changes since then.

You can see from the community response to the initial support announcement that there are a lot of folks that would like to see .NET Core on Mac OS X. Me, too!

@praeclarum wrote a great set of instructions, Building and Running .NET’s CoreCLR on OS X, to help you follow along at home. Check it out.

You'll quickly see that the Mac OS X experience is still pretty raw. We'll publish an official set of instructions when the experience is a bit further along. For now, please do refer to @praeclarum's instructions.

Linux Support

There is also a lot of great work going into Linux. For example, @mikem8361 is working on getting SOS working on Linux. I already mentioned the stack unwinding feature, based on libunwind.

Like Mac OS X, it is still "early days" for Linux support. That said, we have an enterprising community that likes to try things out. We'll publish official Linux instructions when we've got a good experience in place.

@zerosignal00 was one of the first to try out .NET Core on Linux. There is a lot of enthusiasm for .NET Core support on Linux, too.

Tracking progress on library work for .NET Core

The previous .NET Core Open Source Update attached a spreadsheet that allowed you to get a sense for which APIs we foresee .NET Core will have. To make tracking easier, we've create a new repository on GitHub, called dotnet/corefx-progress.

This repository is designed to allow you to track our progress towards publishing all of .NET Core library source. It's only provided as a temporary mechanism to allow you track the difference between what we've already open sourced and what is yet to come. As a result, we don't intend you to contribute to this repository.

This repository contains the following folders:

  • lib-full. These are the reference assemblies (i.e. assemblies without IL) that represent the entire surface area of all .NET Core libraries.
  • src-full. This is the C# representation of the assemblies in lib-full.
  • src-oss. This is the C# representation of the assemblies that are already open sourced, i.e. are available on GitHub.
  • src-diff. For each assembly, this folder has a Markdown file that contains the unified diff between src-full and src-oss.

In order to understand what hasn't been open sourced what is coming, you should take a look at the .NET Core: Library Open Source Progress.

Alternatively you can clone this repository and use your favorite tool to diff the directories src-full and src-oss.

dotNetConf

Set your alarm clocks for March 18th and 19th, for dotNetConf. The .NET Core team, among many other folks, will be talking and demoing on the live dotNetConf stream. We'll be answering questions that we get during our talk.  Expect to see and hear more about .NET Core on Windows, Linux and Mac.

Gitter Chat

We received three PRs recently - Add a Gitter chat badge to README.md, Add a Gitter chat badge to README.md and Add a Gitter chat badge to README.md - to publicize Gitter chat rooms created by @migueldeicaza. This guy must like to talk about .NET!

Please drop into the Gitter rooms and join the crowd.

Gitter is a chat room service, where the each room is tied to a GitHub repo, quite naturally choosing the topics to discuss. I like using it. It's the integration with GitHub (obviously!) that makes it a pleasure to use. That said, if someone likes a different chat room service, we're happy to publicize it, provided you can create an active community around it.

dotnet.github.io

Check out the dotnet.github.io page that we built for the .NET Foundation. It helps you discover .NET Foundation project repos. Repos are ordered in terms of OSS AwesomenessNetflix and Twitter have similar sites that we shamelessly borrowed ideas from, including the hotness algorithm (which we extended as "awesomeness") from Twitter.

In case you didn't know, Microsoft announced the .NET Foundation last year. We have contributed many components to the Foundation, including .NET Core, Roslyn, ASP.NET and Orleans. Several other companies and individuals have contributed components.

Community Thanks!

I was impressed to see the Roslyn team recognize recent committers to the Roslyn repo. I've heard folks on the team refer to these as sha-cups. That's quite classy.

We haven't sent out any cups for corefx and coreclr just yet (should we?). That said, we're equally appreciative for the level of support that we're seeing on a daily basis, almost 24 hours a day. Thanks to everyone who has contributed to the .NET Core project. If you haven't yet, I encourage you to get involved.

Thanks for reading -- Rich

Rich Lander @richlander@runfaster2000

A break from the past: the birth of Microsoft's new web rendering engine

Thu, 02/26/2015 - 09:00

As we announced last month, Project Spartan will be the new browser across all Windows 10 devices, from phones to tablets, PCs and beyond. You’ll hear about the new browser’s features in the coming months but in this post, we want to tell you more about what motivated us to build a new rendering engine focused on interoperability with other modern browsers ― all in the name of making the Web “just work” for our customers. This new rendering engine was designed with Project Spartan in mind, but will also be available in Internet Explorer on Windows 10 for enterprises and other customers who require legacy extensibility support.

Modernizing IE while not breaking the Web

Internet Explorer has been the Microsoft Web browser for 20 years and has evolved to work with ever-changing Web sites over that time. Since 2009, we’ve been systematically bringing major investments to the browser to modernize each of the browser subsystems: from fast new JavaScript and layout engines, GPU-accelerated rendering and 3D graphics, and multi-threaded touch input to improved F12 developer tools and backwards compatibility with Enterprise Mode.


Big changes in 5 years: Fish IE (GPU rendering), Chakra fastest on SunSpider (JavaScript performance), Touch Effects (multi-touch input), new F12 developer tools, Assassin’s Creed Pirates (WebGL)

As we’ve been rolling out these significant changes with major versions of IE, we also did our best to abide by the mantra of “don’t break the Web.” This is an ambitious goal given that the Web consists of over 44 billion Web sites, so we prioritized compatibility testing on the Web’s top 9000 Web sites globally which accounts for roughly 88% of Web traffic. This represented the sweet spot of where the “head of the Web” becomes the “long tail” – allowing us to focus our testing resources on the most impactful sites. Prior to release, we would ensure a compatibility pass rate greater than previous releases of IE and our competitors on those top 9000 sites.

The long tail matters

And yet, even as we released new and improved versions of IE, we heard complaints about some sites being broken in IE – from family members, co-workers in other parts of Microsoft, or online discussions. As we examined these sites, they were often not on our top 9000 list. They were issues like not being able to complete a reservation online for the barbershop down the street, or not being able to log in to see the schedule for the local kids soccer league.

In Windows 10 planning, we set out to tackle this apparent discrepancy – how could our real-world compatibility be declining when our compatibility testing data shows that we are improving?

Rethinking our assumptions

As we dug in, we uncovered a series of issues that led us to realize that we needed to significantly rethink our approach to compatibility with the Web at large:

  • Legacy vs. modern. While we were pushing ahead with new HTML5 features, we were also expected to preserve compatibility with old versions of IE, particularly for enterprise Web apps. Limited compatibility was provided through document compatibility modes within Trident, however compatibility could not be guaranteed and it provided consistent obstacles towards fixing long-standing IE-specific behaviors. Furthermore, fixing long standing interoperability bugs with other modern browsers could actually break sites who have coded to the IE-specific behavior.
  • CV list. Our compatibility pass rates were dependent on the presence of the compatibility view list. This allowed us to “fix” broken sites by forcing them into old document modes which emulated legacy IE behaviors. However, this approach requires testing and maintenance, and doesn’t scale well beyond the top sites.
  • X-UA-Compatible. Some sites forced an older document mode using the “x-ua-compatible” header. However, rather than using it as a temporary stopgap, they would rely upon that to keep that version of the site working in future versions of IE while they developed an evergreen code path of their site for other modern browsers.
  • Standards focus. Our focus on building new HTML5 features was to comply with Web standards, which in turn should lead to interoperability among browsers. However, interpretations of the standards document could easily vary, leading to real-world interoperability gaps and ultimately more bug fixing for Web developers and more broken sites for customers.

A break from the past

Faced with these issues, we were inspired to have “courage in the face of reality”. We needed a plan to make it easy for Web developers to build compatible sites regardless of which browser they develop first for. We needed a plan which ensured that our customers have a good experience regardless of whether they browse the head or tail of the Web. We needed a plan which gave enterprise customers a highly backward compatible browser regardless of how quickly we pushed forward with modern HTML5 features.

In order to really address these challenges, we realized that we couldn’t just incrementally improve on our previous approach, we needed a break from the past – all without losing the major investments that we had been making since 2009.

A pragmatic approach

The break meant bringing up a new Web rendering engine, free from 20 years of Internet Explorer legacy, which has real-world interoperability with other modern browsers as its primary focus – and thus our rallying cry for Windows 10 became “the Web just works.” This pragmatic viewpoint, which was initially proven out by our work in Windows Phone 8.1 Update, meant that Web standards would continue to be important but should function in the background to drive real-world interoperability between browsers.

This interoperability-focused approach brought the obvious question of adopting an existing open-source rendering engine such as WebKit. While there were some advantages, upon further investigation it was not the right path forward for two important reasons. First, the Web is built on the principle of multiple independent, yet interoperable implementations of Web standards and we felt it was important to counter movement towards a monoculture on the Web. Second, given the engineering effort required, we found that we could deliver an interoperability focused engine to customers significantly faster if we started from our own engine (especially if unshackled from legacy compatibility concerns), rather than building up a new browser around an open-source engine. We will continue to look at open source and shared source models where it makes sense and look forward to sharing more details in coming posts.

A new Web rendering engine is born

As detailed in Jacob Rossi’s article for Smashing Magazine, the new engine began as a fork of MSHTML.dll but has since diverged very quickly. By making this split, we were able to keep the major subsystem investments made over the last several years, while allowing us to remove document modes and other legacy IE behaviors from the new engine. On the other hand, our legacy engine (MSHTML.dll) can remain largely unchanged (outside of security and other high priority fixes) to help guarantee legacy compatibility for our enterprise customers. We also built up capabilities to switch between the legacy and new rendering engines seamlessly.

A clean break also necessitates a new user-agent string to ensure that no IE-specific code was being sent. This built upon a long browser history of using whatever tokens are necessary to get the desired content from servers. Although this meant a lower compatibility rate initially, it was also useful to reveal interoperability issues that needed to be fixed!

Finally, since legacy IE code is no longer being sent, we are able to drastically reduce our reliance on the CV list. This means that our top site compatibility rate will match our long tail compatibility much more closely.

Fixing patterns instead of sites

However, a new engine was not enough – we also needed to significantly revamp how we find, track and fix issues on the long tail of the Web. To do so, we do daily analysis on trillions of URLs crawled in conjunction with Bing to detect patterns that exist in the head of the Web and the tail of the Web. By fixing these patterns, sites just end up working. This data is augmented by thousands of daily feedback reports from users via the “smiley face” icon.

In addition, we revised our internal engineering processes to prioritize real-world interoperability issues uncovered by our data analysis. With these processes in place, we set about fixing over 3000 interoperability bugs and adding over 40 new Web standards (to date) to make sure we deliver on our goals.

We don’t see this interoperability effort having an end date – we’ll be continuously checking the data and rolling out improvements to the new rendering engine. For users that upgrade to Windows 10, the engine will be evergreen, meaning that it will be kept current with Windows 10 as a service.

A community effort

Our mission to create a Web that “just works” won’t be successful without your help. That’s why the Project Spartan journey has included a number of ways for you to get involved. Here’s a quick rundown of ways you can help:

  • Join the Windows Insider Program to get the latest Windows 10 previews and test your Web sites on our new rendering engine by enabling experimental Web features in the about:flags page. If you don’t have a device to install it on, try out RemoteIE which will stream our latest browser from the Azure cloud to Windows, iOS or Android devices.
  • If you see any site issues, “Send a Frown” using the smiley icon so we can track it. If you have a more detailed description of the issue, you can open a bug on Connect.
  • Check out our Web platform roadmap at status.modern.ie. If anything is missing, submit a feature request or vote for an existing request at our UserVoice site.
  • Send us a message on Twitter @IEDevChat, or join a monthly #AskIE tweetchat (next one on 2/26 @ 10AM PST). For help on more detailed issues, ask a question on our StackOverflow page.

We believe the break from IE’s past we’ve made to create a new rendering engine will help make the browsing experience for our customers better, and make building Web sites that just work across browsers easier for Web developers. We’ll be sharing more details on specific aspects of the approach outlined above in the coming weeks and look forward to your continued participation in this journey with us.

Charles Morris, Program Manager Lead, Project Spartan

The human touch in computing

Thu, 02/26/2015 - 09:00

The human touch in computing
Researchers explore vision, speech, multimedia, and other aspects of human-centric computing at a joint lab operated by Microsoft Research Asia and the Chinese University of Hong Kong.

...(read more)

Using the Calendar API in PHP

Thu, 02/26/2015 - 08:30

In the same spirit as my Python experiment, I decided to give PHP a try and see how far I could get with the Office 365 APIs. The same disclaimers apply: I am a total PHP neophyte. I'm certain that my code is probably non-optimal, but hey, it's a learning experience!

The idea

I had this idea for a simple app that could make use of the Calendar API. The app is a list of upcoming shows for a local community theater's Shakespearean festival. Visitors to their site can connect their Office 365 account and then add events directly to their calendar for the shows that they are attending.

The setup

There are a lot of options for PHP development, but in my case, I decided to install IIS 8 on my laptop and install PHP via the Web Platform Installer. If you're developing on a Windows machine, this is super easy. In just a few short minutes I was up and running. I didn't have to install anything else, the installer included all of the libraries I needed.

The execution

I started by creating a home page (home.php) that would show a table of upcoming shows. To make it a little dynamic, I created a class to generate the list based on the current date, and to randomize the location and whether or not a voucher was required (more on this later). To keep things constant throughout a session, I save that list in the $_SESSION global so I can reuse it. I ended up with a page that looks like this:

Now to make the "Connect My Calendar" button do something.

OAuth

I decided to create a static class to implement all of the Office 365-related functionality. I created Office365Service.php and created the Office365Service class. The first thing we need it to do is "connect" the user's calendar. Under the covers what that really means is having the user logon and provide consent to the app, then retrieving an access token. If you're familiar with OAuth2, then this is pretty straightforward. Essentially, we need to implement the authorization code grant flow against Azure AD.

To start that process, the "Connect My Calendar" button will send the user right to the authorization code request link. I added the getLoginUrl function to build this link based on my client ID:

private static $authority = "https://login.windows.net";
private static $authorizeUrl = '/common/oauth2/authorize?client_id=%1$s&redirect_uri=%2$s&response_type=code';

// Builds a login URL based on the client ID and redirect URI
public static function getLoginUrl($redirectUri) {
  $loginUrl = self::$authority.sprintf(self::$authorizeUrl, ClientReg::$clientId,
    urlencode($redirectUri));
  error_log("Generated login URL: ".$loginUrl);
  return $loginUrl;
}

I created authorize.php to serve as the redirect page, which is where Azure sends the authorization code response. All this file needs to do is extract the code parameter from the request, and use that to issue a token request.

$session_state = $_GET['session_state'];

...

// Use the code supplied by Azure to request an access token.
$tokens = Office365Service::getTokenFromAuthCode($code, $redirectUri);

So that's the next function I added to Office365Service:

// Sends a request to the token endpoint to exchange an auth code
// for an access token.
public static function getTokenFromAuthCode($authCode, $redirectUri) {
  // Build the form data to post to the OAuth2 token endpoint
  $token_request_data = array(
    "grant_type" => "authorization_code",
    "code" => $authCode,
    "redirect_uri" => $redirectUri,
    "resource" => "https://outlook.office365.com/",
    "client_id" => ClientReg::$clientId,
    "client_secret" => ClientReg::$clientSecret
  );

  // Calling http_build_query is important to get the data
  // formatted as Azure expects.
  $token_request_body = http_build_query($token_request_data);

  $curl = curl_init(self::$authority.self::$tokenUrl);
  curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($curl, CURLOPT_POST, true);
  curl_setopt($curl, CURLOPT_POSTFIELDS, $token_request_body);

  if (self::$enableFiddler) {
    // ENABLE FIDDLER TRACE
    curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
    // SET PROXY TO FIDDLER PROXY
    curl_setopt($curl, CURLOPT_PROXY, "127.0.0.1:8888");
  }

  $response = curl_exec($curl);

  $httpCode = curl_getinfo($curl, CURLINFO_HTTP_CODE);
  if (self::isFailure($httpCode)) {
    return array('errorNumber' => $httpCode,
                 'error' => 'Token request returned HTTP error '.$httpCode);
  }

  // Check error
  $curl_errno = curl_errno($curl);
  $curl_err = curl_error($curl);
  if ($curl_errno) {
    $msg = $curl_errno.": ".$curl_err;
    return array('errorNumber' => $curl_errno,
                 'error' => $msg);
  }

  curl_close($curl);

  // The response is a JSON payload, so decode it into
  // an array.
  $json_vals = json_decode($response, true);
  return $json_vals;
}

As you can see, I used curl for issuing requests. I found it very well suited for the job. I could easily build the request body as a standard array, encode it with http_build_query, and send it. Handling the response was easy too, with the built-in JSON function json_decode. That puts the response into an easy to manage array.

Now, back in authorize.php, I can save the tokens into the $_SESSION global and redirect back to the home page:

// Save the access token and refresh token to the session.
$_SESSION['accessToken'] = $tokens['access_token'];
$_SESSION['refreshToken'] = $tokens['refresh_token'];
// Parse the id token returned in the response to get the user name.
$_SESSION['userName'] = Office365Service::getUserName($tokens['id_token']);

// Redirect back to the homepage.
$homePage = "http".(($_SERVER["HTTPS"] == "on") ? "s://" : "://")
                  .$_SERVER["HTTP_HOST"]."/php-calendar/home.php";
header("Location: ".$homePage);

Notice that I also get the user's name from the ID token. This is just so I can display the logged on user's name in my app. Check the getUserName function in Office365Service.php if you're interested to see how that's done.

Calendar API

Now that the user can login, my homepage looks a little different:

Notice that the buttons now say "Add to Calendar", and the user's name appears in the top right along with a "logout" link. The logout link is very simple, it just goes to:

https://login.windows.net/common/oauth2/logout?post_logout_redirect_uri=<URL to post-logout page>

The value you set in the post_logout_redirect_uri is where Azure will send the browser after logging the user out. In my case, I created logout.php, which removes the data I stored in the $_SESSION global and then redirects to the home page.

CalendarView

Now on to what we came to see, the Calendar API. Clicking on the "Add to Calendar" takes us to the addToCalendar.php page:

So the first use of the Calendar API is the table of events on the right-hand side. To get this data, I created the getEventsForData function in Office365Service.php:

// Uses the Calendar API's CalendarView to get all events
// on a specific day. CalendarView handles expansion of recurring items.
public static function getEventsForDate($access_token, $date) {
  // Set the start of our view window to midnight of the specified day.
  $windowStart = $date->setTime(0,0,0);
  $windowStartUrl = self::encodeDateTime($windowStart);

  // Add one day to the window start time to get the window end.
  $windowEnd = $windowStart->add(new DateInterval("P1D"));
  $windowEndUrl = self::encodeDateTime($windowEnd);

  // Build the API request URL
  $calendarViewUrl = self::$outlookApiUrl."/Me/CalendarView?"
                    ."startDateTime=".$windowStartUrl
                    ."&endDateTime=".$windowEndUrl
                    ."&\$select=Subject,Start,End" // Limit the data returned
                    ."&\$orderby=Start"; // Sort the results by the start time.

  return self::makeApiCall($access_token, "GET", $calendarViewUrl);
}

This function uses the CalendarView API to get the list of events on a specific day. The advantage of using CalendarView is that when responding to a CalendarView request, the server handles expanding recurring meetings to figure out if a recurring meeting has an instance that falls in the specified time window. The instance will be included in the results like a normal appointment!

You may have noticed that the function ends by calling another function, makeApiCall. Even though it's a detour from the Calendar API, let's take a quick look at that function, because it shows some things that apply to all of the Office 365 REST APIs.

// Make an API call.
public static function makeApiCall($access_token, $method, $url, $payload = NULL) {
  // Generate the list of headers to always send.
  $headers = array(
    "User-Agent: php-calendar/1.0", // Sending a User-Agent header is a best practice.
    "Authorization: Bearer ".$access_token, // Always need our auth token!
    "Accept: application/json", // Always accept JSON response.
    "client-request-id: ".self::makeGuid(), // Stamp each request with a new GUID.
    "return-client-request-id: true"// The server will send request-id in response.
  );

  $curl = curl_init($url);

  if (self::$enableFiddler) {
    // ENABLE FIDDLER TRACE
    curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
    // SET PROXY TO FIDDLER PROXY
    curl_setopt($curl, CURLOPT_PROXY, "127.0.0.1:8888");
  }

  switch(strtoupper($method)) {
    case "GET":
      // Nothing to do, GET is the default and needs no
      // extra headers.
      break;
    case "POST":
      // Add a Content-Type header (IMPORTANT!)
      $headers[] = "Content-Type: application/json";
      curl_setopt($curl, CURLOPT_POST, true);
      curl_setopt($curl, CURLOPT_POSTFIELDS, $payload);
      break;
    case "PATCH":
      // Add a Content-Type header (IMPORTANT!)
      $headers[] = "Content-Type: application/json";
      curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "PATCH");
      curl_setopt($curl, CURLOPT_POSTFIELDS, $payload);
      break;
    case "DELETE":
      curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "DELETE");
      break;
    default:
      error_log("INVALID METHOD: ".$method);
      exit;
  }

  curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
  $response = curl_exec($curl);

  $httpCode = curl_getinfo($curl, CURLINFO_HTTP_CODE);

  if (self::isFailure($httpCode)) {
    return array('errorNumber' => $httpCode,
                 'error' => 'Request returned HTTP error '.$httpCode);
  }

  $curl_errno = curl_errno($curl);
  $curl_err = curl_error($curl);

  if ($curl_errno) {
    $msg = $curl_errno.": ".$curl_err;
    curl_close($curl);
    return array('errorNumber' => $curl_errno,
                 'error' => $msg);
  }
  else {
    curl_close($curl);
    return json_decode($response, true);
  }
}

The first thing I want to bring attention to is the set of headers sent with every request. Matthias and I have both talked about this before, but it bears repeating. While Authorization and Accept are self-explanatory, the others are related to what we refer to as client instrumentation. Doing this should be considered a "must-have" when using the REST APIs. It can be invaluable if you run into errors.

The second thing is subtle but important. For POST and PATCH requests, you MUST set the Content-Type header to application/json. If you don't, you'll get an ErrorInvalidRequest error with the message "Cannot read the request body."

Creating Events

Ok, detour over, back to the Calendar API! The second use of the API is to create the event when the user clicks on the "Add to my calendar" button on the addToCalendar.php page. Clicking that button takes the user to doAdd.php, which does the actual adding. For this, I added the addEventToCalendar function to Office365Service.php:

// Use the Calendar API to add an event to the default calendar.
public static function addEventToCalendar($access_token, $subject, $location,
    $startTime, $endTime, $attendeeString) {
  // Create a static body.
  $htmlBody = "<html><body>Added by php-calendar app.</body></html>";

  // Generate the JSON payload
  $event = array(
    "Subject" => $subject,
    "Location" => array("DisplayName" => $location),
    "Start" => self::encodeDateTime($startTime),
    "End" => self::encodeDateTime($endTime),
    "Body" => array("ContentType" => "HTML", "Content" => $htmlBody)
  );

  if (!is_null($attendeeString) && strlen($attendeeString) > 0) {
    $attendeeAddresses = array_filter(explode(';', $attendeeString));

    $attendees = array();
    foreach($attendeeAddresses as $address) {
      error_log("Adding ".$address);

      $attendee = array(
        "EmailAddress" => array ("Address" => $address),
        "Type" => "Required"
      );

      $attendees[] = $attendee;
    }

    $event["Attendees"] = $attendees;
  }

  $eventPayload = json_encode($event);

  $createEventUrl = self::$outlookApiUrl."/Me/Events";

  $response = self::makeApiCall($access_token, "POST", $createEventUrl, $eventPayload);

  // If the call succeeded, the response should be a JSON representation of the
  // new event. Try getting the Id property and return it.
  if ($response['Id']) {
    return $response['Id'];
  }

  else {
    return $response;
  }
}

Notice how easy PHP makes it to build the JSON payloads. All I need do is create an array and use json_encode to generate the payload. Very nice! Again this uses makeApiCall to send the request. We also don't need to worry about sending meeting invites. By adding attendees, the server takes care of that for us!

Adding an attachment

Remember before I said we'd get to the "voucher required" thing later? The voucher was really an excuse to add an attachment. If you add an event that requires a voucher to your calendar, the app will add the voucher as an attachment on the event. To do this, I added the addAttachmentToEvent function:

// Use the Calendar API to add an attachment to an event.
public static function addAttachmentToEvent($access_token, $eventId, $attachmentData) {
  // Generate the JSON payload
  $attachment = array(
    "@odata.type" => "#Microsoft.OutlookServices.FileAttachment",
    "Name" => "voucher.txt",
    "ContentBytes" => base64_encode($attachmentData)
  );

  $attachmentPayload = json_encode($attachment);
  error_log("ATTACHMENT PAYLOAD: ".$attachmentPayload);

  $createAttachmentUrl = self::$outlookApiUrl."/Me/Events/".$eventId."/Attachments";

  return self::makeApiCall($access_token, "POST", $createAttachmentUrl,
    $attachmentPayload);
}

The value of $attachmentData is the binary contents of the file to attach. In this case, it's a simple text file.

At this point you might be wondering: "Why not just include the attachment as part of the event when you created it? Wouldn't that be more efficient?" Well yes, it would, but it doesn't work! In order to add an attachment, you have to create the event first then POST to the event's Attachments collection.

The end result

If I stick with the premise of this being an experiment, then I have to conclude that PHP is a great language for calling the Office 365 REST APIs. Using just the built in libraries I was able to do everything I needed to do in a straightforward way. If PHP is your language of choice, you should have no trouble integrating with Office 365.

The sample app is available on GitHub. As always, I'd love to hear your feedback in the comments or on Twitter (@JasonJohMSFT).

Internet of Things: installing Mono/Linux and Windows Embedded on a BeagleBone Black board (BBB)

Thu, 02/26/2015 - 08:12

The BeagleBone Black, also called Beagle Bone Black or BBB is an interesting board. It’s an AM335x 1GHz ARM Cortex-A8 with 512M of DDR3, 4GB of flash and does have an HDMI, USB client and host interface. But what I prefer are the 2 times 46 pins to have tons of IO available. It does reduce a bit the number if you are using HDMI. But still, it’s an interesting board. On the not as good aspect the analogic entries are only 1.8V max and not protected (so you need to add a 1.8V Zener diode to protect all analogic entries.

The board does natively run Linux on it on the embedded Flash. And you can flash it with another Linux or even Windows Embedded see here.

I’m a C# guy (ok, sometimes I do some C++ too) so I like anything running C#. And I always try to find a way to develop using C# on any board. Sometimes it’s possible like for the RaspberyPi first version, see my blog post here. C” with either the full Microsoft .Net framework or the open source Mono one is a great way to be super productive in terms of development. Even in the embedded world if you don’t need hard code real time. And in the case you need some, then switch to C++ and create your own dll which you’ll be able to use later with C# for the rest.

In this article, I’ll explain how to install a new Linux compatible to install Mono as well as a Windows Embedded image.

 

The Linux way

I’m not a Linux guy and I have to admit, I had to spend quite a lot of time for this step. I realized not all SD cards were compatible and I had no real way to check this on Linux before really booting on the SD. So make sure you have recent SD card and looks like it will work fine.

Step 1: find a version that is BBB compatible

After researches using Bing, I found a link explaining almost everything. The project was about developing using Mono including classes to access the GPIO. You’ll find the full site here. So I learned I need to install on the board an armhf version of Linux. The one present is just arm and Mono does not work on it.

So after spending time figuring out that some of my SD card were not compatible while booting but compatible for flashing under Linux, I’ve finally found one recent compatible one. And to make sure the image was correct, I went for the Windows way of flashing. So I’ve downloaded an img file from here. The BB-ubuntu-14.04.2-console-armhf-2015-02_19-2gb.img.xz is working perfectly. And as recommended, I used the Win32DiskImager to flash the image. Other equivalent tools are also working perfectly.

Once flashed, put the SD card into the SD card reader, press the button which is up to the SD card, plug the power and release the button. This will make the BBB boot on the SD card. If none of the blue led is flashing, then you’ve most likely have the same issue as me with none compatible SD card. Try with another one! If it’s blinking, then everything should be all good.

You can easily test but using PuTTY or equivalent to connect to the board using SSH on port 22 with the IP address the board will get. If you don’t know or if you are not sure, just plug the board with the USB cable on your PC, use the address 192.168.7.2 to connect.

If everything goes correctly, you’ll be able to log with “ubuntu” as username and “temppwd” as the password.

At this point, you can go to the next step. If you don’t have this screen, it does mean you’ve most likely miss something or your SD is not compatible.

Step 2: installation Mono

As explained in the RaspberyyPi first version article, steps are the same. Conect with PuTTY or equivalent, make sure this time, you’re plug to a network cable and have internet access.

sudo apt-get update

The password is the same as for the login so “temppwd”. Now, you are downloading all updates, it can take few minutes.

sudo apt-get install mono-runtime

this will get the mono runtime and will allow you to further develop using Visual Studio and C#. Careful as this image is just a console one. If you want to add a desktop, you need to install one. You can for example install Gnome like this:

sudo apt-get install ubuntu-desktop

Step 3: accessing the GPIO thru C#

That’s where the first project I found is super useful as it’s a great, efficient implementation of OutpuPort and other well know classes from .Net Microrfamework (NETMF). You can download it directly from here. From this point, you can extract the source and directly use them in your project.

You’re now ready to do a C# development using the BBB and some common NETMF classes J For the deployment, either copy your build, or use a network deployment or anything equivalent. I may write as well how to setup the debugger and environment to include a full debugger.

 

The Windows Embedded way

Step 1: prepare an SD Card with Windows Embedded

Download the Windows image from Codeplex here. The download contains an example image.

You’ll need to prepare your SD card. For this, use diskpart with an admin command prompt:

Then find your SD card, it should be easy to find with the size. Here it’s a 8Gb one, so it’s the Disk 2. Select it by typing

Select disk 2

You can then list the partitions with “List part” make sure there is just one partition, if it’s not the case, you may have done a mistake with your disk. If you are not sure, it’s better to exit the tool, format your SD card with Windows the normal way and make sure you just have one partition. Make sure the original partition is formatted in FAT32 or exFAT.

Then select the particition with

Select part 1

Now the partition is selected, you need to make it active. This will allow the boot loader on the board to find the SD card and boot from there. This is done by using the command

active

Exit from diskpart now. Now you need to copy the Windows Embedded files to the bootable SD card you’ve prepared. Launch the TI SDCard boot utility from the tools directory from the downloaded files. Select the MLO file. It will be copied first on the SDCard, then select the other elements so the EBOOTSD.nb0, the logo.bmp and NK.bin files.

Click proceed, skip the formatting of the SD card (should have been done previously and the partition is now active). One done, you’re ready!

Place the SD card in the SD card reader on the board, press the button which is up the SD card reader, plug the power and release the button.

If everything went correctly, you are now running Windows Embedded Compact version on the BeagleBone Black.

You can test by trying to find your board on the network. I’m using Advanced IP Scanner. If you’ve used the board previously, the IP address your DHCP server will give will be the same.

You can now access the board with Telnet or thru FTP. If you’ve done everything correctly, then you’ll be able to see this from the ftp address:

And from Telnet:

From here, you can now play a bit and try to connect remotely to the device with a tool called “Remote Display Control for Windows CE”. You’ll find the tool in the tools directory. Just type in the telnet session

Cerdisp -c

And launch the Host version on your desktop. You will see the board and you can now play with it.

Step 2: installing .NET and using GPIO

OK, for this step, it’s a bit more complicated as it does require to rebuild totally a Windows CE image for this board. The platform elements are included in the BeagleBone WEC7 SDK file. Once you’ve installed the full Windows Embedded Compact 7, you then install the SDK. And you can create a project to include the .NET framework.

I haven’t done it yet but I used Windows CE quite a long time ago and there is a lot of fun building your own OS with component. The god news is that Windows EC is free for individuals:

Individual Non-Commercial Use. If you are an individual (that is, a natural person, and not an organization, business entity, or unit of government), then you may flash or download the Runtime Image to your device for your own Non-Commercial use.

From full license here.

Access to GPIO is then a bit of development as you’ll need to build a DLL to be able to do it or see if there is an existing one in the PAL.

If you do an image, please let me know! Enjoy and have fun.

Device Simulator for Event Hubs

Thu, 02/26/2015 - 08:07

This post will show how to build a simulator to send messages to Azure Event Hubs.

Background

I have been doing a lot of work with Event Hubs stream processing lately, and I find that I need different types of device simulators.  I need one that sends a few messages, one that continuously sends a few messages, one that sends a lot of messages, and so on.  I have probably 10 different console apps that do nothing more than send messages to Event Hubs in a loop, so I decided to make one simulator that fits all my needs.

This code adapts the code found in the Service Bus Event Hubs Getting Started sample.

The application makes it possible to simulate a number of devices sending X number of messages, with a sleep duration between each operation.  You can run a finite number of times, or run continuously, and the application enables the end user to cancel the operation by pressing ENTER.

This is a console application built with .NET, but I could have just as easily created a Java application that uses JMS to send messages.  I showed how to do this in the post Use JMS and Azure Event Hubs with Eclipse

Reusing Existing Code

I created a library called EventHubDemo.Common that has two classes, MetricEvent and EventHubManager.  Those classes are detailed in my post Scaling Azure Event Hubs Processing with Worker Roles as I use the same code between the sender (this post) and the receiver.

MetricEvent.cs

  1. using System.Runtime.Serialization;
  2.  
  3. namespace EventHubDemo.Common.Contracts
  4. {
  5.     [DataContract]
  6.     public class MetricEvent
  7.     {
  8.         [DataMember]
  9.         public int DeviceId { get; set; }
  10.         [DataMember]
  11.         public int Temperature { get; set; }
  12.     }
  13. }

EventHubManager.cs
  1. using Microsoft.ServiceBus;
  2. using Microsoft.ServiceBus.Messaging;
  3. using System;
  4. using System.Diagnostics;
  5.  
  6. namespace EventHubDemo.Common.Utility
  7. {
  8.     public class EventHubManager
  9.     {
  10.         public static string GetServiceBusConnectionString()
  11.         {
  12.             string connectionString = Microsoft.WindowsAzure.CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");
  13.             
  14.             if (string.IsNullOrEmpty(connectionString))
  15.             {
  16.                 Trace.WriteLine("Did not find Service Bus connections string in appsettings (app.config)");
  17.                 return string.Empty;
  18.             }
  19.             ServiceBusConnectionStringBuilder builder = new ServiceBusConnectionStringBuilder(connectionString);
  20.             builder.TransportType = TransportType.Amqp;
  21.             return builder.ToString();
  22.         }
  23.  
  24.         public static NamespaceManager GetNamespaceManager()
  25.         {
  26.             return NamespaceManager.CreateFromConnectionString(GetServiceBusConnectionString());
  27.         }
  28.  
  29.         public static NamespaceManager GetNamespaceManager(string connectionString)
  30.         {
  31.             return NamespaceManager.CreateFromConnectionString(connectionString);
  32.         }
  33.  
  34.  
  35.         public static void CreateEventHubIfNotExists(string eventHubName, int numberOfPartitions, NamespaceManager manager)
  36.         {
  37.             try
  38.             {
  39.                 // Create the Event Hub
  40.                 Trace.WriteLine("Creating Event Hub...");
  41.                 EventHubDescription ehd = new EventHubDescription(eventHubName);
  42.                 ehd.PartitionCount = numberOfPartitions;
  43.                 manager.CreateEventHubIfNotExistsAsync(ehd).Wait();
  44.             }
  45.             catch (AggregateException agexp)
  46.             {
  47.                 Trace.WriteLine(agexp.Flatten());
  48.             }
  49.         }
  50.  
  51.     }
  52. }
Creating the Console Application

I created a new console application named Sender.  I added a reference to my EventHubDemo.Common library, and added NuGet packages for Service Bus and Json.NET.

Realizing that others might use this program, I wanted to give a little documentation at runtime.  I created a class MyArgs.cs that is responsible for parsing arguments, providing help, and providing current values.

MyArgs.cs
  1. using System;
  2. using System.Text;
  3.  
  4. namespace EventHubDemo.Sender
  5. {
  6.     public class MyArgs
  7.     {
  8.         public MyArgs()
  9.         {
  10.         }
  11.  
  12.         public string eventHubName { get; set; }
  13.  
  14.         public int numberOfDevices { get; set; }
  15.  
  16.         public int numberOfMessages { get; set; }
  17.  
  18.         public int numberOfPartitions { get; set; }        
  19.  
  20.         public int sleepSeconds { get; set; }
  21.  
  22.         public int iterations { get; set; }
  23.  
  24.  
  25.         public void ParseArgs(string[] args)
  26.         {
  27.             if (args.Length != 6)
  28.             {
  29.                 throw new ArgumentException("Incorrect number of arguments. Expected 6 args <eventhubname> <NumberOfDevices> <NumberOfMessagesToSend> <NumberOfPartitions> <sleepSeconds> <iterations>", args.ToString());
  30.             }
  31.             else
  32.             {
  33.                 eventHubName = args[0];
  34.                 numberOfDevices = Int32.Parse(args[1]);
  35.                 numberOfMessages = Int32.Parse(args[2]);
  36.                 numberOfPartitions = Int32.Parse(args[3]);
  37.                 sleepSeconds = int.Parse(args[4]);
  38.                 iterations = int.Parse(args[5]);
  39.             }
  40.         }
  41.  
  42.         internal string GetHelp()
  43.         {
  44.             StringBuilder sb = new StringBuilder();
  45.             sb.AppendLine("Usage: Sender.exe EventHubName NumberOfDevices NumberOfMessagesToSend NumberOfPartitions SleepSeconds Iterations");
  46.             sb.AppendLine();
  47.             sb.AppendLine("Parameters:");
  48.             sb.AppendLine("\tEventHubName:\t\tName of the Event Hub to send messages to");
  49.             sb.AppendLine("\tNumberOfDevices:\t\tNumber of devices to simulate");
  50.             sb.AppendLine("\tNumberOfMessagesToSend:\t\tNumber of messages to send");
  51.             sb.AppendLine("\tNumberOfPartitions:\t\tNumber of Event Hub partitions");
  52.             sb.AppendLine("\tSleepSeconds:\t\tNumber of seconds to sleep between iterations (0 to send as fast as possible)");
  53.             sb.AppendLine("\tIterations:\t\tNumber of iterations (-1 to continuously send)");
  54.             sb.AppendLine();
  55.  
  56.             return sb.ToString();
  57.         }
  58.  
  59.         public override string ToString()
  60.         {
  61.             StringBuilder sb = new StringBuilder();
  62.             sb.AppendLine("Event Hub name: " + eventHubName);
  63.             sb.AppendLine("Number of devices: " + numberOfDevices);
  64.             sb.AppendLine("Number of messages: " + numberOfMessages);
  65.             sb.AppendLine("Number of partitions: " + numberOfPartitions);
  66.             sb.AppendLine("Seconds to sleep between iterations: " + sleepSeconds);
  67.             sb.AppendLine("Number of iterations: " + iterations);
  68.  
  69.             return sb.ToString();
  70.         }
  71.     }
  72. }

I then adapted the code from Service Bus Event Hubs Getting Started to create a class responsible for sending messages to Event Hub.

Sender.cs
  1. using EventHubDemo.Common.Contracts;
  2. using Microsoft.ServiceBus.Messaging;
  3. using Newtonsoft.Json;
  4. using System;
  5. using System.Collections.Generic;
  6. using System.Diagnostics;
  7. using System.Text;
  8. using System.Threading.Tasks;
  9.  
  10.  
  11. namespace EventHubDemo.Sender
  12. {
  13.     public class Sender
  14.     {        
  15.         private string eventHubName;
  16.         private int numberOfDevices;
  17.         private int numberOfMessages;
  18.  
  19.  
  20.         public Sender(string eventHubName, int numberOfDevices, int numberOfMessages)
  21.         {
  22.             this.eventHubName = eventHubName;
  23.             this.numberOfDevices = numberOfDevices;
  24.             this.numberOfMessages = numberOfMessages;            
  25.         }
  26.  
  27.         public void SendEvents()
  28.         {
  29.             // Create EventHubClient
  30.             EventHubClient client = EventHubClient.Create(this.eventHubName);
  31.  
  32.             try
  33.             {
  34.                 List<Task> tasks = new List<Task>();
  35.                 
  36.                 // Send messages to Event Hub
  37.                 Trace.TraceInformation("Sending messages to Event Hub " + client.Path);
  38.  
  39.                 Random random = new Random();
  40.                 for (int i = 0; i < this.numberOfMessages; ++i)
  41.                 {
  42.                     // Create the device/temperature metric
  43.                     MetricEvent info = new MetricEvent() { DeviceId = random.Next(numberOfDevices), Temperature = random.Next(100) };
  44.                     var serializedString = JsonConvert.SerializeObject(info);
  45.  
  46.                     EventData data = new EventData(Encoding.UTF8.GetBytes(serializedString));
  47.  
  48.                     // Set user properties if needed
  49.                     data.Properties.Add("Type", "Telemetry_" + DateTime.Now.ToLongTimeString());
  50.                     OutputMessageInfo("SENDING: ", data, info);
  51.  
  52.                     // Send the metric to Event Hub
  53.                     tasks.Add(client.SendAsync(data));
  54.                 }
  55.                 ;
  56.  
  57.                 Task.WaitAll(tasks.ToArray());
  58.             }
  59.             catch (Exception exp)
  60.             {
  61.                 Trace.TraceError("Error on send: " + exp.Message);
  62.             }
  63.  
  64.             client.CloseAsync().Wait();
  65.         }
  66.  
  67.         static void OutputMessageInfo(string action, EventData data, MetricEvent info)
  68.         {
  69.             if (data == null)
  70.             {
  71.                 return;
  72.             }
  73.             if (info != null)
  74.             {
  75.                 Trace.TraceInformation("{0}: Device {1}, Temperature {2}.", action, info.DeviceId, info.Temperature);
  76.             }
  77.         }
  78.     }
  79. }

I’ve used a variation of this Sender class in a number of ways, using various loops to simulate.  What I finally settled on is to call the Sender.cs class and using a little bit of threading code.

Program.cs
  1. using EventHubDemo.Common.Utility;
  2. using System;
  3. using System.Diagnostics;
  4. using System.Threading;
  5. using System.Threading.Tasks;
  6.  
  7. namespace EventHubDemo.Sender
  8. {
  9.     class Program
  10.     {
  11.         protected ManualResetEvent runCompleteEvent = new ManualResetEvent(false);
  12.         protected CancellationTokenSource cancellationTokenSource = new CancellationTokenSource();
  13.         private MyArgs a;
  14.  
  15.         public Program(MyArgs args)
  16.         {
  17.             a = args;
  18.         }
  19.         static void Main(string[] args)
  20.         {
  21.             MyArgs a = new MyArgs();
  22.  
  23.             try
  24.             {
  25.                 a.ParseArgs(args);
  26.             }
  27.             catch
  28.             {
  29.                 Console.WriteLine(a.GetHelp());
  30.                 return;
  31.             }
  32.             
  33.  
  34.             Trace.TraceInformation(a.ToString());
  35.  
  36.             Program p = new Program(a);
  37.                         
  38.             var token = p.cancellationTokenSource.Token;
  39.  
  40.             Task.Factory.StartNew(() => p.Run(token, a));
  41.             Task.Factory.StartNew(() => p.WaitForEnter());
  42.  
  43.             p.runCompleteEvent.WaitOne();            
  44.             
  45.         }
  46.  
  47.         protected void WaitForEnter()
  48.         {
  49.             Console.WriteLine("Press enter key to stop worker.");
  50.             Console.ReadLine();
  51.             cancellationTokenSource.Cancel();
  52.         }
  53.  
  54.  
  55.         private void Run(CancellationToken token, MyArgs a)
  56.         {
  57.  
  58.             if (a.iterations == -1)
  59.             {
  60.                 //Continuously iterate
  61.                 while (!token.IsCancellationRequested)
  62.                 {
  63.                     SendMessages(a);
  64.  
  65.                     //Convert to milliseconds
  66.                     Thread.Sleep(a.sleepSeconds * 1000);
  67.                 }
  68.                 runCompleteEvent.Set();
  69.             }
  70.             else
  71.             {
  72.                 //Iterate a finite number of times, enabling the user
  73.                 //  to cancel the operation
  74.                 for (int i = 0; i < a.iterations; i++)
  75.                 {
  76.                     if (!token.IsCancellationRequested)
  77.                     {
  78.                         SendMessages(a);
  79.  
  80.                         //Convert to milliseconds
  81.                         Thread.Sleep(a.sleepSeconds * 1000);
  82.                     }
  83.                     else
  84.                     {
  85.                         break;
  86.                     }
  87.                 }
  88.                 runCompleteEvent.Set();
  89.             }                           
  90.         }
  91.  
  92.         private static void SendMessages(MyArgs a)
  93.         {
  94.             var namespaceManager = EventHubManager.GetNamespaceManager();
  95.             EventHubManager.CreateEventHubIfNotExists(a.eventHubName, a.numberOfPartitions, namespaceManager);
  96.  
  97.             Sender s = new Sender(a.eventHubName, a.numberOfDevices, a.numberOfMessages);
  98.             Trace.TraceInformation("Sending events");
  99.             s.SendEvents();
  100.         }        
  101.     }
  102. }

Our program’s entry point is, of course, the static Main method.  We parse the arguments and then reference the cancellation token.  This allows us to check for token.IsCancellationRequested to see if the user cancelled the current operation.  We then use a ManualResetEvent to wait for a signal that the application is complete.  This allows us to run for a single iteration and then terminate, or to continuously iterate. 

When the Run method is called, we check to see if the user wants to run for a finite number of times or to continuously execute.   In both cases, we enable the user to cancel the operation.

Side note: See Lync and Outlook teams?  The white screen of death is avoidable.

We call the Service Bus library to send messages to the Event Hub, and sleep for X number of seconds between iterations. 

App.Config

This part is also worth mentioning… we store the connection string for Service Bus and the Azure Storage Account connection string in app.config.  I also added the System.Diagnostics.ConsoleTraceListener so that all Trace output messages appear in the Console window.

Code Snippet
  1. <appSettings>
  2.   <!-- Service Bus specific app setings for messaging connections -->
  3.   <add key="Microsoft.ServiceBus.ConnectionString"
  4.        value="Endpoint=sb://kirkevans.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=REDACTED=" />
  5.   <!--<add key="AzureStorageConnectionString" value="UseDevelopmentStorage=true;DevelopmentStorageProxyUri=http://127.0.0.1:10000/" />-->
  6.   <add key="AzureStorageConnectionString"
  7.        value="DefaultEndpointsProtocol=https;AccountName=kirkestorage;AccountKey=REDACTED=="/>
  8. </appSettings>
  9. <system.diagnostics>
  10.   <trace autoflush="true">
  11.     <listeners>
  12.       <add name="consoleListener"
  13.            type="System.Diagnostics.ConsoleTraceListener"/>
  14.     </listeners>
  15.   </trace>
  16. </system.diagnostics>
Debugging With Visual Studio

To enable debugging in Visual Studio, you need to provide the command-line arguments.  Realizing that you might not have had to write a console application in a few years, just go to the project’s properties and set the command line arguments on the Debug tab.

The command line arguments I used were:

devicereadings 1000 5 16 1 –1

The Event Hub name is “devicereadings”, I am simulating 1000 devices, sending 5 messages per batch.  There are 16 partitions used for the Event Hub, and we sleep for 1 second between batches.  The program runs continuously until the user terminates.

The Result

We start the Sender using our desired parameters.

To see if it is actually working, I run a worker role that processes the Event Hub streams in the local compute emulator (details on how to create this are in the post Scaling Azure Event Hubs Processing with Worker Roles).

Summary

This post shows how to create a simple device simulator to send messages to Azure Event Hubs.  I realize that using Console Applications is not the most interesting demo in the world, and it would be nice to do something with this data besides just dumping it to Console output.  We’ll look at that in some upcoming posts.

For More Information

Service Bus Event Hubs Getting Started

Scaling Azure Event Hubs Processing with Worker Roles

Use JMS and Azure Event Hubs with Eclipse

Imagine - Make It Happen Student Event

Thu, 02/26/2015 - 07:38

We live in a time where we can imagine the impossible and make it probable.  

Hello and welcome to the 2015 Imagine Make It Happen Event! This unique event brings together influential tech speakers, students and start-ups redefining the rules to make their dreams a reality.

Tuesday, 31 March 2015 from 09:30 to 18:00 London,

 

The day is split into two sessions, you can choose to attend either or both.

 

Morning Sessions: 9:30 AM-1 2:30 PM

Registration at 9:00 AM


Being a startup takes just the right amount of ‘crazy’ and some strong core skills. Our morning sessions features keynote from Troy Collins, Founder of SecretEscapes.com, and are designed to help you get lean, keep investors keen and get your startup seen, as well as nuts and bolts workshops on how to scale, how to be productive and more.

 

Afternoon Sessions: 1:00 PM- 18:00 PM

Registration at 12:30 PM


Calling Students and futurists of all ages, to come and be inspired at the Imagine Cup 2015 UK finals. The largest Student Technology competition in the world. Yes it is a competition but more importantly it is a celebration of student innovation, the power to not just look into the future but to change it for the better.

The half day event will be hosted by Dave Coplin, Chief Envisioning Officer for Microsoft UK, and is an educational and inspirational afternoon that showcases the field of Computer Science through a celebration of student innovation! It will consist of exciting demonstrations that will showcase 12 of the most innovative student start up’s in Tech for Apps & Games & talks from Cambridge professor’s and Tech Industry experts. We hope to inspire students to consider fantastic opportunities available to them in becoming the next generation of Computer Scientists.

At the Microsoft UK HQ we are heads down currently creating the agenda for the day that as well as talks, will consist of breakout sessions where students can learn to code, test out the start-ups Apps & Games, meet the Xbox team and learn about the Microsoft developer tools.

Register here before all the places are gone -

: https://www.eventbrite.co.uk/e/imagine-make-it-happen-tickets-15650107894

Pages

Drupal 7 Appliance - Powered by TurnKey Linux