Azure Event Hub Message Publishing from Connected Devices

Building an application for a connected device like a mobile phone, tablet, or a microcomputer is easier than ever these days. A variety of tools and supported programming languages allow developers to quickly build applications that collect sensor data, capture telemetry information or location data, and send the collected information to backend services for processing.

Connected devices communication challenges

A functional prototype can often be built in a matter of days or even hours but a number of challenges arise and must be addressed in order to prepare the application for a wider distribution. Some of the most common challenges are:

  1. Platform support – applications running on various platforms must be able to send information.
  2. Reliability – the backend service must be able to reliably accept information.
  3. Security – only authorized devices should be able to send information and the information must be protected from unauthorized access and manipulation.
  4. Latency – the network calls must be as quick as possible to avoid network interruptions, preserve system resources and battery power.
  5. Scalability – in many cases, the backend service must be able to handle massive amounts of data.

It is obviously possible to build a custom solution that satisfies all of your application-specific requirements but would it bring you any business value or would you rather focus on the actual application functionality? Luckily, there are cloud services specifically designed for these types of scenarios, and they can significantly simplify your development efforts.

Azure Event Hubs service introduction

Meet Azure Event Hubs – a highly scalable, low latency and high availability event ingestor service capable of reliably handling millions of events per second and buffering them for further processing for anywhere from 24 hours up to 7 days. The service supports HTTP and AMQP protocols which makes it an attractive option for a wide variety of devices and platforms.

The Event Hubs REST API is pretty straight-forward and easy to use. Simply submit an HTTP POST request to your event hub endpoint and set the request body to a JSON-encoded string that contains one or more messages. For example, below is a sample Send Event request, courtesy of Event Hubs REST API Reference on MSDN.

POST https://your-namespace.servicebus.windows.net/your-event-hub/messages?timeout=60&api-version=2014-01 HTTP/1.1
Authorization: SharedAccessSignature sr=your-namespace.servicebus.windows.net&sig=tYu8qdH563Pc96Lky0SFs5PhbGnljF7mLYQwCZmk9M0%3d&se=1403736877&skn=RootManageSharedAccessKey
Content-Type: application/atom+xml;type=entry;charset=utf-8
Host: your-namespace.servicebus.windows.net
Content-Length: 42
Expect: 100-continue

{ "DeviceId":"dev-01", "Temperature":"37.0" }

Generating a request like this should not require much effort in any programming language on any platform, be it a mobile app or a Node.js application running on a Raspberry Pi. The only part that deserves special attention is the Authorization request header. The Authorization header contains a Shared Access Signature (SAS) token that can be generated by any client that has access to the signing key specified in the shared access authorization rule. Below are some of the best practices to follow when generating SAS tokens for connected devices.

Azure Event Hubs SAS token generation best practices

  1. Never use the RootManageSharedAccessKey shared access policy to generate SAS tokens for your connected devices since it’s a highly privileged policy. Follow the principle of least privilege and always create a new policy with Send-only permission for each event hub.
  2. Never send or store your policy keys on connected devices as it may expose your keys for unauthorized access and makes it difficult to rotate or revoke them. Always generate your SAS tokens on the server and only store the generated tokens on the devices.
  3. Never generate SAS tokens and submit HTTP POST requests to the common event hub URI as the same SAS token would be valid for any device. Always target unique device-specific event hub endpoints when generating tokens and publishing events.

Additional resources

Amazon Web Services Tips for Developers and Solutions Architects

Amazon Web Services (AWS) provides a full range of services that allow developers and solutions architects to design and build scalable and fault tolerant applications without a large up-front hardware investment. There’s a vast amount of information about what AWS has to offer available online but the following features are worth mentioning again.

  • AWS Free Tier: included services and limits, billing alerts
  • AWS Accounts and IAM Users: quick overview and best practices
  • AWS EC2: IAM Roles for EC2, Instance Metadata and User Data
  • AWS S3: Lifecycle rules and Amazon Glacier
  • AWS Architecture Center: reference architectures, architecture whitepapers and official icons

AWS Free Tier

AWS Free Tier allows you to use most of the core AWS services free of charge for 12 months. As of August 2014, these services include, but are not limited to:

  • Amazon EC2 – resizable compute capacity in the Cloud. 750 hours of t2.micro instance usage per month.
  • Amazon S3 – highly scalable, reliable, and low-latency data storage infrastructure. 5 GB of Standard Storage, 20,000 Get Requests and 2,000 Put Requests.
  • AWS Trusted Advisor – AWS Cloud Optimization Expert. 4 best-practice checks on performance and security. Notification and customization features.
  • Amazon Mobile Analytics – fast, secure mobile app usage analytics. 100 Million free events per month.
  • Amazon Cognito – mobile user identity and synchronization. Unlimited user authentication and ID generation. 10 GB of cloud sync storage. 1,000,000 sync operations per month.
  • Amazon DynamoDB – fully managed NoSQL database service with seamless scalability. 100 MB of storage, 5 Units of Write Capacity and 10 Units of Read Capacity.

Additional information about the free AWS offerings can be found on AWS Free Tier page.

Keep in mind that not all of the AWS services are included in the free usage tier so it’s very easy to accidentally start accumulating balance while exploring the various AWS services. Billing alarms can be used to generate notification emails once your account balance reaches a certain threshold to avoid unexpected billing charges.

AWS Accounts and IAM Users

It’s tempting to start using your new AWS account (email address and password combination) to access the AWS resources but that goes against the AWS security best practices. AWS Identity and Access Management (IAM) users and groups should be used to manage access to AWS resources. Here’s a quick summary that describes each of these account types:

  • AWS account – this is the account you create when you sign up for AWS and it represents a business relationship with AWS. AWS accounts have root permissions to all AWS resources and services and should not be used for day-to-day interactions with AWS.
  • IAM users – can be a person, service, or application that needs access to your AWS resources. Best practice is to create IAM users and assign them individual security credentials needed to access AWS services and resources. You can also create an IAM user for yourself, grant it administrative privileges, and use that IAM user to access the AWS management console or the APIs.

For more details, refer to the IAM Best Practices article and the AWS Security Best Practices whitepaper.

AWS EC2

It may be challenging to securely distribute and rotate AWS credentials used by your EC2 instances to communicate with other AWS services and resources. In a typical application, the AWS access keys are included in the application configuration file which means that they are visible to anyone who has access to the EC2 instance and makes it difficult to rotate the credentials on a regular basis when you have a large number of running EC2 instances. IAM Roles were designed specifically to address this problem and they let you delegate permissions to your EC2 instances to make API requests without the need to manage security credentials at the application level.

You can read more about AWS IAM Roles in the IAM Roles for Amazon EC2 article.

In addition to AWS credentials, your application may need to retrieve additional information about the EC2 instance it’s running on. For example, when logging application errors you may want to also include the EC2 instance ID or the AMI ID used to launch the instance. Another common requirement is passing configuration information to a newly launched EC2 instance. AWS offers an elegant solution for these problems called Instance Metadata and User Data. The instance metadata is organized in categories and is accessible from within the instance via the following URL: http://169.254.169.254/latest/meta-data

To get the instance AMI ID, simply call http://169.254.169.254/latest/meta-data/ami-id or call http://169.254.169.254/latest/meta-data/hostname to get the hostname of the current EC2 instance.

To retrieve user data available to the instance, use the following URL: http://169.254.169.254/latest/user-data

To learn more about, visit the Instance and Metadata and User Data page.

AWS S3

AWS Simple Storage Service (S3) is a well-known cloud file storage service. One of the lesser known features of Amazon S3 is the ability to auto archive content to Amazon Glacier, an extremely low cost cloud archive service optimized for infrequently accessed data. Content archival is controlled by Lifecycle rules that enable you to ensure that data is automatically stored on the cloud storage option that is most cost-effective for your needs. Be aware that Amazon Glacier is not currently available on the AWS Free Tier.

For more information, please visit the Amazon Glacier section of the Amazon S3 FAQs article.

AWS Architecture Center

AWS Architecture Center is the go-to place to find the guidance and best practices necessary to build highly scalable and reliable applications on the AWS platform. Some of the highlights are:

  • AWS Reference Architectures – single-page datasheets that provide you with the architectural guidance on how to take full advantage of AWS services.
  • Architecture Whitepapers from AWS – offers in-depth articles that focus on particular concepts such as fault-tolerance or security best practices in the AWS Cloud. SharePoint 2013 on AWS whitepaper will teach you how to deploy SharePoint 2013 on AWS, following best practices for deploying a secure and highly available architecture across multiple Availability Zones.
  • AWS Simple Icons – an official icon set that includes icons for several AWS products and resources. Available in MS PowerPoint, MS Visio and SVG and EPS formats.