Memory-Optimized Tables – Helps in Performance and Scalability

Today, while performing the code review on one of my project, which is getting developed using many Azure services/technologies.

Being ‘Internet of Things’/IOT scenario, the requirements demands the use of “Polyglot Persistence” pattern. Because solution need to store the structured/SQL as well unstructured/NoSQL data. And as we know, to store the structured/relational data ‘SQL Azure’ is the default technology choice being on Azure and Microsoft person 🙂

So, while analyzing the stored procedure’s T-SQL code, observed that many of the SPs are utilizing the temporary tables for data computation/processing operations to improve the overall performance. Using temporary tables, table variables, or table-valued parameters was a reasonable/acceptable practice when I was a programmer 🙂 But started wondering if anything new added to this approach/pattern to improve for better. By using Bing, I quickly discovered that we really have something new and better of course namely, “Memory-Optimized Tables“. This is part of In-Memory OLTP, which is the premier technology available in SQL Server and Azure SQL Database for optimizing performance of transaction processing, data ingestion, data load, and transient data scenarios.

As MS docs says, Memory-optimized tables are tables, created using CREATE TABLE with “MEMORY_OPTIMIZED = ON” option. Memory-optimized tables are fully durable by default, and, like transactions on (traditional) disk-based tables, fully durable transactions on memory-optimized tables are fully atomic, consistent, isolated, and durable (ACID). Memory-optimized tables and natively compiled stored procedures support a subset of Transact-SQL. More details.

Hence, if you are exploring the options to enhance your SPs/T-SQL code on SQL Azure then please refer here for performance and scalability considerations.

The details scenario are documented with instructions @ Replace global tempdb ##table and Replace session tempdb #table

So, next time whenever you see CREATE TABLE #temptable and/ CREATE TABLE #temptable and choose to replace by memory optimization option then make sure you visit this blog to say thanks you 🙂

 

 

 

Designing High Availability and Disaster Recovery for IoT/Event Hub

Before we jump directly to the topic, it requires some pre-requisites. So, make yourself comfortable with them.

As per wiki, High Availability (HA) is a characteristic of a system, which aims to ensure an agreed level of operational performance, usually uptime, for a higher than normal period. It is measured as a percentage of uptime in a given year. For details, please refer.

And, Disaster Recovery (DR) involves a set of policies, procedures and tools to enable the recovery or continuation of vital technology infrastructure and systems following a natural or human-induced disaster. Disaster recovery focuses on the IT or technology systems supporting critical business functions. For details, please refer.

Azure like any other cloud provider, has many built-in platform features that support highly available applications. However, you need to design the application specific logic (checklist) which absorbs fluctuations in availability, load, and temporary failures in dependent services and hardware. So that, the overall solution continues to perform acceptably, as defined by business requirements or application service-level agreements (SLAs). For details, please refer.

Hoping above info provides the high-level picture of HA/DR. Remaining post is more focused on a specific scenario in Internet of Things (IOT), basically the headline 🙂

I’m intentionally skipping the conceptual part of HA/DR importance, how to measure it, and different enables. As enough literature is available of the www.

Designing HA/DR for a solution which is using IoT/Event Hub has few considerations –

  • Devices are Smart – The devices should either have logic to differentiate between the primary and secondary region/site or shouldn’t declaratively aware of any endpoint. One of the way is to devices regularly check a concierge service for the current active endpoint. The concierge service can be a web service that is replicated and kept reachable using DNS-redirection techniques (Example, Azure Traffic Manager or AWS Route 53). So, you need to ask yourself what will happened to messages when cloud endpoint is not available? Message loss is acceptable/not? If yes, then fine otherwise you need some offline storage/queue at device end also.
  • Devices Identities – Generally endpoint understand the devices identities, if so then all device identities should be geo-replicated/backups and pushed to the secondary IoT hub before switching the active endpoint for the devices. Accordingly, the concierge service and ultimately devices must be made aware of this change in the endpoint. Also you need to develop the tools/utilities to quickly upload/push devices metadata to the IoT Hub.
  • Delta Identification and Merge – Once the primary region becomes available again, all the state and data that have been created in the secondary site must be migrated back to the primary region. This state and data mostly relates to device identities and application metadata, which must be merged with the primary IoT hub and any other application-specific stores in the primary region.

How much time it should take to fall back to secondary site and recover from it, is something which is solution specific and depends on solution’s RPOs and RTOs.

The overall approach includes following considerations in two major areas –

  • Device – IoT Hub
    • A secondary IoT hub
    • Backup Identities to a geo-redundant store
    • Device routing logic
    • Merging identities, when Primary is back
    • Either interim message store on device or message loss acceptable.
  • Application Components/Storages
    • A secondary App/Services Instance
    • Enable geo-redundant for all storages
    • Restoration of data/states from used storages (SQL & NoSQL)
    • Anything custom

Here is the conceptual architecture diagram, which depicts the proposed solution.

Although diagram is self-explanatory – but feel free to comment/ask on anything.

 

The Azure Architecture Center is available now

The Azure Architecture Center is available now in the documentation section of Azure. It’s Open to everyone with no cost to access the information. The Architecture Center is an extremely valuable resource as it brings –

  • Information for all cloud users ranging from beginners to specialists.
  • Best practices for security, availability, scalability, performance, cost, and manageability.
  • Tested, proven, and verified guidance. Not theoretical designs, they have been built and successfully run and ready for production.
  • Prepared deployment scripts and diagrams that anyone can use to get started quickly

The main areas of the architecture center covers are –

  • Application Architecture Guide – This guide presents a structured approach for designing applications on Azure that are scalable, resilient, and highly available.
  • Reference Architectures – Scenarios with related architectures grouped together.
  • Cloud Design Patterns – These design patterns are useful for building reliable, scalable, secure applications in the cloud.

One of interesting topic is a special section for customers coming from compete cloud provider namely, AWS. It helps Amazon Web Services (AWS) experts understand the basics of Microsoft Azure accounts, platform, and services. It also covers key similarities and differences between the AWS and Azure platforms, here.

Lastly, the people who are deep in architectural/design work should visit here. This provides resources including icons, Viso templates, PNG files, and SVG files that are useful for producing your own architecture diagrams. A direct link to download.

LAMP and Azure – Misconceptions vs Possibilities

A discussion of the Microsoft Platform (Windows, IIS, SQL Server and ASP.NET) vs LAMP (Linux, Apache, MySQL and PHP) topic covers a large set of topics.

My intent is not compare 1:1 but commenting on a scenario.

In many discussions, I realized many people have perception/misconceptions that, Azure is not really meant for traditional web-based applications built on the LAMP (Linux Apache MySQL PHP).

However, truth is that you can deploy LAMP stack on Azure to rapidly build, deploy, and dynamically scale websites and web apps using IaaS (VM scale sets) and PaaS (Azure Web Apps)

 

 

So, Customers who want to – Upgrade web apps to the cloud for scalability, high availability and other cloud traits like global presence, and dynamically scale (up and down) websites in a cost-effective. You should consider Azure as you get Architectural choices for hosting websites to choose from a wide array of architectures (containers, VMs, PaaS services, Azure Functions, etc.) and languages (node.js, PHP, Java, etc.). Linux web apps, let us create node and Java script websites that are fully managed.

Providers like, Bitnami provides images which are pre-configured, tested and optimized for Microsoft Azure and portable across platforms. Which provides quick and ready to use services.

For more information please feel free to visit @ https://azure.microsoft.com/en-in/overview/choose-azure-opensource/

File Storage and Functions – A files import story in Azure

The story goes like this – you have set of files which should be imported into a solution hosted on Azure.

Idea is to cover the scenario technically – the key players are Azure File Storage, Azure Functions.

If you don’t know already then quick summary –

  1. Azure File storage
    It’s a service that offers file shares in the cloud using the standard Server Message Block (SMB) Protocol. With Azure File storage, you can migrate legacy applications that rely on file shares to Azure quickly and without costly rewrites. Applications running in Azure virtual machines or cloud services or from on-premises clients can mount a file share in the cloud, just as a desktop application mounts a typical SMB share. Any number of application components can then mount and access the File storage share simultaneously. Since a File storage share is a standard SMB file share, applications running in Azure can access data in the share via file system I/O APIs. For more details please refer here. [Reference: Azure docs]
  2. Azure Functions – It’s a service that offers a server-less compute service that enables you to run code on-demand without having to explicitly provision or manage infrastructure. Use Azure Functions to run a script or piece of code in response to a variety of events. So, a solution for easily running small pieces of code, or “functions,” in the cloud. You can write just the code you need for the problem at hand, without worrying about a whole application or the infrastructure to run it. Functions can make development even more productive, and you can use your development language of choice, such as C#, F#, Node.js, Python or PHP. For more details please refer here. [Reference: Azure docs]

The Overall process –

  • Define the structure for Input files location – In file storage, defines a structure for Input file, Processed and Failed file by using ‘Share(s)’ and ‘Directory(s)/Files(s)’.
  • New file detection mechanism – check the presence of new file(s) as per predefined schedule and add message to a queue for further processing. Using a Function triggered by timer.
  • Import the files/data into system – A Function which process the input file(s) and ultimately imports the data.
  • Perform cleanup at Input files location – Mark files as processed, or move files to processed/failed directory for reference/tracking purpose.

Now, the devil is in the detail –

The Azure File service offers the four resources: the storage account, shares, directories, and files. The File service REST API provides a way to work with share, directory, and file resources via HTTP/HTTPS operations. So, instead of UNC/file-share/mapping, you need to use Azure Storage SDK which is a wrapper over Azure Storage REST API. This should avoid any UNC/mapping/related issues.