File Storage and Functions – A files import story in Azure

The story goes like this – you have set of files which should be imported into a solution hosted on Azure.

Idea is to cover the scenario technically – the key players are Azure File Storage, Azure Functions.

If you don’t know already then quick summary –

  1. Azure File storage
    It’s a service that offers file shares in the cloud using the standard Server Message Block (SMB) Protocol. With Azure File storage, you can migrate legacy applications that rely on file shares to Azure quickly and without costly rewrites. Applications running in Azure virtual machines or cloud services or from on-premises clients can mount a file share in the cloud, just as a desktop application mounts a typical SMB share. Any number of application components can then mount and access the File storage share simultaneously. Since a File storage share is a standard SMB file share, applications running in Azure can access data in the share via file system I/O APIs. For more details please refer here. [Reference: Azure docs]
  2. Azure Functions – It’s a service that offers a server-less compute service that enables you to run code on-demand without having to explicitly provision or manage infrastructure. Use Azure Functions to run a script or piece of code in response to a variety of events. So, a solution for easily running small pieces of code, or “functions,” in the cloud. You can write just the code you need for the problem at hand, without worrying about a whole application or the infrastructure to run it. Functions can make development even more productive, and you can use your development language of choice, such as C#, F#, Node.js, Python or PHP. For more details please refer here. [Reference: Azure docs]

The Overall process –

  • Define the structure for Input files location – In file storage, defines a structure for Input file, Processed and Failed file by using ‘Share(s)’ and ‘Directory(s)/Files(s)’.
  • New file detection mechanism – check the presence of new file(s) as per predefined schedule and add message to a queue for further processing. Using a Function triggered by timer.
  • Import the files/data into system – A Function which process the input file(s) and ultimately imports the data.
  • Perform cleanup at Input files location – Mark files as processed, or move files to processed/failed directory for reference/tracking purpose.

Now, the devil is in the detail –

The Azure File service offers the four resources: the storage account, shares, directories, and files. The File service REST API provides a way to work with share, directory, and file resources via HTTP/HTTPS operations. So, instead of UNC/file-share/mapping, you need to use Azure Storage SDK which is a wrapper over Azure Storage REST API. This should avoid any UNC/mapping/related issues.