Yes, the stat is correct. The number would have been even closer to 100% if it weren't for some enterprise customers who have their subscriptions created and "initialized" by their IT team before furnishing the subscriptions to the business units that will be using them.
This issue is specific to a type of Azure Marketplace offer called Azure Applications. And within that offer type, which has two plan types, it applies mainly to a plan type called Azure Managed Application (AMA). We'll briefly explain what AMA is and provide more details in the Background section below.
In this post, we'll go over why this issue happens and how you can easily fix it manually or using our automated tooling available in Maestro Studio ENSEMBLE.
If you are new to the Azure Marketplace and prefer to go through the background first, you can click here to jump to the "The Background" section below.
Azure Applications Overview: Solution Template & Managed Applications
A Solution Template is an Azure Marketplace plan that allows software vendors to publish their Azure-based solution by providing some marketing copy text, some screenshots and videos, a few other details like markets supported and finally a zip file containing the technical artifacts used for deploying the solution from the Azure Marketplace by customers. The technical assets comprise of two required JSON files. The first file describes the deployment's desired state (Azure resources like VMs, networking configuration, storage, etc); it is called an Azure Resource Manager (ARM) template, and is used by Azure developers, architects and DevOps engineers to build solutions on Azure. The second file describes what the UI would look like and what any UI interactions will be like. It describes UI widgets, their placement and any functions (think Excel equations) that would get triggered when a customer interacts with the UI, selecting certain options. The UI described in the UI definition file is what customers end up seeing in the Azure Marketplace when they select a products and click on the Create button. The UI definition collects information from the user and feeds it to the deployment template, providing a customizable deployment of a vendor's solution.
To make deploying a solution robust, Solution Templates contain a validation mechanism that checks whether a Solution Template is going to run into a VM quota issue, a resource support failure and many other trivial, yet time-consuming, negative experiences. The great thing is that this validation mechanism kicks in before deployment starts.
Solution Templates enable great flexibility but have some limitations:
- They are not monetizable. They cannot have a price assigned to them. The pricing will have to come from any underlying Virtual Machine or Container offers published in the Marketplace.
- They cannot be easily secured. A customer can accidentally delete resources that are part of the deployed solution and break the whole deployment.
- They are not easy to update. Infrastructure updates must be done manually, most likely using scripts provided by the vendor. This adds risk as a user/customer can accidentally break the deployment while deploying update/upgrade scripts on their own, without vendor supervision.
- They are not easy to support. Going back to the update/upgrade example of the previous bullet point, if a vendor were to provide support to the customer while they're deploying the update/upgrade script, they will need access to their deployment, at least to the resource group (a logical unit that acts as a bucket/container of resources, allowing resources to be grouped together under a logical unit even if they "physically" exist in different Azure data centers; this means they can easily be assigned permissions together or deleted together) where the resources are deployed. For some organizations with strict security and compliance policies, this might not be acceptable as doing it manually can punch holes in the organization's security posture, holes that are usually left open as customers' IT can forget to remove the access after the operation that required it had concluded.
To solve these limitations, Azure introduced the Azure Managed Application (AMA). An AMA is not unique to the Azure Marketplace. It can be used to deploy internal applications by utilizing an Azure feature called Service Catalog. But in terms of its applicability to the Azure Marketplace, it helped address the four key limitations of a Solution Template, list above. Here is how:
- Monetization: AMAs can have pricing associated with them, just like Virtual Machine and Container offers. However, Microsoft recently started advising publishers to only use the AMA pricing to cover support plans, not IP and software, and rely on the underlying VM or Container offers for cost. Some, who don't have an underlying VM or container are still using AMAs for monetization. See more on this topic in the Background section below.
- Security: AMAs separate the deployed solution's assets—the Azure resources—into a separate resource group with special permissions to help prevent customers from accidentally deleting resources or otherwise corrupting the configuration of the deployed vendor solution.
- Supportability: There is no more need for asking the customer for access to their subscription and resource group. AMAs can specify the level of access they need to have in order to be able to support a deployed solution, and a customer can accept that access and can revoke it at any later time or they can reject it. The ability to request just-in-time (JIT) access, which is time-limited, makes AMAs so much more powerful and helps reduce the potential attack surface.
- Updates/upgrades: The same mechanism that enables the supportability story can be leveraged for providing customers with updates and upgrades without any additional requests. The whole access control mechanism is transparent to the customer, and they can revoke permissions at any time.
- Post-Deployment UI: The ability to provide customers with clear next steps after a solution is deployed has always been a dilemma with Solution Templates. Vendors had to resort to documentation, both in the offer copy text itself and in separate documents linked to from that text. With AMAs, publishers can provide a special UI that is shown after the solution is deployed. They can provide markdown in the UI, including images, to guide that customer through the next steps.
Azure Managed Apps (AMA) enable monetization, improve security, streamline supportability and updates/upgrades, and provide a post-deployment UI.
In terms of popularity, Solution Templates are still more popular than AMAs, but the rate of progress of AMAs is a lot better than Solution Templates, as most new AI solutions are choosing AMAs over Solution Templates for some or all of the reasons listed in the bullet points above. This can be seen in the following log-based chart:
The Problem
We mentioned earlier that Azure provided a validation mechanism that checked whether a Solution Template is going to run into a VM quota issue, a resource support failure and many other trivial, yet time-consuming, negative experiences. This mechanism depends on the fact that the Azure Portal will have access to the deployment template, mainTemplate.json, in order to be able to enumerate its resources and any nested templates or resources.
The problem with Azure Managed Applications is that after they are published to the Azure Marketplace or to Service Catalog, the resource definition is a black box. It does not expose the ARM template until deployment time. This means that Azure's validation mechanism, now unable to peek into the ARM template, becomes useless! This also means that deploying an AMA is like rolling the dice and hoping that the deployment makes it through successfully.
Deploying an AMA is like rolling the dice and hoping that the deployment makes it through successfully.
This is a sad situation since AMAs provide powerful features (as mentioned above), but at a very costly price: losing deployment reliability.
The reason this is a big issue is simple: Whenever an Azure customer creates a new subscription, the subscription only has a handful of Resource Providers "pre-registered". These are the ones essential to Azure's tracking of resource consumption, billing and other core features of the Azure platform.
Whenever an Azure customer creates a new subscription, the subscription only has a handful of Resource Providers "pre-registered".
The following screenshot shows the typical list of resource providers that are RegistrationFree. They don't need to be registered. They can be re-registered but cannot be unregistered since some of them are essential for Azure consumption tracking and billing management.
In large organizations where there are Enterprise Agreements (EA) signed with Microsoft, IT teams have the ability to automate the creation of Azure subscriptions, assigning them to business units and priming them by registering a core set of resource providers, like the 3 essential ones:
- Compute
- Networking
- Storage
Even in some non-EA companies, the IT department would run some scripts to prime the subscriptions before sharing them with the users.
The problem here is that they will not register "all" resource providers—that would be an overkill. So, there is always a good chance that some AMA would have resources whose RPs are not registered in their target subscription. The priming of the subscriptions only helps decrease the chance of deployment failures.
The Solutions
There are two ways to solve the problem of validation in Azure Managed Applications:
- Manually
- Using automated tools
The Manual Way
The manual way involves the following:
- Scanning an ARM template's "resources" section (which is an array of object since the Azure Marketplace does not support "Language v2" features yet).
- For each resource, get the resource provider from the resource type namespace. For example, a virtual network resource will have "Microsoft.Network/virtualNetworks" as its type—that's the namespace, and the resource provider, or RP, is "Microsoft.Network".
- There will be several resources from the same resource provider even though they have different namespaces. You only need to represent them once.
- You must then add an "ArmApiControl" resource to the "basics" section of the createUiDefinition.json file. For the "Microsoft.Network" RP discussed above, the following would be an example of a registration API call.
{
"name": "registerMicrosoftNetworkArmApiControl",
"type": "Microsoft.Solutions.ArmApiControl",
"request": {
"method": "POST",
"path": "[concat(subscription().id, '/providers/Microsoft.Network/register?api-version=2021-04-01')]"
}
}
Be aware that making mistakes, like typos, in the resource provider declaration does not provide any feedback indicating failure. And that's the key risk in the manual approach—you're more prone to making typo-type mistakes, and you're also less likely to notice that unless you thoroughly test everything. Keep that in mind when deciding whether you want to embark on the manual approach or go with the automated one.
This way, when the Azure Portal loads the Managed Application, it will process the CUID first, which automatically registers the RPs. So by the time the solution is deployed, the RPs have been registered and the failures related to non-registered resource providers will not occur. Note that resource provider registration does take some time to fully complete, but that will not block the ARM template deployment as long as the RP's status is Registering or Registered.
It's important to emphasize that this solution and the automated one below only address the Azure Portal limitation. When deploying ARM templates using the Azure CLI, PowerShell or through IaC tools, it's your responsibility to create the script to register the RPs:
- For Azure CLI, you could use:
az provider register
- For PowerShell:
Register-AzResourceProvider
The Automated Way
We wouldn't be wasting your time with this post if the manual way was the only way. Doing it manually is OK for small solutions but is not sustainable for large, often-changing, Azure solutions. You have to remember to go check the ARM template after every change to make sure you haven't missed anything in the corresponding CreateUiDefinition. Manual methods are prone to errors, human mistakes, typos.
With Maestro Studio ENSEMBLE, all you need to do is click on the "Add Missing Resource Provider registrations" menu item. Maestro Studio ENSEMBLE will find any resource providers specified in the ARM template, (there could be multiple resources specified that reference a single resource provider), and add to the CreateUiDefinition widgets, to check for the registration status of the specified resource provider and attempt to register it if not already registered.
You can click this button as many times as you wish, and it will only add new resource providers. ENSEMBLE also ensures that certain resources which are "RegistrationFree", i.e. do not need to be explicitly registered, are not added unnecessarily.
So, instead of spending minutes scouring through the ARM template, figuring out the resource providers, and then hopefully adding the required registrations without making any mistakes, you get to do it with the click of a button, and within seconds, you're done!
A word on the Total Cost of Ownership (TCO)
Given that Microsoft has not always been able to dedicate sufficient resources to support AMA publishers, the burden has fallen squarely on the shoulders of ISVs' engineers and System Integrators (SI), when budgets allow it. However, the cost of ownership remains high, as institutional knowledge often departs with the engineers who leave. And with an SI, ISVs lose the in-house expertise and are dependent on the SI for any future updates of hotfixes which slows down release cycles and introduces security risks.
Maestro Studio ENSEMBLE reduces the Total Cost of Ownership (TCO) because you, not an SI, own your assets and IP. New in-house resources can get up-to-speed quickly since all they need is inside ENSEMBLE's visual IDE, a few clicks or drag-and-drops away. No need to worry about typos anymore—that's ENSEMBLE's job.
Maestro Studio ENSEMBLE
The following diagram summarizes the powerful functionality that Maestro Studio ENSEMBLE offers:
You can visit the ENSEMBLE product page on the StratusOn website to learn more about Maestro Studio ENSEMBLE.
What about Microsoft's Certification Process/Team?
Unfortunately, Microsoft outsources the certification of all Azure Marketplace offers to a third-party vendor (a "v dash" in Microsoft lingo).
The certification vendor's resources (humans) have limited automated tools. For example, they cannot validate whether the assets constituting a Solution Template or AMA are not broken. They sometimes run the ARM Test Toolkit (arm-ttk), but their use of the tool is not consistent. In addition, that tool only detects best practices violations and malformed JSON, but it cannot validate broken deployment assets or broken UI definitions or ARM templates. As a result of this, there are published Azure Applications today that will blow up when you attempt to load them in the Azure Marketplace UI inside the Azure Portal, due to broken UI definitions (which is what gets loaded and rendered when you pick an offer and plan in the Azure Marketplace UI—the ARM template validation happens only after you've walked through the UI steps and are ready to deploy).
Many Azure Marketplace publishers use Maestro Studio ENSEMBLE in order to ensure they can validate offers and plans before they send them to the Azure Certification Team for approval.
Background
As mentioned above, the issue discussed in this post pertains to the Azure Managed Application variation of Azure Applications. But what are Azure Applications?
The term Azure Applications can have a different meaning depending on who you talk to. It can be an "application" (for example a web application or a SaaS website) that is published on Azure, hence "Azure" application. But the truth is, Azure Applications are quite different. They're their own thing and are super powerful and unique to Azure. No other hyperscaler has anything like it—they may have bits and pieces similar to some of its bits and pieces, but not the whole thing.
Ibiza
The Azure Marketplace started as an "add-on" to allow some key customers to publish VM images. Microsoft's initial thought process was that no one needed IaaS (Infrastructure-as-a-Service), and that PaaS (Platform-as-a-Service) was what customers needed. The initial incarnation of Azure, based on RDFE (Red Dog Front End) had web roles, worker roles (PaaS), and then later VM roles to allow customizing the underlying VMs of the PaaS roles. All this happened while AWS was making big dough from IaaS and customers that needed it and its flexibility as it allowed many customers to lift-and-shift their VM images from their on-premises data center to AWS cloud. It took some time for Azure to wake up, and that's when they started onboarding some third-party customers, mainly with Linux images to Azure since back then, there was only Windows. It was maintained using a private GitHub repo to track the vendor's offers—yes, that was the Azure Marketplace, at least that's my recollection of it.
There was no automated process or a website to streamline the configuration of Azure Marketplace listings like there is today. This happened over several iterations that started with the "orange-themed" Microsoft Azure Publishing portal, followed by the Cloud Partner Portal (CPP). Finally, Partner Center brought the unification of the publishing experience for all Microsoft partners.
Fast forward to 2014 when Ibiza, the code name for the new Azure Portal, was out in preview with a spanking new high horsepower engine called Azure Resource Manager (ARM). With ARM came a declarative, JSON-based, language for describing an Azure environment in a blueprint-like manner. This was not new and had been done before by OpenStack and AWS CloudFormation. Azure's architects followed the trend with the introduction of ARM "templates". This was an exciting time as the "old" portal with its "Azure Service Management" (ASM) APIs, based on Azure's old RDFE architecture, had reached its limits and the "classic" resource providers in the old architecture were not meant for the scalability needed for Azure as a new set of customers—other than the enterprise customers and internal business units that have been carrying Azure up till this point—started arriving to Azure. Linux solutions could now be easily used on Azure, with certain limitations since the "engine" was initially tuned for Windows. But the new Formula 1 beast, ARM, came with incredible flexibility and extensibility.
The "New" Azure Marketplace
As Ibiza started taking shape, the ability to publish VM images, initially based on "first-party" images that Microsoft closely maintained (even Linux ones), became more automated. Well, automated using the publishing portal experience, not through APIs. That came way later, and it took longer than it should have to make it happen.
In order to support publishing custom solutions to the new Azure Marketplace, a new experience was created in which a solution could be defined using both the declarative deployment description—via the ARM template—and a similar declarative dialect for defining the UI and its interactions. This marked the beginning of the low-code language, called CreateUiDefinition, that allowed building UIs using a set of widgets that were similar to those used to build Ibiza. That was a game changer. No one else had that (and they still don't as far as I know).
This new Azure Marketplace solution listing experience was called Solution Templates, because it consisted of ARM "templates" that described a "solution".
A quick word on the early Azure Marketplace team members—the unsung superheroes:
In the early days of the Azure Marketplace, there was a small and very talented team of engineers dedicated to carrying the burden of running this new marketplace, managing certification of new offers, helping guide and educate customers and partners (and internal engineers) on the new way and its nuances and tricks. Their job was the hardest, and they were tasked with flying a plane while it was being built!
If it weren't for their patience and perseverance, there would be no Azure Marketplace today. Huge kudos to everyone, from the early certification and technical support team to the CRP (Azure Compute) team who leaned in to help support the Solution Template approach and provide guidance. There were also a lot of other folks, like engineers from the Commercial Software Engineering (CSE) group, whose depth of knowledge was second to none.
It certainly took an army to get this thing going.
Solution Templates
The early days of the Azure Marketplace, in particular for publishing Solution Templates, were extremely painful. The resource types that were allowed to be used in ARM templates were limited. The resource providers that managed the endpoints, the control planes, of the resources were incomplete and not everything was well documented (or documented at all) in the ARM template schemas. ARM template schemas are JSON draft-04 schemas, but even today, there are a lot of features that are not documented in the schemas and are interpreted "dynamically".
On top of all that, there was no way to know if an ARM template that was created and tested in some Azure region like East US would work fine if deployed in another, less popular, Azure region. Different regions had different support for resources and some VM SKUs for example were not supported in all Azure regions. So, partners who published Solution Templates on the Azure Marketplace expected that they would work fine in all Azure regions since they were told that the Azure Marketplace replicated offers to all Azure regions.
Imagine how surprised and frustrated those partners—Azure Marketplace vendors—were when they realized that the success rate of their Solution Template offers was dismal. Customers deploying their solutions would run into issues as silly as not having compute cores quota in their subscription enough to deploy certain VM sizes, and they would not find out about it until after the deployment of the ARM template had started. This resulted in the need for someone to clear up partially successful deployments manually.
Microsoft finally listened, and Azure provided a validation mechanism that checked whether a Solution Template was going to run into a VM quota issue, a resource support failure and many other trivial, yet time-consuming, negative experiences. This was huge, and it still is.
A Solution Template can have 2 key files, along with any number of supporting files (scripts, etc):
- mainTemplate.json: The ARM template
- createUiDefinition.json: The pre-deployment UI definition
Managed Applications (AMA)
As powerful as Solution Templates were, they had a lot of limitations and downsides.
- Security: Anyone can access the resources and accidentally delete them.
- Supportability: They were hard to maintain by vendors without asking the customer for access to their subscription and resource group, a big pain in some corporations with strict governance policies.
- Monetization: Solution Templates cannot have pricing assigned. AMAs can have pricing associated with them, just like Virtual Machine and Container offers. However, Microsoft recently started advising publishers to only use the AMA pricing to cover support plans, not IP and software, and rely on the underlying VM or Container offers for cost.
The Azure Marketplace's move advising publishers to only use the AMA pricing to cover support plans, not IP and software, and relying on the underlying VM or Container offers for cost is a very short-sighted move. The reasons are:
- Not every solution uses a VM or a container.
- Container offers are not popular in the Azure Marketplace. There are only about 125 container offers today, which is really tiny.
- There are many solutions that are based on Azure App Service connected to some database solution (Azure SQL Database, Cosmos DB, MySQL, etc) that have deployment scripts that get published to the Azure App Service. In those solutions, either a Deployment Script resource is used to spin up a container and run the scripts or a VM is used just for running the Custom Script Extension to run the scripts, and then the VM is deleted (or deallocated) after the solution is deployed successfully.
Trying to normalize the plethora of scenarios under just VMs and containers limits the flexibility that AMAs were supposed to provide.
Many publishers who have invested hundreds or thousands of hours building AMA artifacts don't pay attention to this guidance and still utilize AMAs for monetization. While commendable, this approach is not advisable as they risk having their offers delisted from the Azure Marketplace.
While we do not agree with the Azure Marketplace's decision to kill AMA monetization, we don't advise ignoring the new rules unless it is explicitly discussed with the Azure Marketplace and an exception is granted.
With AMA, there is now a single resource that encapsulates the solution. The real parts of the solution (e.g. VM(s), VNET, storage account(s), etc) are deployed in a resource group that has access control set on it in such a way to allow only read-only customer access or read-write customer access. In addition, publisher management access can be set up to allow providing support of the customer deployment without having access to the customer's data.
Pretty powerful technical features. There are also other powerful features related to co-sell and private offers that are beyond the scope of this post.
An AMA can have 3 key files, along with any number of supporting files (scripts, etc, just like with Solution Templates):
- mainTemplate.json: The ARM template
- createUiDefinition.json: The pre-deployment UI definition
- viewDefinition: The post-deployment UI definition
Azure Managed Applications can also be used to build internal applications in an organization or enterprise, applications that don't need to ship via the Azure Marketplace. This is done by publishing the AMA artifacts into Service Catalog and enabling access via configuring RBAC (role-based access control) on the Managed Application resource.
AMAs power some of the most popular Azure features like Azure Databricks, who is one of the first Microsoft partners that demanded the features that, at their inception were called "ARM Appliances", later became Azure Managed Applications.
Lighthouse
One of AMA's most powerful features that allows delegation of permissions across tenants later became its own feature. Internally called Towboat and publicly known as Azure Lighthouse, Azure Managed Services is a powerful feature that allows broader and more granular customization suitable for managing large infrastructure deployments. This is well-suited for Cloud Solution Providers (CSP) and Managed Services vendors, like RackSpace for example, who manage large, complex customer infrastructure deployments.
Conclusion
The features mentioned in this article is available in the Free Trial of Maestro Studio ENSEMBLE which allows you to kick the tires, so to speak, and get a first-hand experience of the product and the value it can bring and benefits it can add to your publishing process so you don't have to deal with the mundane details and focus on your own product.
We hope you found this blog article useful. Please provide us with feedback using our Contact form. You can also send us feedback from within the product using our Megaphone feature.
Note: Microsoft, Microsoft Azure, Windows are trademarks or registered trademarks of Microsoft Corporation.