Where can we host a Small Language Model (SLM) in the cloud so that edge devices can download and run the model locally?

Hello @Prasanna Sai Kommineni

Yes, you can host a custom-trained SLM in Azure and let edge devices download the model artifact and run inference locally/offline. The key distinction is that you don't need to deploy the model to an Azure online endpoint if inference happens on the device.

A good Azure pattern is:

Training → Azure ML Model Registry / Azure Storage → authenticated model download → local edge runtime

Azure Machine Learning lets you register and version custom models as model assets in a workspace or registry. That gives you centralized model lifecycle/version management in Azure.

For example, after fine-tuning your model, you can register the model artifacts:

az ml model create \
  --name my-edge-slm \
  --version 1 \
  --path ./model \
  --type custom_model

The edge application can then download the approved model version during provisioning or an update window and store it locally. Once downloaded, inference can run completely offline using a runtime appropriate to the model format.

For the on-device runtime, Foundry Local is also worth considering. Microsoft describes it specifically as an on-device AI solution where models execute locally, user data stays on the device, and applications can continue operating without network connectivity.

If your devices are IoT/industrial edge devices, Microsoft also documents an Azure IoT Edge architecture where an IoT Edge module downloads an AI model and performs inference locally; ONNX and other optimized model formats are commonly used for this pattern.

So I would choose between:

• Azure ML Model Registry - best when you want model versioning, governance, and MLOps.
• Azure Blob Storage - simpler when you primarily need a secure model-download repository.
• Foundry Local - useful when the target application/device supports Microsoft's local AI runtime.
• Azure IoT Edge - useful when these are managed IoT/industrial edge devices.

One practical point: make sure the model is actually suitable for the target hardware. A Hugging Face/PyTorch model might need conversion or quantization to something like ONNX or another optimized local format before deployment to CPU/NPU-constrained devices.

You also don't have to expose the model publicly. You can protect the download location using Microsoft Entra ID/managed identity, SAS tokens, or another authenticated distribution mechanism.

Sharing these references with you:

Microsoft Learn – Register and manage Azure ML models.

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-models?view=azureml-api-2

Microsoft Learn – Foundry Local for offline/on-device AI.

https://learn.microsoft.com/en-us/azure/foundry-local/what-is-foundry-local

Microsoft Learn – Machine learning inference with Azure IoT Edge.

https://learn.microsoft.com/en-us/azure/architecture/guide/iot/machine-learning-inference-iot-edge

Microsoft Learn – Azure ML registries for sharing/versioning models.

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-share-models-pipelines-across-workspaces-with-registries?view=azureml-api-2

Please "Accept the Answer" if this information helped you. This will help us and others in the community.

Hi Prasanna Sai Kommineni,
Greetings of the day!
Yes, Azure supports this scenario. You can upload your custom-trained Small Language Model (SLM) to Azure, allow edge devices to download it, and then run the model locally without an internet connection.

1. Azure Machine Learning – Recommended

You can use Azure Machine Learning to register and manage your custom SLM.

This is useful when you want to maintain different versions of your model, for example:

• SLM v1
• SLM v2
• SLM v3

You can keep the model files in Azure and allow your edge application to download the required version.

Azure ML also provides model versioning and management capabilities.

Microsoft Learn – Register and manage Azure ML models

2. Azure Blob Storage – Simple option

If your main requirement is simply to store the model and allow edge devices to download it, Azure Blob Storage is a good option.

You can store model files such as:

• GGUF
• ONNX
• Safetensors
• Tokenizer and configuration files

The edge device can download the model once and store it locally. After that, the device can perform inference without continuously connecting to Azure.

3. Foundry Local – For local/offline inference

If your target device is supported, Foundry Local can be used to run AI models directly on the device.

Microsoft Learn – Foundry Local

4. Azure IoT Edge – For IoT devices

For IoT or industrial edge devices, Azure IoT Edge can also be considered for deploying and running machine learning workloads locally.

The model can be deployed to the edge device, and inference can happen locally instead of sending data to the cloud.

Microsoft Learn – Machine learning inference with Azure IoT Edge

Recommended approach:

For a production scenario, we would recommend using Azure Machine Learning Model Registry to manage and version the custom SLM, and Azure Blob Storage or another suitable Azure storage option to distribute the model to the edge devices.

The edge device can download the model, store it locally, and use a suitable runtime such as Foundry Local, ONNX Runtime, or llama.cpp, depending on the model format and device capabilities.

Azure ML registries can also be used when you need to share and version models across different Azure ML workspaces.

Microsoft Learn – Azure ML registries