@microsoft/azure-data-factory-utilities fails to download script from internet in securized private build agent

Question

@microsoft/azure-data-factory-utilities fails to download script from internet in securized private build agent

Jean-Christophe Gauvreau 20

In this link (https://www.npmjs.com/package/@microsoft/azure-data-factory-utilities?activeTab=readme) it has been mentioned that the package @microsoft/azure-data-factory-utilities downloads a JavaScript (https://adf.azure.com/assets/cmd-api/main.js) from internet. In a secured work environment, it is not allowed to download bundles from Internet into private build agents.

Therefore, is it possible to add the dependencies into the same package @microsoft/azure-data-factory-utilities so that we can validate and export without download the main.js file from internet in private build agents.

I am successfull validating and exporting locally, but it doesn't work from the build agent.

Also, this issue is related : https://github.com/azure/azure-datafactory/issues/635

Locked Question. You can vote on whether it's helpful, but you can't add comments or replies or follow the question.

Anonymous

2024-01-09T08:26:52.01+00:00
@Jean-Christophe Gauvreau

Thank you for reaching out to the MS Q&A forum with your query.

Regarding your question, it is not possible to add the dependencies into the same package @microsoft/azure-data-factory-utilities to validate and export without downloading the main.js file from the internet in private build agents. The package requires the main.js file to function correctly, and it cannot be modified to include the file.

However, you can try the following workaround to resolve the issue:

Download the main.js file from the internet and save it to a local directory.

Modify the package.json file of @microsoft/azure-data-factory-utilities to point to the local directory where the main.js file is saved.

Validate and export the package using the modified package.json file.

The package @microsoft/azure-data-factory-utilities does indeed download a JavaScript file from the internet, which can be a problem in a secured work environment

However, there is an option to point to an existing downloaded bundle’s share or direct path. This might be useful when there is a download issue due to networking challenges such as proxy requirements_._

You can use the --path argument to specify the location of the downloaded bundle. Here is an example command:

npm run start --path \\\\server\\share\\main.js export C:\\DataFactories\\DevDataFactory /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/testResourceGroup/providers/Microsoft.DataFactory/factories/DevDataFactory ArmTemplateOutput

In this command, \\\\server\\share\\main.js is the path to the downloaded main.js file

Regarding the issue you mentioned, it seems to be related to the same problem you are facing. You can follow the discussion on the GitHub issue to see if there are any updates or solutions.

I hope this helps. Let us know if you have any further questions or concerns.
Anonymous

2024-01-10T10:38:51.1633333+00:00

@Jean-Christophe Gauvreau We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Answer accepted by question author

Anonymous

@Jean-Christophe Gauvreau
Welcome to Microsoft Q&A platform and thanks for posting your question.

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Ask: is it possible to add the dependencies into the same package @microsoft/azure-data-factory-utilities so that we can validate and export without download the main.js file from internet in private build agents?

I am successful validating and exporting locally, but it doesn't work from the build agent.

Solution: 2 possible solutions:

1 - Download the script manually, upload it to our internal artifact directory, then download the artifact to the agent from the internal artifact directory.

2 - Open/whitelist the communication to this specific webpage from the private agent https://adf.azure.com/assets/cmd-api/main.js

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Anonymous

2024-01-12T07:43:45.8433333+00:00

@Jean-Christophe Gauvreau Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

3 additional answers

Anonymous

2024-01-09T08:26:52.01+00:00

@Jean-Christophe Gauvreau

Thank you for reaching out to the MS Q&A forum with your query.

Regarding your question, it is not possible to add the dependencies into the same package @microsoft/azure-data-factory-utilities to validate and export without downloading the main.js file from the internet in private build agents. The package requires the main.js file to function correctly, and it cannot be modified to include the file.

However, you can try the following workaround to resolve the issue:

Download the main.js file from the internet and save it to a local directory.

Modify the package.json file of @microsoft/azure-data-factory-utilities to point to the local directory where the main.js file is saved.

Validate and export the package using the modified package.json file.

The package @microsoft/azure-data-factory-utilities does indeed download a JavaScript file from the internet, which can be a problem in a secured work environment

However, there is an option to point to an existing downloaded bundle’s share or direct path. This might be useful when there is a download issue due to networking challenges such as proxy requirements_._

You can use the --path argument to specify the location of the downloaded bundle. Here is an example command:

npm run start --path \\\\server\\share\\main.js export C:\\DataFactories\\DevDataFactory /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/testResourceGroup/providers/Microsoft.DataFactory/factories/DevDataFactory ArmTemplateOutput

In this command, \\\\server\\share\\main.js is the path to the downloaded main.js file

Regarding the issue you mentioned, it seems to be related to the same problem you are facing. You can follow the discussion on the GitHub issue to see if there are any updates or solutions.

I hope this helps. Let us know if you have any further questions or concerns.
Anonymous

2024-01-10T10:38:51.1633333+00:00

@Jean-Christophe Gauvreau We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Anonymous

2024-01-12T07:43:45.8433333+00:00

@Jean-Christophe Gauvreau Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Bánhidy Attila 20

...to be honest, the main question would be: how can I solve, without any patch or packgae related file modification, that process not try to download main.js file, instead use mine downloaded main.js file...
If I added --path flag to npm run build (or start) command after it, tried to download the main.js file...what is not possible...

0 comments

Answer 2

Hi,

I'm not sure whether this issue is still active, but I'm currently fighting a very similar problem.

My company's network is highly restricted, so the pipeline is not able to download the main.js file.

What I did:

Downloaded the file manually from: https://adf.azure.com/assets/cmd-api/main.js

Added it to the pipeline repository.

My package.json looks like this because I can only download the utility package from our Nexus repository:

{
 "scripts": {
 "build": "node node_modules/@microsoft/azure-data-factory-utilities/lib/index"
 },
 "dependencies": {
 "@microsoft/azure-data-factory-utilities": "^1.0.0"
 }
}

The installed version is 1.0.3.

So the utility package is installed locally, not globally.

I also patched bundle.manager.js:

set -euo pipefail

BUNDLE_FILE="$(Build.SourcesDirectory)/node_modules/@microsoft/azure-data-factory-utilities/lib/bundle.manager.js"

sed -i "s|if (!fs.existsSync(BundleManager.defaultBundleDir)) {|this._runCommand('$(Build.SourcesDirectory)/_scripts/AdfPublishTool/main.js', args); return;\\
if (!fs.existsSync(BundleManager.defaultBundleDir)) {|g" "$BUNDLE_FILE"

grep -n "_scripts/AdfPublishTool/main.js" "$BUNDLE_FILE"

I also tried using the --path option to specify the location of my downloaded main.js file. However, when using --path, the process still attempts to download main.js, which is not possible in our environment.

At this point I am quite confused about what the correct solution should be. I have already spent several days trying to solve this issue, but I still have not managed to get it working.

If you need any additional information and it is not sensitive, I will gladly provide it.

Any help would be greatly appreciated.

Thanks!

Erland Sommarskog 137.1K Reputation points MVP Volunteer Moderator

2026-08-25T18:13:11.5566667+00:00

You should ask a new Question and not piggyback on old threads. As a moderator I will lock this thread.

Answer 3

Jean-Christophe Gauvreau 20

2 possible solutions :

1 - Download the script manually, upload it to our internal artifact directory, then download the artifact to the agent from the internal artifact directory.

2 - Open/whitelist the communication to this specific webpage from the private agent https://adf.azure.com/assets/cmd-api/main.js

Anonymous

2024-01-11T04:54:04.8733333+00:00

@Jean-Christophe Gauvreau
Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.