While trying to download files ( in GB size), we are getting the above message. We are using curl command to retrive the data. Anyone faced same issue earlier? Is there any solution for the same?
Command is as below.
curl -k -X GET -H 'Authorization: Token xxxxxxxxxxxxxxxxxxxxxxx' -o NAS-2017-07-25-NORMALIZEDMP-Data-1-of-1.csv.gz https://10.192.6.221/RestApi/v1/StandardExtractions/UserPackageDeliveries%28%270x05cf06a3fedb3026%27%29/%24value
It seems like the connection has been closed, while the file has been downloading. It might be connection issue between your machine and the firewall.
Curl provides an option to resume download. This option may help you continue the download, once the issue occurs for huge file. Below is the sample command. It can check download progress from the existing file name, and then continue downloading the data.
curl -k -H 'Authorization: Token xxxxxxxxxxxxxxxxxxxxxxx' -o NAS-2017-07-25-NORMALIZEDMP-Data-1-of-1.csv.gz -C - https://10.192.6.221/RestApi/v1/StandardExtractions/UserPackageDeliveries%28%270x05cf06a3fedb3026%27%29/%24value
Hope this helps.
Hi, Thanks for the reply, but it actually doesnot resolve my problem. Your suggestion is to download the file with -C option which will resume the download from the same place where it left. Now I have a few questions here.
1. In worst case scenario, what is the guarantee that it will not terminate the download again after resuming it?
2. If it terminates too, then how many times we must try to complete the download?
3. How will we know that the entire files got downloaded?
Hope I am able to elaborate my problem. Please help.
Yes. The curl option should not solve the issue. It is a workaround once the issue occurs, so you do not need to re-download the file from begining. The issue can reoccur again on this command, once there is a connection closed issue. If the entire file gets the download, the command will exit without any error message.
I am able to download the same file with the curl command in my environment, so I suspect that the issue is in the network and firewall in your environment. Could you contact your administrator to verify your network and firewall?
I got the same error when using netsh command to do port forwarding.
C:\netsh interface portproxy add v4tov4 listenport=443 listenaddress=127.0.0.1 connectport=443 connectaddress=hosted.datascopeapi.reuters.com
Then, I used this curl command on Windows to download a VBD file.
curl -k -X GET -H "Authorization: Token <token>" -o NAS-2017-07-25-NORMALIZEDMP-Data-1-of-1.csv.gz https://127.0.0.1/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05cf06a3fedb3026')/$value
After that, I cut the connection between netsh and datascope.
The curl command showed the following error:
curl: (18) transfer closed with 1789333843 bytes remaining to read
Like Veerapath mentioned, you need to contact the firewall admin to verify the connection and settings.
To remove the netsh port forwarding, I used the following command.
C:\netsh interface portproxy delete v4tov4 listenport=443 listenaddress=127.0.0.1
