Answering my own question: As far as I can tell, this is not currently possible. DSS would have to provide a signed URL authorizing this request.

When downloading TRTH data from AWS, you get the pre-signed URL. The question could be how to copy an object from the pre-signed URL. I found a few answers via the stack overflow.

• How do I copy an object with presigned URL?

Someone has mentioned about a pre-signed PUT URL with the x-amz-copy-source header.

Can someone please update the latest situation in this regards?

What is the Best Method for DIRECTLY COPYING the VBD "Venue by Day" Data Files from Refinitiv AWS S3 Bucket to the Client's AWS S3 Bucket, without having to first download and then again upload these files individually from the client machine? What are the best practices for this particular type of AWS S3 direct copy method?

Please provide links to the example codes to accomplish the same.

Regards

@Christiaan Meihsl @Janik.Zikovsky @jirapongse.phuriphanvichai @veerapath.rungruengrayubkul

Hi Pankaj,

I have been able to do this partially - technically you are still downloading to the local machine but by passing the streams around you do it in a streaming way, from one inputStream to another, and avoid having to create a local file to then upload. In this code snippet awsURI is the signed URL returned by DSS :

    URLConnection urlConnection = awsURI.toURL().openConnection();
    // In some cases (reposted files) data as reported by DSS is incorrect. Happened on May 11, 2020.
    // Need to rely on HTTP headers instead
    long contentLength = urlConnection.getContentLengthLong();
    if (contentLength <= 0) {
      // But as a failure scenario, use the DSS data.
      contentLength = task.getFileSize();
    }
    // Same for MD5 sum
    String md5sum = urlConnection.getHeaderField("x-amz-meta-md5sum");
    if (StringUtils.isBlank(md5sum)) {
      md5sum = task.getContentMd5();
    }
    LOGGER.info("Starting download of {} with content length {}", awsURI.toURL(), contentLength);
    try (InputStream stream = urlConnection.getInputStream()) {
      String filename = FilenameUtils.getName(awsURI.toURL().getPath());
      ObjectMetadata metadata = new ObjectMetadata();
      metadata.setContentLength(contentLength);

      // Convert hex-encoded MD5 to raw bytes, then base64-encode
      byte[] binaryMd5 = DatatypeConverter.parseHexBinary(md5sum);
      String base64md5 = Base64.encodeBase64String(binaryMd5);
      metadata.setContentMD5(base64md5);
      // Save a simple MD5 sum because the ETag cannot be used in multi-part uploads.
      // Metadata has to start with "x-amz-meta-" to be accepted
      metadata.setHeader("x-amz-meta-" + VbdUtility.CHECKSUM_METADATA_NAME, task.getContentMd5());

      Date date = new Date();
      metadata.setLastModified(date);

      PutObjectRequest request = new PutObjectRequest(config.downloadBucket(), destKey, stream, metadata);
      request.setSdkClientExecutionTimeout(-1);  // disabled

      // Start the upload in the background, but then wait for completion
      Upload upload = transferManager.upload(request);
      upload.waitForCompletion();

      TransferState transferState = upload.getState();
      LOGGER.info("Transfer status of {} is {} and upload.isDone: {}", filename, transferState, upload.isDone());
      return transferState;
    }
  }

Hi Pankaj,

Yes, the upload's MD5 is validated - the AWS S3 SDK does this for you because of the "metadata.setContentMD5(base64md5);" call - see AWS S3 API docs for more details. I don't believe I've seen a case where the MD5 validation failed but there is a retry mechanism implemented so I may not have noticed.

Sincerely,

Janik Zikovsky