Read Octet Stream

My understanding is that the Java SDK sets by default a Content-Type: application/octet-stream header if none is provided by the user when adding an object.
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/AmazonS3Client.java#L4147

On the other hand, I've noticed that files uploaded via AWS CLI's s3api put-object .. get assigned a Content-Type of binary/octet-stream.
I suppose that this value is assigned by the S3 backend, because I couldn't find any Content-Type header set in the PUT request according to --debug logs.

Uploading a file via the S3 Web console produces a binary/octet-stream MIME type as well.
(For the record, Google Cloud Storage also uses binary/octet-stream.)

I was wondering if you had any comment on this difference between application/octet-stream and binary/octet-stream?

Hi don't believe the CLI actually sets binary/octet-stream. You can verify this by issuing a command like aws s3 cp MY_BINARY_FILE s3://MY_BUCKET --debug and see that it does not send a Content-Type header.

Per the S3 docs, the default is binary/octet-stream if the header is not set: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html

Oops sorry looks like I didn't read your question close enough! I'm not sure what the exact reason is for the discrepancy but I suspect it's because application/octet-stream is registered with the IANA, and binary/octet-stream does not seem to be. In any case, I don't think it's possible for us to change this now because it would be subtle breaking change for customers.

Hum, if the default behavior can't be changed (what kind of 'subtle change' do you expect by the way?), what about an option to specify the default MIME type?
Or is the only solution for the API user to explicitly set binary/octet-stream for each PUT operation?

As far as breaking change, customers may be relying on the fact that the default MIME for objects uploaded using the Java SDK to be application/octet-stream rather than binary/octet-stream.

Do you expect all of your objects to have binary/octet-stream? If so, you can use ClientConfiguration#withHeader. Note that this will override the value set on ObjectMetadata#setContentType so it won't work if the you need to be able to override the default value.

Another option might be a custom RequestHandler

We can also look into adding a default object MIME if that would be easiest for you.

Thanks for the tips on the various methods available.
I have full control of the code that uses the API so I can explicitly use ObjectMetadata#setContentType for each call, that's not a big deal.
Yet, I am curious what you meant in your last sentence.

Are you referring to

We can also look into adding a default object MIME if that would be easiest for you.

?

I was imagining just a new option like setDefaultObjectMimeType on S3ClientOptions that works sort of like ClientConfiguration#withHeader but at the S3 level so you can change mimetype per PUT call.

Yes, I think such an option would be useful. That would even give some API visibility to the fact that the Java SDK uses a 'non-standard' default MIME type.

By the way, speaking of 'application/octet-stream' header, I saw in debug traces that (all?) HTTP requests sent to S3 contain such a header, even in case of a GET or HEAD.
I suppose that's not forbidden per se according to the spec, but I find it strange.

I have a server that responds with binary data and an octet-stream header:

The response is a Buffer, so nock converts it to hex before storing in a file. I've truncated the response here and above, but it should be enough to get the idea.

Outlook 2003. Open Control Panel (Start>Settings>Control Panel) Select Add or Remove Programs. Select Microsoft Office/Outlook; Click the Change. Select the Reinstall or Repair button, click Next. Select Detect and Repair. Click Install and follow any prompts provided by the repair tool. Click here to see your Outlook Calendar on your Windows. Outlook 2010, Outlook 2007, Outlook 2003. Office 2013 & Office 2016. Use the Repair process in the Control Panel's Programs and Features or Windows 10 Settings. You can open it by typing Add or Remove Programs on the Start screen or in Search. In Windows 10, open the Control Panel then find Programs and Features or Add or Remove Programs. Jun 22, 2016  Outlook 2016 reinstall Following a reset of Windows 10 I had to reinstall Office 2016. To my amazement everything was returned including my files. Very pleased. Problem is that now Outlook 2016 will neither send nor receive email! Outlook / Windows 10 / Office 2016; Answer FR. FredGLawson Replied on June 22, 2016.

When reading back from the file, I expect it to be converted back into a Buffer via Buffer.from(response.body, 'hex'). Instead it gets converted into a normal Buffer, e.g. the first byte is 'f', the second byte is '0', instead of one byte 0xf0.

I can work around this for the time being by setting Content-Encoding: identity, which triggers the right nock logic. But it would seem preferable to e.g. read the application/octet-stream response header and try to convert from hex based on that.

Hey @kevinburke can you check if #1067 fixes your problem?

Apologies but I've since moved on from that engagement and probably won't have time to test :( best of luck!

A way to test would probably be to start with the JSON I put up above, read it back into memory, and verify the length of the buffer in memory matches the Content-Length header.

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue and add a reference to this one if it’s related. Thank you!