[ACCEPTED]-Java -- How to read an unknown number of bytes from an inputStream (socket/socketServer)?-inputstream

Accepted answer
Score: 27

You need to expand the buffer as needed, by reading 2 in chunks of bytes, 1024 at a time as in 1 this example code I wrote some time ago

    byte[] resultBuff = new byte[0];
    byte[] buff = new byte[1024];
    int k = -1;
    while((k = sock.getInputStream().read(buff, 0, buff.length)) > -1) {
        byte[] tbuff = new byte[resultBuff.length + k]; // temp buffer size = bytes already read + bytes last read
        System.arraycopy(resultBuff, 0, tbuff, 0, resultBuff.length); // copy previous bytes
        System.arraycopy(buff, 0, tbuff, resultBuff.length, k);  // copy current lot
        resultBuff = tbuff; // call the temp buffer as your result buff
    }
    System.out.println(resultBuff.length + " bytes read.");
    return resultBuff;
Score: 14

Assuming the sender closes the stream at 1 the end of the data:

ByteArrayOutputStream baos = new ByteArrayOutputStream();

byte[] buf = new byte[4096];
while(true) {
  int n = is.read(buf);
  if( n < 0 ) break;
  baos.write(buf,0,n);
}

byte data[] = baos.toByteArray();
Score: 11

Read an int, which is the size of the next 9 segment of data being received. Create a 8 buffer with that size, or use a roomy pre-existing 7 buffer. Read into the buffer, making sure 6 it is limited to the aforeread size. Rinse 5 and repeat :)

If you really don't know the size 4 in advance as you said, read into an expanding 3 ByteArrayOutputStream as the other answers 2 have mentioned. However, the size method 1 really is the most reliable.

Score: 8

The simple answer is:

byte b[] = byte[BIG_ENOUGH];
int nosRead = sock.getInputStream().read(b);

where BIG_ENOUGH is big enough.


But 28 in general there is a big problem with this. A 27 single read call is not guaranteed to return all that the 26 other end has written.

  • If the nosRead value is BIG_ENOUGH, your 25 application has no way of knowing for sure 24 if there are more bytes to come; the other 23 end may have sent exactly BIG_ENOUGH bytes ... or 22 more than BIG_ENOUGH bytes. In the former case, you 21 application will block (for ever) if you 20 try to read. In the latter case, your application 19 has to do (at least) another read to get the 18 rest of the data.

  • If the nosRead value is less than 17 BIG_ENOUGH, your application still doesn't know. It might 16 have received everything there is, part 15 of the data may have been delayed (due to 14 network packet fragmentation, network packet 13 loss, network partition, etc), or the other 12 end might have blocked or crashed part 11 way through sending the data.

The best answer 10 is that EITHER your application needs to know 9 beforehand how many bytes to expect, OR the 8 application protocol needs to somehow tell the 7 application how many bytes to expect or 6 when all bytes have been sent.

Possible 5 approaches are:

  • the application protocol uses fixed message sizes (not applicable to your example)
  • the application protocol message sizes are specified in message headers
  • the application protocol uses end-of-message markers
  • the application protocol is not message based, and the other end closes the connection to say that is the end.

Without one of these strategies, your 4 application is left to guess, and is liable 3 to get it wrong occasionally.

Then you use 2 multiple read calls and (maybe) multiple 1 buffers.

Score: 8

Without re-inventing the wheel, using Apache 5 Commons:

IOUtils.toByteArray(inputStream);

For example, complete code with 4 error handling:

    public static byte[] readInputStreamToByteArray(InputStream inputStream) {
    if (inputStream == null) {
        // normally, the caller should check for null after getting the InputStream object from a resource
        throw new FileProcessingException("Cannot read from InputStream that is NULL. The resource requested by the caller may not exist or was not looked up correctly.");
    }
    try {
        return IOUtils.toByteArray(inputStream);
    } catch (IOException e) {
        throw new FileProcessingException("Error reading input stream.", e);
    } finally {
        closeStream(inputStream);
    }
}

private static void closeStream(Closeable closeable) {
    try {
        if (closeable != null) {
            closeable.close();
        }
    } catch (Exception e) {
        throw new FileProcessingException("IO Error closing a stream.", e);
    }
}

Where FileProcessingException is your app-specific 3 meaningful RT exception that will travel 2 uninterrupted to your proper handler w/o 1 polluting the code in between.

Score: 1

Stream all Input data into Output stream. Here 1 is working example:

    InputStream inputStream = null;
    byte[] tempStorage = new byte[1024];//try to read 1Kb at time
    int bLength;
    try{

        ByteArrayOutputStream outputByteArrayStream =  new ByteArrayOutputStream();     
        if (fileName.startsWith("http"))
            inputStream = new URL(fileName).openStream();
        else
            inputStream = new FileInputStream(fileName);            

        while ((bLength = inputStream.read(tempStorage)) != -1) {
                outputByteArrayStream.write(tempStorage, 0, bLength);
        }
        outputByteArrayStream.flush();
        //Here is the byte array at the end
        byte[] finalByteArray = outputByteArrayStream.toByteArray();
        outputByteArrayStream.close();
        inputStream.close();
    }catch(Exception e){
        e.printStackTrace();
        if (inputStream != null) inputStream.close();
    }
Score: 0

Either:

  1. Have the sender close the socket 5 after transferring the bytes. Then at the 4 receiver just keep reading until EOS.

  2. Have 3 the sender prefix a length word as per Chris's 2 suggestion, then read that many bytes.

  3. Use 1 a self-describing protocol such as XML, Serialization, ...

Score: 0

Use BufferedInputStream, and use the available() method which returns 3 the size of bytes available for reading, and 2 then construct a byte[] with that size. Problem 1 solved. :)

BufferedInputStream buf = new BufferedInputStream(is);  
int size = buf.available();
Score: 0

Here is a simpler example using ByteArrayOutputStream...

        socketInputStream = socket.getInputStream();
        int expectedDataLength = 128; //todo - set accordingly/experiment. Does not have to be precise value.
        ByteArrayOutputStream baos = new ByteArrayOutputStream(expectedDataLength);
        byte[] chunk = new byte[expectedDataLength];
        int numBytesJustRead;
        while((numBytesJustRead = socketInputStream.read(chunk)) != -1) {
            baos.write(chunk, 0, numBytesJustRead);
        }
        return baos.toString("UTF-8");

However, if 6 the server does not return a -1, you will 5 need to detect the end of the data some 4 other way - e.g., maybe the returned content 3 always ends with a certain marker (e.g., ""), or 2 you could possibly solve using socket.setSoTimeout(). (Mentioning 1 this as it is seems to be a common problem.)

Score: 0

This is both a late answer and self-advertising, but 2 anyone checking out this question may want 1 to take a look here: https://github.com/GregoryConrad/SmartSocket

Score: 0

This question is 7 years old, but i had 46 a similiar problem, while making a NIO and 45 OIO compatible system (Client and Server 44 might be whatever they want, OIO or NIO).

This 43 was quit the challenge, because of the blocking 42 InputStreams.

I found a way, which makes 41 it possible and i want to post it, to help 40 people with similiar problems.

Reading a 39 byte array of dynamic sice is done here 38 with the DataInputStream, which kann be simply wrapped 37 around the socketInputStream. Also, i do 36 not want to introduce a specific communication 35 protocoll (like first sending the size of 34 bytes, that will be send), because i want 33 to make this as vanilla as possible. First 32 of, i have a simple utility Buffer class, which 31 looks like this:

import java.util.ArrayList;
import java.util.List;

public class Buffer {

    private byte[] core;
    private int capacity;

    public Buffer(int size){
        this.capacity = size;
        clear();
    }

    public List<Byte> list() {
        final List<Byte> result = new ArrayList<>();
        for(byte b : core) {
            result.add(b);
        }

        return result;
    }

    public void reallocate(int capacity) {
        this.capacity = capacity;
    }

    public void teardown() {
        this.core = null;
    }

    public void clear() {
        core = new byte[capacity];
    }

    public byte[] array() {
        return core;
    }
}

This class only exists, because 30 of the dumb way, byte <=> Byte autoboxing 29 in Java works with this List. This is not 28 realy needed at all in this example, but 27 i did not want to leave something out of 26 this explanation.

Next up, the 2 simple, core 25 methods. In those, a StringBuilder is used 24 as a "callback". It will be filled 23 with the result which has been read and 22 the amount of bytes read will be returned. This 21 might be done different of course.

private int readNext(StringBuilder stringBuilder, Buffer buffer) throws IOException {
    // Attempt to read up to the buffers size
    int read = in.read(buffer.array());
    // If EOF is reached (-1 read)
    // we disconnect, because the
    // other end disconnected.
    if(read == -1) {
        disconnect();
        return -1;
    }
    // Add the read byte[] as
    // a String to the stringBuilder.
    stringBuilder.append(new String(buffer.array()).trim());
    buffer.clear();

    return read;
}

private Optional<String> readBlocking() throws IOException {
    final Buffer buffer = new Buffer(256);
    final StringBuilder stringBuilder = new StringBuilder();
    // This call blocks. Therefor
    // if we continue past this point
    // we WILL have some sort of
    // result. This might be -1, which
    // means, EOF (disconnect.)
    if(readNext(stringBuilder, buffer) == -1) {
        return Optional.empty();
    }
    while(in.available() > 0) {
        buffer.reallocate(in.available());
        if(readNext(stringBuilder, buffer) == -1) {
            return Optional.empty();
        }
    }

    buffer.teardown();

    return Optional.of(stringBuilder.toString());
}

The first 20 method readNext will fill the buffer, with byte[] from 19 the DataInputStream and return the amount 18 bytes read this way.

In the secon method, readBlocking, i 17 utilized the blocking nature, not to worry 16 about consumer-producer-problems. Simply readBlocking will block, untill a new 15 byte-array is received. Before we call this 14 blocking method, we allocate a Buffer-size. Note, i 13 called reallocate after the first read (inside 12 the while loop). This is not needed. You 11 can safely delete this line and the code 10 will still work. I did it, because of the 9 uniqueness of my problem.

The 2 things, i 8 did not explain in more detail are: 1. in 7 (the DataInputStream and the only short 6 varaible here, sorry for that) 2. disconnect 5 (your disconnect routine)

All in all, you 4 can now use it, this way:

// The in has to be an attribute, or an parameter to the readBlocking method
DataInputStream in = new DataInputStream(socket.getInputStream());
final Optional<String> rawDataOptional = readBlocking();
rawDataOptional.ifPresent(string -> threadPool.execute(() -> handle(string)));

This will provide 3 you with a way of reading byte arrays of 2 any shape or form over a socket (or any 1 InputStream realy). Hope this helps!

More Related questions