Buffer slicing

Theslice()method creates a kind ofsub-bufferfrom an existing buffer. That is, it creates a new buffer that shares its data with a portion of the original buffer.

This is best explained with an example. Let's start by creating aByteBufferof length 10:

ByteBuffer buffer = ByteBuffer.allocate( 10 );

We fill this buffer with data, putting the numbernin slotn:

for (int i=0; i<buffer.capacity(); ++i) {

buffer.put( (byte)i );

}

Now we'llslicethe buffer to create a sub-buffer that covers slots 3 through 6. In a sense, the sub-buffer is like awindowonto the original buffer.

You specify the start and end of the window by setting thepositionandlimitvalues, and then call theBuffer's slice()method:

buffer.position( 3 );

buffer.limit( 7 );

ByteBuffer slice = buffer.slice();

sliceis a sub-buffer ofbuffer. However,sliceandbuffershare the same underlying data array, as we'll see in the next section.

Buffer slicing and data sharing

We've created a sub-buffer of our original buffer, and we know that the two buffers and the sub-buffers share the same underlying data array. Let's see what this means.

We run through the sub-buffer, and alter each element by multiplying it by 11. This changes, for example, a 5 into a 55.

for (int i=0; i<slice.capacity(); ++i) {

byte b = slice.get( i );

b *= 11;

slice.put( i, b );

}

Finally, let's take a look at the contents of the original buffer:

buffer.position( 0 );

buffer.limit( buffer.capacity() );

while (buffer.remaining()> 0) {

System.out.println( buffer.get() );

}

The result shows that only the elements in the window of the sub-buffer were changed:

$ java SliceBuffer

1

2

33

44

55

66

7

8

9

Slice buffers are excellent for facilitating abstraction. You can write your functions to process an entire buffer, and if you find you want to apply that process to a sub-buffer, you can just take a slice of the main buffer and pass that to your function. This is easier than writing your functions to take additional parameters specifying what portion of the buffer should be acted upon.

Read-only buffers

Read-only buffers are very simple -- you can read them, but you can't write to them. You can turn any regular buffer into a read-only buffer by calling itsasReadOnlyBuffer()method, which returns a new buffer that is identical to the first (and shares data with it), but is read-only.

Read-only buffers are useful for protecting data. When you pass a buffer to a method of some object, you really have no way of knowing if that method is going to try to modify the data in the buffer. Creating a read-only bufferguaranteesthat the buffer won't be modified.

You cannot convert a read-only buffer to a writable buffer.

Direct and indirect buffers

Another useful kind ofByteBufferis the direct buffer. Adirect bufferis one whose memory is allocated in a special way to increase I/O speed.

Actually, the exact definition of a direct buffer is implementation-dependent. Sun's documentation has this to say about direct buffers:

Given a direct byte buffer, the Java virtual machine will make a best effort to perform native I/O operations directly upon it. That is, it will attempt to avoid copying the buffer's content to (or from) an intermediate buffer before (or after) each invocation of one of the underlying operating system's native I/O operations.

You can see direct buffers in action in the example program FastCopyFile.java, which is a version of CopyFile.java that uses direct buffers for increased speed.

You can also create a direct buffer using memory-mapped files.

Memory-mapped file I/O

Memory-mapped file I/O is a method for reading and writing file data that can be a great deal faster than regular stream- or channel-based I/O.

Memory-mapped file I/O is accomplished by causing the data in a file to magically appear as the contents of a memory array. At first, this sounds like it simply means reading the entire file into memory, but in fact it does not. In general, only the parts of the file that you actually read or write are brought, ormapped, into memory.

Memory-mapping isn't really magical, or all that uncommon. Modern operating systems generally implement filesystems by mapping portions of a file into portions of memory, doing so on demand. The Java memory-mapping system simply provides access to this facility if it is available in the underlying operating system.

Although they are fairly simple to create, writing to memory-mapped files can be dangerous. By the simple act of changing a single element of an array, you are directly modifying the file on disk. There is no separation between modifying the data and saving it to a disk.

Mapping a file into memory

The easiest way to learn about memory mapping is by example. In the example below, we want to map aFileChannel(all or a portion of it) into memory. For this we use theFileChannel.map()method. The following line of code maps the first 1024 bytes of a file into memory:

MappedByteBuffer mbb = fc.map( FileChannel.MapMode.READ_WRITE,

0, 1024 );

Themap()method returns aMappedByteBuffer, which is a subclass ofByteBuffer. Thus, you can use the newly-mapped buffer as you would any otherByteBuffer, and the operating system will take care of doing the mapping for you, on demand.

Scattering and gathering

Scattering and gathering overview

Scatter/gather I/O is a method of reading and writing that uses multiple buffers, rather than a single buffer, to hold data.

A scattering read is like a regular channel read, except that it reads data into an array of buffers rather than a single buffer. Likewise, a gathering write writes data from an array of buffers rather than a single buffer.

Scatter/gather I/O is useful for dividing a data stream into separate sections, which can help implement complicated data formats.

Scatter/gather I/O

Channels can optionally implement two new interfaces:ScatteringByteChannelandGatheringByteChannel. AScatteringByteChannelis a channel that has two additional read methods:

long read( ByteBuffer[] dsts );

long read( ByteBuffer[] dsts, int offset, int length );

Theselong read()methods are rather like the standardreadmethods, except that instead of taking a single buffer they take an array of buffers.

In ascattering read, the channel fills up each buffer in turn. When it fills up one buffer, it starts filling the next one. In a sense, the array of buffers is treated like one big buffer.

Applications of scatter/gather

Scatter/gather I/O is useful for dividing a piece of data into sections. For example, you might be writing a networking application that uses message objects, and each message is divided into a fixed-length header and a fixed-length body. You create one buffer that's just big enough for the header, and another buffer that's just big enough for the body. When you put these two in an array and read into them using a scattering read the header and body will be neatly divided between two buffers.

The convenience that we already get from buffers applies to buffer arrays as well. Because each buffer keeps track of how much room it has for more data, the scattering read will automatically find the first buffer with room in it. After that's filled up, it moves onto the next one.

Gathering writes

Agathering writeis like a scattering read, only for writing. It too has methods that take an array of buffers:

long write( ByteBuffer[] srcs );

long write( ByteBuffer[] srcs, int offset, int length );

A gathering write is useful for forming a single data stream from a group of separate buffers. In keeping with the message example described above, you could use a gathering write to automatically assemble the components of a network message into a single data stream for transmission across a network.

You can see scattering reads and gathering writes in action in the example program UseScatterGather.java.

File locking

File locking overview

File-locking can be confusing at first. Itsoundslike it refers to preventing programs or users from accessing a particular file. In fact, file locks are just like regular Java object locks -- they areadvisorylocks. They don't prevent any kind of data access; instead, they allow different parts of a system to coordinate through the sharing and acquisition of locks.

You can lock an entire file or a portion of a file. If you acquire an exclusive lock, then no one else can acquire a lock on that same file or portion of a file. If you acquire a shared lock, then others can acquire shared locks, but not exclusive locks, on that same file or portion of a file. File locking is not always done for the purpose of protecting data. For example, you might temporarily lock a file to ensure that a particular write operation is made atomically, without interference from other programs.

Most operating systems provide filesystem locks, but they don't all do it in the same way. Some implementations provide shared locks, while others provide only exclusive locks. And some implementations do, in fact, make a locked portion of a file inaccessible, although most do not.

In this section, you'll learn how to do a simple file locking procedure in NIO, and we'll also talk about some of the ways you can ensure your locked files are as portable as they can be.

Locking a file

To acquire a lock on a portion of a file, you call thelock()method on an openFileChannel. Note that you must open the file for writing if you want to acquire an exclusive lock.

RandomAccessFile raf = new RandomAccessFile( "usefilelocks.txt", "rw" );

FileChannel fc = raf.getChannel();

FileLock lock = fc.lock( start, end, false );

After you have the lock, you can carry out any sensitive operations that you need to, and then release the lock:

lock.release();

After you have released the lock, any other programs trying to acquire the lock will have a chance to do so.

The example program, UseFileLocks.java, is meant to be run in parallel with itself. This program acquires a lock on a file, holds it for three seconds, and then releases it. If you run several instances of this program at the same time, you can see each one acquiring the lock in turn.

File locking and portability

File locking can be tricky business, especially given the fact that different operating systems implement locks differently. The following guidelines will help you keep your code as portable as possible:

Only use exclusive locks.

Treat all locks as advisory.

Networking and asynchronous I/O

Networking and asynchronous I/O overview

Networking is an excellent foundation for learning about asynchronous I/O, which is of course essential knowledge for anyone doing input/output procedures in the Java language. Networking in NIO isn't much different from any other operation in NIO -- it relies on channels and buffers, and you acquire the channels from the usualInputStreams andOutputStreams.

In this section we'll start with the fundamentals of asynchronous I/O -- what it is and what it is not -- and then move on to a more hands-on, procedural example.

Asynchronous I/O

Asynchronous I/O is a method for reading and writing datawithout blocking. Normally, when your code makes aread()call, the code blocks until there is data to be read. Likewise, awrite()call will block until the data can be written.

Asynchronous I/O calls, on the other hand, do not block. Instead, you register your interest in a particular I/O event -- the arrival of readable data, a new socket connection, and so on -- and the system tells you when such an event occurs.

One of the advantages of asynchronous I/O is that it lets you do I/O from a great many inputs and outputs at the same time. Synchronous programs often have to resort to polling, or to the creation of many, many threads, to deal with lots of connections. With asynchronous I/O, you can listen for I/O events on an arbitrary number of channels, without polling and without extra threads.

We'll see asynchronous I/O in action by examining an example program called MultiPortEcho.java. This program is like the traditionalecho server, which takes network connections and echoes back to them any data they might send. However, it has the added feature that it can listen on multiple ports at the same time, and deal with connections from all of those ports. And it does it all in a single thread.

Selectors

The explanation in this section corresponds to the implementation of thego()method in the source code forMultiPortEcho, so take a look at the source for a fuller picture of what is going on.

The central object in asynchronous I/O is called theSelector. ASelectoris where you register your interest in various I/O events, and it is the object that tells you when those events occur.

So, the first thing we need to do is create aSelector:

Selector selector = Selector.open();

Later on, we will call theregister()method on various channel objects, in order to register our interest in I/O events happening inside those objects. The first argument toregister()is always theSelector.

Opening a ServerSocketChannel

In order to receive connections, we need aServerSocketChannel. In fact, we need one for each of the ports on which we are going to listen. For each of the ports, we open aServerSocketChannel, as shown here:

ServerSocketChannel ssc = ServerSocketChannel.open();

ssc.configureBlocking( false );

ServerSocket ss = ssc.socket();

InetSocketAddress address = new InetSocketAddress( ports[i] );

ss.bind( address );

The first line creates a newServerSocketChanneland the last three lines bind it to the given port. The second line sets theServerSocketChannelto benon-blocking. We must call this method on every socket channel that we're using; otherwise asynchronous I/O won't work.

Selection keys

Our next step is to register the newly openedServerSocketChannelswith ourSelector. We do this using the ServerSocketChannel.register() method, as shown below:

SelectionKey key = ssc.register( selector, SelectionKey.OP_ACCEPT );

The first argument toregister()is always theSelector. The second argument,OP_ACCEPT, here specifies that we want to listen foracceptevents -- that is, the events that occur when a new connection is made. This is the only kind of event that is appropriate for aServerSocketChannel.

Note the return value of the call toregister(). ASelectionKeyrepresents this registration of this channel with thisSelector. When aSelectornotifies you of an incoming event, it does this by supplying theSelectionKeythat corresponds to that event. TheSelectionKeycan also be used to de-register the channel.

The inner loop

Now that we have registered our interest in some I/O events, we enter the main loop. Just about every program that usesSelectorsuses an inner loop much like this one:

int num = selector.select();

Set selectedKeys = selector.selectedKeys();

Iterator it = selectedKeys.iterator();

while (it.hasNext()) {

SelectionKey key = (SelectionKey)it.next();

// ... deal with I/O event ...

}

First, we call theselect()method of ourSelector. This method blocks until at least one of the registered events occurs. When one or more events occur, theselect()method returns the number of events that occurred.

Next, we call theSelector'sselectedKeys()method, which returns aSetof theSelectionKeyobjects for which events have occurred.

We process the events by iterating through theSelectionKeysand dealing with each one in turn. For eachSelectionKey, you must determine what I/O event has happened and which I/O objects have been impacted by that event.

Listening for new connections

At this point in the execution of our program, we've only registeredServerSocketChannels, and we have only registered them for "accept" events. To confirm this, we call thereadyOps()method on ourSelectionKeyand check to see what kind of event has occurred:

if ((key.readyOps() & SelectionKey.OP_ACCEPT)

== SelectionKey.OP_ACCEPT) {

// Accept the new connection

// ...

}

Sure enough, thereadOps()method tells us that the event is a new connection.

Accepting a new connection

Because we know there is an incoming connection waiting on this server socket, we can safely accept it; that is, without fear that theaccept()operation will block:

ServerSocketChannel ssc = (ServerSocketChannel)key.channel();

SocketChannel sc = ssc.accept();

Our next step is to configure the newly-connectedSocketChannelto be non-blocking. And because the purpose of accepting this connection is to read data from the socket, we must also register theSocketChannelwith ourSelector, as shown below:

sc.configureBlocking( false );

SelectionKey newKey = sc.register( selector, SelectionKey.OP_READ );

Note that we've registered theSocketChannelforreadingrather thanacceptingnew connections, using theOP_READargument toregister().

Removing the processed SelectionKey

Having processed theSelectionKey, we're almost ready to return to the main loop. But first we must remove the processedSelectionKeyfrom the set of selected keys. If we do not remove the processed key, it will still be present as an activated key in the main set, which would lead us to attempt to process it again. We call the iterator'sremove()method to remove the processedSelectionKey:

it.remove();

Now we're set to return to the main loop and receive incoming data (or an incoming I/O event) on one of our sockets.

Incoming I/O

When data arrives from one of the sockets, it triggers an I/O event. This causes the call toSelector.select(), in our main loop, to return with an I/O event or events. This time, theSelectionKeywill be marked as anOP_READevent, as shown below:

} else if ((key.readyOps() & SelectionKey.OP_READ)

== SelectionKey.OP_READ) {

// Read the data

SocketChannel sc = (SocketChannel)key.channel();

// ...

}

As before, we get the channel in which the I/O event occurred and process it. In this case, because this is an echo server, we just want to read the data from the socket and send it right back. See the source code (MultiPortEcho.java) inResourcesfor details on this process.

Back to the main loop

Each time we return to the main loop we call theselect()method on ourSelector, and we get a set ofSelectionKeys. Each key represents an I/O event. We process the events, remove theSelectionKeys from the selected set, and go back to the top of the main loop.

This program is a bit simplistic, since it aims only to demonstrate the techniques involved in asynchronous I/O. In a real application, you would need to deal with closed channels by removing them from theSelector. And you would probably want to use more than one thread. This program can get away with a single thread because it's only a demo, but in a real-world scenario it might make more sense to create a pool of threads for taking care of the time-consuming portions of I/O event processing.

Character sets

Character sets overview

According to Sun's documentation, aCharsetis "a named mapping between sequences of sixteen-bit Unicode characters and sequences of bytes." In practice, aCharsetlets you read and write character sequences in the most portable way possible.

The Java language is defined as being based on Unicode. In practice, however, many people write programs under the assumption that a single character is represented on disk, or in a network stream, as a single byte. This assumption works in many cases, but not all, and as computers become more Unicode-friendly, it becomes less true every day.

In this section, we'll see how to useCharsetsto process textual data in conformance with modern text formats. The sample program we'll work with here is rather simple; nevertheless, it touches on all the crucial aspects of usingCharsets: creating aCharsetfor a given character encoding, and using thatCharsetto decode and encode text data.

Encoders/decoders

To read and write text, we are going to useCharsetDecoders andCharsetEncoders, respectively. There's a good reason why these are calledencodersanddecoders. Acharacterno longer represents a particular bit-pattern, but rather an entity within a character system. Thus, characters represented by an actual bit pattern must therefore be represented in some particularencoding.

ACharsetDecoderis used to convert the bit-by-bit representation of a string of characters into actualcharvalues. Likewise, aCharsetEncoderis used to convert the characters back to bits.

Next, we'll take a look at a program that reads and writes data using these objects.

The right way to process text

We'll take a look now at the example program, UseCharsets.java. This program is very simple -- it reads some text from one file, and writes it to another file. But it treats the data as textual data, and reads it into aCharBufferusing aCharsetDecoder. Likewise, it writes the data back out using aCharsetEncoder.

We're going to assume that our characters are stored on disk in the ISO-8859-1 (Latin1) character set -- the standard extension of ASCII. Even though we must be prepared for Unicode, we also must realize that different files are stored in different formats, and ASCII is of course a very common one. In fact, every Java implementation is required to come complete with support for the following character encodings:

US-ASCII

ISO-8859-1

UTF-8

UTF-16BE

UTF-16LE

UTF-16

The sample program

After opening the appropriate files reading the input data into aByteBuffercalledinputData, our program must create an instance of an ISO-8859-1 (Latin1) character set:

Charset latin1 = Charset.forName( "ISO-8859-1" );

Then, we create a decoder (for reading) and encoder (for writing):

CharsetDecoder decoder = latin1.newDecoder();

CharsetEncoder encoder = latin1.newEncoder();

To decode our byte data into a set of characters, we pass ourByteBufferto theCharsetDecoder, resulting in aCharBuffer:

CharBuffer cb = decoder.decode( inputData );

If we wanted to process our characters, we could do it at this point in the program. But we only want to write it back out unchanged, so there's nothing to do.

To write the data back out, we must convert it back to bytes, using theCharsetEncoder:

ByteBuffer outputData = encoder.encode( cb );

After the conversion is complete we can write the data out to a file.

Summary

Summary

As you've seen, there are a lot of features in the NIO library. While some of the new features -- file locking and character sets, for example -- provide new capabilities, many of the features excel in the area of optimization.

At a fundamental level, there's nothing that channels and buffers can do that we couldn't do using the old stream-oriented classes. But channels and buffers allow for the possibility of doing the same old operationsmuch faster-- approaching the maximum allowed by the system, in fact.

But one of the greatest strengths of NIO is that it provides a new -- and much needed -- structuring metaphor for doing input/output in the Java language. Along with such new conceptual (and realizable) entities as buffers, channels, and asynchronous I/O comes the opportunity to rethink I/O procedures in your Java programs. In this way, NIO breathes new life into even the most familiar procedures of I/O and gives us the opportunity to do them differently, and better, than we have before.