Posts: 135 / Nickname: fsommers / Registered: January 19, 2002 7:24 AM
From JavaOne 2009: On Trust and Types
June 11, 2009 8:00 AM
|
In this interview with Artima, Gwyn Fisher, CEO of Klocwork, describes cases where well-known Java APIs work differently based on deployment platform, and how well-defined type systems can make resource management more reliable:
http://www.artima.com/lejava/articles/javaone2009_gwyn_fisher.html What do you think of the notion that types can help you improve trust in APIs? |
Posts: 128 / Nickname: watson / Registered: September 7, 2005 3:37 AM
Re: From JavaOne 2009: On Trust and Types
June 12, 2009 7:47 AM
|
> What do you think of the notion that types can help you
> improve trust in APIs? OK, I'll just go ahead and confess that I don't know how types would do that. It's actually a little confusing to me because Java has a notion of 'type' that I don't think is the same as what is being discussed here. P.S. I hate to do the grammar police thing but I see this error so often and I find it to be really annoying. I normally don't say anything but this is ostensibly an edited article. The phrase "you loose track", the word you want here is 'lose' not 'loose'. I know it's confusing because 'lose' rhymes with 'choose', but there's a big difference in meaning between the words. |
Posts: 135 / Nickname: fsommers / Registered: January 19, 2002 7:24 AM
Re: From JavaOne 2009: On Trust and Types
June 12, 2009 9:04 AM
|
> P.S. I hate to do the grammar police thing but I see this
> error so often and I find it to be really annoying. I > normally don't say anything but this is ostensibly an > edited article. The phrase "you loose track", the word > you want here is 'lose' not 'loose'. I know it's > confusing because 'lose' rhymes with 'choose', but there's > a big difference in meaning between the words. Thanks! That was a typo. We just fixed it. |
Posts: 12 / Nickname: thomasm / Registered: January 27, 2009 1:41 AM
Re: From JavaOne 2009: On Trust and Types
June 12, 2009 11:59 AM
|
Hi James,
Your solution is a 'closure'. Too bad this pattern is hard to use in Java: 'final' everywhere, and calling the method is complicated. 'My' solution is simpler to use, unfortunately it closes the stream a little bit too late (the code I posted up to 1 second, but using some tricks it could be reduced to a few milliseconds on average). My solution can still be improved. > What's happening in Java 7 to change this? See Project Coin - http://openjdk.java.net/projects/coin/ - specially Automatic Resource Management, V.2: http://mail.openjdk.java.net/pipermail/coin-dev/2009-April/001481.html - it's not accepted yet, but I hope it will. |
Posts: 128 / Nickname: watson / Registered: September 7, 2005 3:37 AM
Re: From JavaOne 2009: On Trust and Types
June 12, 2009 0:25 PM
|
> 'My' solution is simpler to use
I guess I'm not really seeing that. I don't see why it's any easier to put synchronized block around your read loop than to put a try catch block around your read loop. I use the technique I posted above for database access and it can be akward and verbose at times but it's definitely worth the payoff. |
Posts: 12 / Nickname: thomasm / Registered: January 27, 2009 1:41 AM
Re: From JavaOne 2009: On Trust and Types
June 12, 2009 9:41 PM
|
> 'My' solution is simpler to use:
than try...catch:
The problem if your solution is that you can only access variables outside the block if they are declared final. |
Posts: 128 / Nickname: watson / Registered: September 7, 2005 3:37 AM
Re: From JavaOne 2009: On Trust and Types
June 13, 2009 4:43 PM
|
> The problem if your solution is that you can only access
> variables outside the block if they are declared final. That's true if you use an anonymous inner class and the variables are local. In that case, it generally isn't a big deal. I usually end adding items to a list or calling some other method on my enclosing class so it doesn't matter. In the cases where I need to write once to a local, I just declare it like so: int[] out = {0}; Pretty, no, but it works just fine. The other thing about using the closure-style or pseudo-functional style is that you it's easy to write reusable classes to pass into these functions or write inner classes or nested classes. I readily admit that this is verbose but it works really well in practice. The problem I see with using threads and synchronized blocks is that you could end up creating lots of little threads (and negatively impacting performance.) One thing you might consider for this is using a single-threaded Executor instead of spawning a thread for each call. |
Posts: 12 / Nickname: thomasm / Registered: January 27, 2009 1:41 AM
Re: From JavaOne 2009: On Trust and Types
June 11, 2009 10:37 PM
|
There is a workaround to auto-close resources:
InputStream in = ... synchronized (closeLater(in)) { // reading } This is guaranteed to close the input stream 1 second after it is no longer used (synchronized). The closeLater method looks like this: static InputStream closeLater(final InputStream in) { new Thread(new Runnable() { public void run() { try { synchronized (in) { in.wait(1000); in.close(); } } catch (Exception e) { } } }).start(); return in; } I know it's not perfect, but it might be good enough until Java 7. |
Posts: 128 / Nickname: watson / Registered: September 7, 2005 3:37 AM
Re: From JavaOne 2009: On Trust and Types
June 12, 2009 7:26 AM
|
> There is a workaround to auto-close resources:
I prefer this approach: class InputStreamProcessor { private final InputStream stream; public InputStreamProcessor(InputStream stream) { this.stream = stream; } void process(StreamReceiver receiver) throws IOException { byte[] buffer = new byte[1024]; try { do { int read = stream.read(buffer); if (!receiver(buffer, read) return; } while (read >= 0); } finally { stream.close(); } } public interface Receiver { boolean process(byte[], int len); } } This is a simplified example but it should give the basic idea. If more APIs were developed in this style, I think this wouldn't be such a big issue. > but it might be good enough until Java 7. What's happening in Java 7 to change this? Sorry, I haven't been following this for a while. Closures would help with the above but is there something else you are talking about here? |
Posts: 48 / Nickname: mthornton / Registered: October 16, 2005 11:22 PM
Re: From JavaOne 2009: On Trust and Types
June 14, 2009 3:47 AM
|
> There is a workaround to auto-close resources:
> InputStream in = ... > synchronized (closeLater(in)) { > // reading > } It is at least theoretically possible that your main thread won't execute the synchronized statement until after the thread started by closeLater has completed. With a 1000ms wait this is unlikely, but not quite impossible. |
Posts: 1 / Nickname: michid / Registered: March 9, 2008 3:25 AM
Re: From JavaOne 2009: On Trust and Types
June 15, 2009 1:49 AM
|
Another approach
interface Block { void execute(); } interface Resource { void close(); } public static <R extends Resource> void with(R resource, Block block) { try { block.execute(); } finally { resource.close(); } } Assuming we have a Resource r which can doSomething: with(r, new Block() { public void execute() { r.doSomething(); }}); |
Posts: 98 / Nickname: achilleas / Registered: February 3, 2005 2:57 AM
Re: From JavaOne 2009: On Trust and Types
June 15, 2009 3:54 AM
|
It seems Java needs C++'s RAII...
|
Posts: 18 / Nickname: cgross / Registered: October 16, 2006 3:21 AM
Re: From JavaOne 2009: On Trust and Types
June 15, 2009 3:38 PM
|
> It seems Java needs C++'s RAII...
++ In GScript:
The 'using' statement guarantees that resources are closed appropriately. GC'd languages that don't support stack-derived life spans for objects absolutely need this feature. It makes resource-based API's much more brain dead. Cheers, Carson |
Posts: 12 / Nickname: thomasm / Registered: January 27, 2009 1:41 AM
Re: From JavaOne 2009: On Trust and Types
June 14, 2009 6:34 AM
|
> possible that your main thread won't execute ... until after the thread started by closeLater has completed.
True. A better workaround is: static void closeLater(final InputStream in, final Object sync) { if (!Thread.holdsLock(sync)) { throw new IllegalStateException("must first synchronize on " + sync); } Thread t = new Thread(new Runnable() { public void run() { try { synchronized (sync) { in.close(); } } catch (Exception e) { } } }); t.start(); while (t.getState() != State.BLOCKED) Thread.yield(); } Usage is a bit different, one example is: synchronized void test() throws Exception { InputStream in = new ByteArrayInputStream(new byte[0]); closeLater(in, this); // read... } It is a bit more complicated to understand. |
Posts: 48 / Nickname: mthornton / Registered: October 16, 2005 11:22 PM
Re: From JavaOne 2009: On Trust and Types
June 14, 2009 6:57 AM
|
I'd prefer a solution like this
CloseableStack.Mark mark = CloseableStack.mark() try { InputStream in = CloseableStack.open(new FileInputStream(...)); // ... read, possibly open more streams } finally { mark.close(); } CloseableStack would manage thread local stacks. CloseableStack.Mark.close() would close all objects opened since its creation (and not throw any checked exceptions). While this still requires one finally clause it isn't as complex as that usually required. |
Posts: 37 / Nickname: miata71 / Registered: March 29, 2006 6:09 AM
Re: From JavaOne 2009: On Trust and Types
June 14, 2009 0:43 PM
|
I like Mark's idea, cause it nicely handles where you have multiple streams to open/close. Reminds me of an old Amiga option to remember all the mallocs you were making, then you could close them all i one call.
Does CloseableStack code exist, or it is a dream? I tried (briefly) writing code to do similar but got stuck. |
Posts: 48 / Nickname: mthornton / Registered: October 16, 2005 11:22 PM
Re: From JavaOne 2009: On Trust and Types
June 14, 2009 0:50 PM
|
No CloseableStack is just an idea, but I can't see any difficulty in implementing it.
|
Posts: 12 / Nickname: thomasm / Registered: January 27, 2009 1:41 AM
Re: From JavaOne 2009: On Trust and Types
June 14, 2009 7:34 AM
|
Another option is to use closeQuietly as in:
http://commons.apache.org/dbutils/apidocs/org/apache/commons/dbutils/DbUtils.html If you implement those methods yourself, you might want to log errors when closing. |
Posts: 12 / Nickname: thomasm / Registered: January 27, 2009 1:41 AM
Re: From JavaOne 2009: On Trust and Types
June 14, 2009 8:13 AM
|
The advantage of the 'synchronized solution' is that you can actually enforce the 'correct' usage. Let's say you wrote a library with a method "public InputStream getStream()". There is no way to enforce that the caller of the method calls in.close(). If you change the API to "public InputStream getStream(Object sync)" and use my solution, then the caller doesn't have to close the stream (he still can), but he has to use a synchronized block - your library can enforce this using "if (!Thread.holdsLock(in)) throw ...". Your library will close the stream if the caller forgot (shortly after the synchronized block ends).
|
Posts: 128 / Nickname: watson / Registered: September 7, 2005 3:37 AM
Re: From JavaOne 2009: On Trust and Types
June 14, 2009 5:05 PM
|
> The advantage of the 'synchronized solution' is that you
> can actually enforce the 'correct' usage. I have to disagree. There's nothing forcing me to use close-later. The solution I posted doesn't give the client code the option to not close properly. That's enforced by the solution. Of course, as I wrote it, no one is forced to wrap their stream with this object. The idea, though, is that if you started with this kind of approach instead of the more imperative approach you eliminate the possibility of not releasing resources properly. I don't see any corresponding possibilities with the synchronized approach or with Mark's approach. Those two solutions require that the client explicitly invoke them. |
Posts: 12 / Nickname: thomasm / Registered: January 27, 2009 1:41 AM
Re: From JavaOne 2009: On Trust and Types
June 14, 2009 9:16 PM
|
> There's nothing forcing me to use close-later
The library calls it (I thought that's clear). > The solution I posted doesn't give the > client code the option to not close properly. I know. The only problem is that it's hard to use. > I don't see any corresponding possibilities > with the synchronized approach Well, I do: if the library calls closeLater. > Those two solutions require that the client > explicitly invoke them. Not if the library calls closeLater and only exposes getStream(Object sync). |
Posts: 128 / Nickname: watson / Registered: September 7, 2005 3:37 AM
Re: From JavaOne 2009: On Trust and Types
June 15, 2009 5:16 AM
|
> > There's nothing forcing me to use close-later
> > The library calls it (I thought that's clear). I think you mean each and every library. > > The solution I posted doesn't give the > > client code the option to not close properly. > > I know. The only problem is that it's hard to use. When you wrote "The advantage of the 'synchronized solution' is that you can actually enforce the 'correct' usage" I thought you were implying that the closure solution couldn't enforce correct usage. Otherwise, how would it be an advantage? It really isn't hard to use. It's probably just different from what you are used to. > > I don't see any corresponding possibilities > > with the synchronized approach > > Well, I do: if the library calls closeLater. > > > Those two solutions require that the client > > explicitly invoke them. > > Not if the library calls closeLater and only exposes > getStream(Object sync). If I call getStream(Object sync), I need to synchronize on sync within a second, right? And I need to maintain that synchronization until I am done with the stream. Also, each library needs to do this. So the synchronized solution doesn't enforce anything. Each library that uses it is required to enforce this. |
Posts: 12 / Nickname: thomasm / Registered: January 27, 2009 1:41 AM
Re: From JavaOne 2009: On Trust and Types
June 15, 2009 7:03 AM
|
> the closure solution couldn't enforce correct usage
It does. You can enforce using the closure in a library, but it would be quite hard to use. > It really isn't hard to use. > It's probably just different from what you are used to. I'm used to. Runnable, Arrays.sort(.., Comparator) and so on. It is hard to use: How do you change a variable of the outer block in the inner block? It's possible (for example using one-element arrays) but it's ugly. Even accessing a variable of the main block is complicated (you need to make it final). The closure (Block.execute in the example above) needs to declare exception(s) the block can throw, and the caller has to catch that (even if he doesn't throw any). Or you don't declare, but then the caller can't use checked exceptions. Then the size of the code is bigger. > If I call getStream(Object sync), > I need to synchronize on sync within a second, right? No, you need to sync on the object _before_ calling getStream. Example: synchronized void exampleClient() throws Exception { // anything InputStream in = myApi.getStream(this); // read... // no need to call in.close() (but you can) // the stream is closed even if you throw exceptions } In this case 'this' is the sync object. Another example: Binary b = node.getProperty("jcr:content").getBinary(); synchronized(b) { InputStream in = b.getStream(); // read... same as above } In this case the sync object is not even passed. Instead, the sync object is 'b' (the object where you call getStream()). > I think you mean each and every library. > Also, each library needs to do this. Well, obviously. I'm quite sure you didn't understand my solution, otherwise you wouldn't ask. The 'synchronized' solution enforces using synchronized in the client code, if the 'synchronized' solution is used in the library. |
Posts: 128 / Nickname: watson / Registered: September 7, 2005 3:37 AM
Re: From JavaOne 2009: On Trust and Types
June 15, 2009 8:13 AM
|
> > the closure solution couldn't enforce correct usage
> It does. You can enforce using the closure in a library, but it would be quite hard to use. I know it does. I was pointing out that you implied it didn't. What is so difficult about it? > I'm used to. Runnable, Arrays.sort(.., Comparator) and so > on. It is hard to use: How do you change a variable of the > outer block in the inner block? It's possible (for example > using one-element arrays) but it's ugly. Even accessing a > variable of the main block is complicated (you need to > make it final). The closure (Block.execute in the example > above) needs to declare exception(s) the block can throw, > and the caller has to catch that (even if he doesn't throw > any). Or you don't declare, but then the caller can't use > checked exceptions. Then the size of the code is bigger. I don't see how declaring a variable final is complicated. It's just an extra keyword. In fact, I think Java would be a much better language if variables defaulted to final and had to be explicitly declared to be writable. I use this approach all the time. The need to modify a local variable from the rarely comes up and you can always use a named local class, inner class, or nested class to avoid the issue entirely. Exception handling is a downside but again, in the infrequent cases that I need to, it's worth wrapping and unwrapping the runtime errors. The size of the code is not a real issue, in my opinion. > > I think you mean each and every library. > > Also, each library needs to do this. > > Well, obviously. I'm quite sure you didn't understand my > solution, otherwise you wouldn't ask. The 'synchronized' > solution enforces using synchronized in the client code, > if the 'synchronized' solution is used in the library. I understand the approach perfectly and I'm not asking. I'm clarifying. As stated you made it sound as if the solution enforced correct usage in itself which you are now readily admitting that it does not. The closure solution does enforce correct usage in itself. Basically, the solution you are proposing requires that all work with the resource object be executed within the same stack as the request to get the resource object. You can't easily pass this resource to another class for it to use. This limits the kinds of designs you can use with this approach. You can't easily reuse this resource object or share it with other parts of the code. The real problem I see with the synchronization solution, other than being inelegant is that synchronization is associated with many types of usage errors in itself. Many of these usage errors are extremely difficult to find in code and can depend on the JVM, platform, or hardware. |
Posts: 12 / Nickname: thomasm / Registered: January 27, 2009 1:41 AM
Re: From JavaOne 2009: On Trust and Types
June 15, 2009 9:30 AM
|
> I was pointing out that you implied it didn't
I'm sorry if you think I implied that. > The size of the code is not a real issue, in my opinion Let's say we have different opinions. > you made it sound as if the solution enforced > correct usage in itself I'm sorry if you think it does. > The closure solution does enforce correct usage in itself. You seem to use a different meaning for "in itself". Closures only solve the problem if they are used. I avoid closures in Java because they are cumbersome to use. > within the same stack as the request to get > the resource object Only if the API enforces it. The same with closures, if the API enforces to use closures. > The real problem I see with the synchronization > solution, other than being inelegant How is it inelegant? If a solution is inelegant in Java, then it's the closures solution :-) > many types of usage errors in itself You probably mean Java level deadlocks. Yes, that's a problem. To avoid it, you need to synchronized on a different objects for each resource. > Many of these usage errors are extremely difficult to find Multithreading is complicated, no question. |
Posts: 128 / Nickname: watson / Registered: September 7, 2005 3:37 AM
Re: From JavaOne 2009: On Trust and Types
June 15, 2009 9:49 AM
|
> How is it inelegant? If a solution is inelegant in Java,
> then it's the closures solution :-) I guess if you think elegance is about how your code looks, then can see that. I mean that it's inelegant from a design perspective. It probably won't be a big deal from a usage perspective but one thing I noticed is that the synchronized solution can be subverted even if the library only provides a getResource(lock) method. The simple way is to call System.exit() from within the synchronized block. A more general way follows. Granted, someone would probably have to go out of their way to do this but it's possible that this kind of scenario could arise in multi-threaded code unintentionally: class Blocker<E> { private volatile Exception exception; private volatile E it; public void start(Action<E> action) { new Thread(new BlockingRunnable(action)).start(); } public E get() throws Exception { while (it == null && exception == null) { try { Thread.sleep(100); } catch (InterruptedException e) { /* buzz off */ } } if (exception != null) throw exception; return it; } private class BlockingRunnable implements Runnable { private final Action<E> action; BlockingRunnable(Action<E> action) { this.action = action; } public void run () { synchronized(Blocker.this) { try { it = action.execute(); } catch (Exception e) { exception = e; } try { while(true) { Thread.sleep(3600000); } } catch (InterruptedException e) { // snooze... } } } } public interface Action<E> { E execute() throws Exception; } } An example of how this could be used: public static void main(final String[] args) { try { final Blocker<InputStream> blocker = new Blocker<InputStream>(); blocker.start(new Action<InputStream>() { @Override public InputStream execute() throws FileNotFoundException { return StreamProvider.getStream(blocker, args[0]); } }); System.err.println("Begin access outside of synchronized block"); InputStream stream = blocker.get(); for (int b; (b = stream.read()) > -1;) { System.out.print((char) b); } } catch (Exception e) { e.printStackTrace(); } finally { System.exit(0); } } |
Posts: 12 / Nickname: thomasm / Registered: January 27, 2009 1:41 AM
Re: From JavaOne 2009: On Trust and Types
June 15, 2009 10:33 AM
|
> it's inelegant from a design perspective.
You are completely right. The closure solution would be a lot better technically, too bad that Java doesn't support it well. The 'synchronized' solution is an incomplete workaround, or you can call it a hack. The main problem is probably that you can't guarantee when exactly the resource is closed. The language I like most at the moment is Scala. It nicely supports closures. The only problem in Scala was that it doesn't support 'break' (which I use from time to time). Until I found out you can implement 'break' yourself (by throwing an exception). In any case, 'break' will be supported in the next version. I think that's great. The only remaining problem for Scala is the poor tools support (Eclipse plugin :-) |
Posts: 128 / Nickname: watson / Registered: September 7, 2005 3:37 AM
Re: From JavaOne 2009: On Trust and Types
June 15, 2009 11:00 AM
|
> > it's inelegant from a design perspective.
> > You are completely right. The closure solution would be a > lot better technically, too bad that Java doesn't support > it well. The 'synchronized' solution is an incomplete > workaround, or you can call it a hack. The main problem is > probably that you can't guarantee when exactly the > resource is closed. Do you know what's going on with the closure proposals for Java 7? I was following that for a while but I got tired of all the different proposals coming out of the woodwork. It seems to me that you need a short anonymous inner class syntax. A checked exception pass-through mechanism would be cool and a way to write to local would really top things off. In order to do that, however, it seems that the language needs a way to distinguish between synchronous-one-time closures and everything else. > The language I like most at the moment is Scala. It nicely > supports closures. The only problem in Scala was that it > doesn't support 'break' (which I use from time to time). > Until I found out you can implement 'break' yourself (by > throwing an exception). In any case, 'break' will be > supported in the next version. I think that's great. The > only remaining problem for Scala is the poor tools support > (Eclipse plugin :-) Scala's a cool language but I kind of feel that the complexity level is bit higher than what I would like for team development. |
Posts: 12 / Nickname: thomasm / Registered: January 27, 2009 1:41 AM
Re: From JavaOne 2009: On Trust and Types
June 15, 2009 10:19 PM
|
> closure proposals for Java 7?
As far as I know, it will not be implemented (reasons: lack of time, risk of making stuff too complicated). The only changes that are planned are Project Coin, and I don't think it's decided yet: http://openjdk.java.net/projects/coin - Automatic Resource Allocation is one of the proposed features. > Scala ... complexity level I agree. Some features should be removed, I'm not sure which ones. 'Removed' could also mean 'not allowed to use', basically you could restrict yourself, with tools similar to Checkstyle / FindBugs. |
Posts: 3 / Nickname: i30817 / Registered: June 26, 2007 0:24 PM
Re: From JavaOne 2009: On Trust and Types
June 16, 2009 1:27 PM
|
/** * Close closeables. Use this in a finally clause. */ public static void close(Closeable... closeables) { for (Closeable c : closeables) { if (c != null) { try { c.close(); } catch (Exception ex) { Logger.getLogger(IoUtils.class.getName()).log(Level.WARNING, "Couldnt close Closeable", ex); } } } } /** * Sockets are not closeable... wtf */ public static void close(Socket socket, Closeable... closeables) { for (Closeable c : closeables) { if (c != null) { try { c.close(); } catch (Exception ex) { Logger.getLogger(IoUtils.class.getName()).log(Level.WARNING, "Couldn't close Closeable.", ex); } } } if (socket != null) { try { socket.close(); } catch (Exception ex) { Logger.getLogger(IoUtils.class.getName()).log(Level.WARNING, "Couldn't close Socket.", ex); } } } import static IoUtils.*; void whatever(){ Resource wtfClosureFetishists = null; try{ ... }finally{ close(wtfClosureFetishists); } } |
Posts: 4 / Nickname: articulate / Registered: September 19, 2005 8:54 AM
Re: From JavaOne 2009: On Trust and Types
October 6, 2009 0:43 PM
|
I think the real time community is more sophisticated than enterprise developers in this area. For example pre-emption - if a higher priority request arrives, a lower priority request that holds required resources could be pre-empted, and their resource released for the high priority user. I have not supplied a priority, and a timeout to file open in any environment I have used.
I notice static types adding complexity to solutions where a closure returns values. I don't see types adding anything in this area. I do see features like closures, ThreadLocal, and aspects (AOP) helping centralize resource management. Each use point of API's like java.sql and java.io must manage resources by calling close. Spring lets you use many paradigms for solving these problems, including using closure like objects, annotations on functions, and helper functions for special cases. http://static.springsource.org/spring/docs/3.0.x/spring-framework-reference/html/ch12s02.html#jdbc-JdbcTemplate http://static.springsource.org/spring/docs/3.0.x/spring-framework-reference/html/ch10s05.html#transaction-declarative-annotations Solutions where a separate thread does resource cleanup, like garbage collection, have to be able to stall a thread which makes an allocation request, when resources are exhausted, and do a clean up. It seems like a complicated solution. An explicitly parallel resource server where one server processes close and open might work better. |