Yup, I’ve moved again. Old posts and comments should be imported pretty soon.
I’ve been spending really a lot of time over the last 18 months or so trying to come up with some kind of sane, interoperable, and performant way to provide a uniform WildFly security layer that allows the many features that have been requested of us (remote authentication, authorization propagation, etc.). Over this slow road to madness, I’ve come to realize that we’re at the mercy of a truly awful piece of technology: JAAS. And there’s no escape from it, for you or me.
In a somewhat recent discussion on the jigsaw-dev mailing list, the issue of circular dependencies during compile time versus run time came up again (previously discussed way back in March 2012) without apparent resolution. At run time, systems like JBoss Modules can take advantage of the JDK’s lazy class loading behavior in order to lazily link modules, even circularly, as required to link module classes. Introducing support for circular runtime dependencies was relatively simple using a multi-phase resolution strategy. This is (uncoincidentally) very similar, in fact, to the mechanism (and reasoning behind the mechanism) of JDK class loaders themselves. However, things become uglier at compile time. The compiler must have a complete picture: any symbol referenced from any source file must be defined in either a class file on the class path or another source file, otherwise the compiler can’t tell if your Foo.Bar is a field on class Foo or a nested class called Bar inside of Foo or a top-level class Bar in a package named Foo. But what do you do when you have two modules that have requirements on one another? The current javax.tools API only has a notion of a single source and target, hence the March 2012 discussion. If you want to compile two interdependent modules today, you can either build them as one (which, if you’re using Maven like 90% of the world, results in one JAR unless you do some trickery) or you hack javax.tools with some scheme to map the class files to their proper output directories - in both cases building the two modules at the same time.
It sounds like a simple problem: if I want JBoss Modules to choose a directory to read a Java native (JNI) library from, such that each platform (which is comprised of OS, CPU architecture, and ABI variations) gets a separate directory, I should just be able to probe a system property and thus know right where to go, right? Wrong. Until Java 8, the JDK only defines “os.arch” and “os.name” to go on; and these names are far from uniform or standardized. CPU names for equivalent architectures can vary (e.g. “x86_64” versus”AMD64”); CPUs may evolve linearly (e.g. “i586” becomes “i686”) or non-linearly (e.g. ARM instruction set offshoots and extended instruction sets like Wireless MMX). In the end you might be able to load, say, a linux-i386 native library on linux-i686, or linux-armv6 on linux-armv7a. And detecting ABI variation is (at least until Java 8) a black art. In the end I’ve wound up with this monstrosity. Without a doubt, I will have to fiddle with it as more weird variations are reported. (By the way, if you discover more weird variations, please reportthem!)
Yes, I have once again found a new home for this tripe. Enjoy!
A simple pattern to downgrade a write lock to a read lock safely:
A new release of XNIOis available which fixes several bugs. For more information about XNIO 2.0, see this previous post. Users of XNIO 2.0.0 and 2.0.1 are encouraged to update to 2.0.2 to take advantage of these bug fixes:
Some of you may have noticed that JBoss Marshalling 1.2.0.GA has been present in the maven repositoryfor some time. Now that the websiteupgrade is complete, I’m happy to announce that it is now available from the downloads pageas well.
Why is it so difficult to write a modularity specification for Java? The requirements appear to be relatively simple:
I’ve uploaded a minor bugfix release for XNIO 2.0.0. It fixes a selector wakeup bug, as well as a minor API bug and a minor optimization for blocking I/O with the NIO-based provider.
XNIO is a simplified low-level I/O layer which can be used anywhere you are using NIO today. It frees you from the hassle of dealing with Selectors and NIO’s poor support for SSL, multicast sockets and non-socket I/O, while still maintaining all the capabilities present in NIO. The XNIO 2.0.0 release includes all of the features of the 1.x series, including:
XNIO 2.0 is getting very close to completion. The latest CR, 2.0.0.CR4, is now available from the downloads page.
Since my last post several months ago, I’ve been quite busy with a number of project releases.
Today I ran across a problem that I’ve hit frequently in the past, this time while designing an extension to JBoss Marshallingwhich allows users to annotate classes which are to be specially externalized. The problem is that one wants to “remember” what
Externalizer instances go with what classes on a semi-permanent basis; however, one does not wish to leak class references (which can lead to the permanent generation filling up, and your application crashing with an
OutOfMemoryError, especially if you’re frequently loading and unloading classes).
It’s official… I’ve checked in the first cut of the Java Serialization from-scratch implementation.
It's not complete yet \- it doesn't even pass all the tests (Update: everything works now - check it out from SVN and try it!) - but I expect that it will be fully functional by the 1.1.0 release (and, coincidentally, by the time I will be giving my JBoss Marshalling talk at TSSJS). Also in progress is Ron Sigal’s JBoss Serialization implementation, which should be in 1.2.0 at the latest.
The final GA release of XNIO 1.2.0 is available for download. This release represents a “settling in” to a 16-week release cycle that should continue from now on. Here’s a complete changelog since 1.1.0.GA:
The XNIO 1.2.0.CR2 release is up on the project page. This release features cleaned up Javadoc, several bug fixes, and a few handy new features (in particular, static methods to perform blocking reads/writes for most common channel types).
This is the last Beta before the CR (candidate for release) series. As always, the download is available at the Remoting download page. This release differs from Beta1 in several ways, not the least of which is the more complete distribution, which includes source JARs and API Javadoc as well as the Remoting JARs themselves. View the release notesfor the full list of differences.
XNIO 1.2.0.CR1 is available for download at the XNIO project download page, sporting lots of cool new features, like:
The download is available at the Remoting download page. The structure is similar to the milestone releases; though it may be changing slightly for the next Beta. I expect that there will be at most one more Beta, followed by a CR series.
JBoss Marshalling is an alternative serialization API that fixes many of the problems found in the JDK serialization API while remaining fully compatible with
java.io.Serializable and its relatives, and adds several new tunable parameters and additional features, all of which are pluggable via factory configuration (externalizers, class/instance lookup tables, class resolution, and object replacement, to name a few).
The release available on the downloads page. This is a bugfix release; I encourage all users of 1.1.0.GA to upgrade.
A couple random ideas about how to bind privileged (<1024) port numbers using XNIO. It might be possible to utilize the UNIX domain socket API (slated for 1.2.0) and use the file descriptor-passing mechanism. A root-privileged server (written in C) could be started that binds to a UNIX domain socket on the filesystem, whose access permissions are limited. The XNIO-based application can use this domain socket to request bound ports and get the file descriptor back, which it would use to construct a server. Another idea would be to turn it around, and have the XNIO process run a setuid root program (via ProcessBuilder) which binds the port and hands it back through a socket created by the Java program.
I’m pleased to announce that the second beta of the new marshalling framework has arrived. The API is pretty much final at this point; now it’s just a bug quest before the first CR is released in a couple weeks. For more information about the features of the framework, please see the original Beta1 announcement.
For XNIO 1.2.0, I’m doing away with the separated API versus standalone JAR. My original notion was that the standalone JAR would contain the whole API, plus the
Xnio bootstrapping class, plus at least one implementation (NIO right now, as that is currently the only complete implementation). But I think it will be simpler going forward to just add
Xnio into the API itself, and ship an API JAR plus one JAR per implementation. So from now on (including any future snapshots), the JARs will follow this new structure instead.
XNIO 1.1.0.GA is available on the XNIO downloads page. Not much change since 1.1.0.CR1. Just a Javadoc update, and one interface rename:
TcpClient has been renamed to
TcpChannelSource (see XNIO-55for more information). Also see the online Javadocsfor usage information.
I’ve gotten some feedback about my closure idea. Some of it bad, some of it… also bad. I thought I’d reply to all the comments in a new posting here rather than try to squeeze it all in to the crappy comment thing.
Truly pluggable locking strategies:
As a part of the JBoss Remoting project (and hopefully several other projects as well), I’ve developed a separated marshalling (a.k.a. serialization) framework. This framework was inspired by the need for certain features unavailable with the standard
As the title states - 1.1.0.CR1 is available at the XNIO project download page. The main list of changes:
Everyone wants tail recursion. I’ve heard that some JVMs automatically detect tail recursion even. How about a simple syntax change to do it at a language level?
Or: How to add closures to Java without mutilating the language
One of the most common problems that I see users having on the ##java IRC channel is improper resource management. By “resource management”, I mean “things that need cleaning up after”, such as streams, files, sockets, JDBC statements, etc. The tendency is not so much to ignore the problem, but to use an improper, unsafe, or just unnecessarily complex or ugly approach to solving it.
The first GA release of XNIO is now available. There is only one substantial fix in this release. The standalone API did not present a way to close ChannelSource and Connector instances that were created via an Xnio instance. This has been fixed, and I think the wait has been long enough.
Well, I lied. I’ve found a couple more minor API issues which I’ve fixed for CR3 - a misspelling in a method name (“interruptible”, not “interruptable”), a typo which prevented junit from being downloaded, one interface rename (“Client” becomes “ChannelSource”), and a few other minor fixes are included in this release. I think this one will become GA unless I find a glaring bug in the next week or so.
In a previous post, I talked about the impracticality of asynchronous reads in a scalable server situation. The problem I cited was that if there are large amounts of pending reads, each with its own preallocated buffer, a large quantity of resources can possibly be consumed, thus impeding scalability.
This new version fixes several problems in CR1 and is closer to a final product now. I do not anticipate any further API changes, so this version is safe to develop against in anticipation for the final GA release (which I have marked for July 28 - not too far in the future, but hopefully far enough that any remaining bugs will be found and flushed out).
I’ve released XNIO 1.0.0.CR1. XNIO is a replacement for NIO that vastly simplifies the handling of non-blocking I/O. XNIO gives you the full power of NIO channels while eliminating the headache of thread and Selector management. Also, it is dramatically simpler to implement an XNIO provider than it is to implement an NIO provider - opening the door to APR or other native implementations as well as non-network transport types such as serial ports, and platform-specific types like UNIX domain sockets.
If you haven’t heard of JSR-308, take a few minutes to read the specificationand familiarize yourself with it. This JSR is designed “permit annotations to appear in more places than Java 6 permits”.
For those who aren’t aware, the AIO read idiom works something like this pseudocode:
For those of you who aren’t aware, JSR-310is devoted to developing a (much-needed) new and improved replacement date and time API for Java. Since the spec lead, Stephen Colebourne (who is the author of the excellent JODA date and time library), has made the wise decision to make the process public for the JSR, you can read (and join) the discussions taking place around this effort.
Remoting 3.0.0-M2 is released. I’ve got the jars uploaded in a ZIP file, but you’ll want to check out the project and look in the “samples/” for samples of using the standalone API. The Microcontainer integration is scheduled for M3, so standalone is pretty much all that you can really play with at the moment. Report bugs in JIRA as “found in: 3.0.0-M2” and “fix for: 3.0.0-M3” to make sure I see them.