vote up 1 vote down
star
1

The Blue Obelisk page about "Open Standards" says that MDL and SMILES are "proprietary format[s]" and "not Open as there is no community process for its development; it can, in principle be modified by the owner at any stage."

I have strong feelings about this topic. Why is SMILES (and I mean here the representation and not the canonicalization) a proprietary format? It was published in a journal paper, included in a book chapter, and the details are available online for many to read. There are many different implementations of the language, and some groups have added their own extensions to SMILES. Daylight and the authors of SMILES have no trademark to the term SMILES and have no patent claim. Even if Daylight were to make a new version which was not compatible to existing SMILES, almost no one would follow suit unless it proved useful.

By any definition I can come up with, SMILES is an open standard. If you count OpenSMILES as a relevant community (which is hard to justify, since nothing from OpenSMILES has made its way even into the existing open source projects) then even that objection is gone.

By comparison, several participants of Blue Obelisk say that CML is an Open Standard. (Since I can't find the license for it, I'm not sure what that means, but I've asked for that clarification as another topic.) CML is controlled by one or two people, and it seems that while the CML specification can be distributed to others, it cannot be changed without the permission of those authors.

Is that really a community process, and for that matter, what does a community process mean?

Yet another example is the MDL connection table formats. That is listed as a proprietary format, but it's also a format which is well documented by the primary maintainer, described in published papers and books, implemented in a number of code bases, including open source ones. What makes it not open, under the Blue Obelisk definition? By the way, the vendor supplied format definition should be an example of what a good specification looks like. CML's format-through-schema-dump is not.

One possibility is that the MDL specification cannot be redistributed, legally. I have a copy of it but the copyright statement prohibits sharing that spec. Is that what makes it not Open? But in that case, the ISO 8601 date format is also not an open specification, because it costs about 85 Euros to purchase and cannot be copied without permission.

If this is a problem then two ways to resolve it are 1) ask for redistribution rights, so long as there are no changes (putting it on par with CML) or 2) develop an equivalent but open documentation.

Another possibility for not calling the MDL format "open" is that they can change the format without consulting "the community." They have changed the format once, although I suspect that that involved feedback from their user community. They introduced the V2000 format, and carefully made it backwards compatible with the old file format. If they had not, others would not have followed their lead. Indeed, I haven't seen that many V2000 files in the wild. It doesn't seem like they have strong control over the format.

On the other hand, I see that the Blue Obelisk page[1] lists InChI as an open format, base solely I assume on the source code being accessible since it is a somewhat closed development process which has changed the output several times. I have looked at the source code and can say that it is not a simple specification. I am unaware of any other implementation which can parse InChI strings with near 100% success rates.

Because of the lack of a good stand-alone specification, any developer who reads the LGPL-based source code in order to reimplement it might legally be considered "dirty" in terms of clean-room coding practices, meaning that the new code will also have to be LGPL. For reference, I prefer BSD. I would have thought that a good Open Standard also means the freedom to pick a different license, including free ones which are not compatible with the GPL family of licenses.

What then are the exemplar "Open Standard" formats in this field, why are they "Open", and why are they more Open than both the MDL and SMILES formats?

ADDENDUM

I was hoping specially for feedback on which formats are open and which aren't, and why. Is SMILES open or not? Why does OpenSMILES make a difference? Are the MDL formats open? What would be needed for an OpenMDL? Is the PDB format open?

If the format is published in a journal which does not allow redistribution without paying a standard copying fee, is the format itself open or closed? For example, the SMD format published in JCICS the early 1990s was meant to be a community-driven format definition, using a context-free grammar. Is this format open or not?

flag

2 Answers

vote up 0 vote down

OpenSMILES clearly changed this game. But I guess OpenSMILES is also the argument showing that SMILES itself was a 'proprietary' format: important aspects were not properly documented, leading to competing SMILES implementations. I think it is comparable with the MS original document format: it was describes, but badly and has so many undocumented cases, that it was about impossible to write a proper implementation. And this is besides the unique SMILES issue.

The specifications to be hidden is certainly not helping making something an Open standard. I have a copy of the ctfile.pdf at home too. It has actually been unavailable from the web for a while.

The CML project comes pretty close to be a proper Open Standard and Open Project, and indepdent groups have worked with Peter on extension of CML, like CMLReact (Prof. Thornton's group), CMLSpect (Dr. Steinbeck's group), CMLRSS (me), and there is development undergoing of using CML in metabolomics (groups in Leiden and Halle), and a bit more.

The InChI project is another thing... there is now an advisory board, but the InChI lacks indeed a source code independent specification. I have many people comment on this, who'd liked to implement the functionality in other languages and implement parts of it. Personally, I'd love to have just the atom numbering part separate.

Perhaps the Blue Obelisk community should indeed draft a chapter on the minimum requirements for something to be Open...

link|flag
I don't see how OpenSMILES changed the game. There is a spec, yes, but it's incomplete and I don't see that anyone has really been affected by its presence. Not one toolkit has made a change because of it. Besides, if that's all that's needed to make the format be non-proprietary, then would an "OpenCT" group and document be sufficient to make the MDL formats be considered non-propritary? And if so, how detailed would that spec need to be? "Open Standards" is one of the three legs for Blue Obelisk, but so far the primary distinction seems to be "was not developed by a company." – Andrew Dalke Dec 5 at 13:47
I think this is partly because this specification show such a high overlap with the original SMILES specification, that no one cared to much to update their parsers for the differences... or? – Egon Willighagen Feb 15 at 22:52
Then it didn't "clearly change the game", yes? – Andrew Dalke Feb 27 at 19:19
No, I think it did. Things go slowly, but I think it nicely shows that closed access journals is not the only way to distribute specifications, and that modifying them to clarify things should be part of the process. – Egon Willighagen Feb 27 at 20:19
I think OpenSMILES is a weak example. Neither Craig nor I are research scientists, and our careers are not really affected by not publishing in peer-reviewed literature. Indeed, I don't see the cost as justifiable for my consulting business. If the goal is only to show that specifications can be distributed via some other mechanism, then the IETF RFCs made that clear decades ago. Plus, OpenSMILES as well as the number of SMILES variants shows that a copyright restriction on the original spec documents didn't affect things, and that modification was already part of the process. – Andrew Dalke Mar 2 at 16:08
show 4 more comments
vote up 0 vote down

Open Data

Data that has a clear statement of copyright and a permission of the copyright owners that allow redistribution and modification.

This might be a clear public domain like statement by the copyright owners, or a more formal CC0 license. Some earlier data projects use a GNU FDL or Open Source license, which is OK too.

Open Source

This one is easy. The use of licenses is well-established here, and written up as list of OSI-approved licenses.

Open Specifications

Rather than Open Standards, a specification that has a clear statement of copyright and a permission of the copyright owners that allow independent and free implementation of that specification as well as provide a free and complete documentation of the specification, where the freedoms must include redistribution.

The requirements of modification is not agreed upon. Some BO members indicate that this is not a requirement, and that the modification conflicts with the idea of a standard, which is why I personally now prefer to use the term Open Specification, to explicitly allow modification. Following this line of thinking, an Open Standards would be a specific version of a particular Open Specification, so that implementers can standardize on that particular version of the specification. I think this is very much like the way the W3C standards work too.

link|flag
SMILES was first published in the 1980s in JCICS, which isn't freely redistributable. That's what makes it non-free? SMD (Standard Molecular Data) was a community-based format published in JCICS in the early 1990s. Is it also not free? Can any format described in a non-open-access journal ever count as an open specification? Is the PDB specification open? I can't find a distribution policy or even copyright statement. Is it even legitimate to say that InChI is an open specification, given that it doesn't have a published spec other than the code? – Andrew Dalke Feb 13 at 3:57
I had not thought about the non-OA nature of the (original) SMILES specification. That would certainly violate the Open Specification definition. Good. Case closed on that one. Also good that there now is OpenSMILES, which does not suffer from that issue. – Egon Willighagen Feb 15 at 7:57
I am not familiar with SMD. I also like to note that the problem of being published in a non-OA is not a problem in itself, but only of the specification is not available otherwise. – Egon Willighagen Feb 15 at 7:58
PDB? Does that even have a specification? I only know there are so many special cases, that is it very hard to parse PDB files properly... InChI? Well, InChI is Open Source... let's just settle on agreeing with that... – Egon Willighagen Feb 15 at 8:00
Summarizing: if you can point me to a downloadable document which I can modify and redistribute, then ask me again if it is an Open Specification... if there is no such download, and very likely it is not. – Egon Willighagen Feb 15 at 8:00
show 9 more comments

Your Answer

Get an OpenID
or

Not the answer you're looking for? Browse other questions tagged or ask your own question.