Friday, February 24, 2012

RPM + OSGi automation: Step 1 completed

Recent RPM versions include the concept of dependency generators. This opens the door for a number of automations and simplifications for people using RPM as their distribution format. 

In Fedora we stepped in and worked on making Java land fell less of an alien in Linux distributions. 

Not every OSGi feature is considered because some of them don't match with the goals of distributing a single project. Everything is based on the Bundle-SymbolicName and Require-Bundle headers (for now). 

  • Bundle-SymbolicName header is used for generating a virtual provide for the rpm the bundle is part of in the format Provides: osgi(bundle-symbolicName) = bundle-version . This is done for every OSGi bundle in the RPM so the number of these provides will be the same as the number of bundles in it. This provides generation was enabled early in the Fedora 17 development phase and will be available for use in the soon to be released Fedora 17. How will this improve the packager's life one may ask?  - Do you know which package e.g. javax.servlet bundle is in? You don't have to anymore. Put Requires: osgi(javax.servlet) in your spec file is all you need. Or you can use yum to query for it `repoquery --whatprovides "osgi(javax.servlet)"`
  • Require-Bundle headers are used for auto generating requires so people don't end up with dependencies not being installed. The format used is pretty much the same Requires: osgi(bundle-symbolicName). Few things needs mentioning here - optional requires are discarded because RPM(in Fedora) doesn't have the concept of soft-requires and versions are not added to the requires for a reason that will be explained later. The requires generator have just been enabled in Rawhide(future Fedora 18) and up to now it's showing no major problems with a pretty huge test case - the Eclipse SDK.
 Import/Export-Package is a huge part of the OSGi specification and even the recommended way for specifying dependencies. Here I'll explain why it doesn't make sense for RPM packages without questioning it's general usability, it just doesn't fit our workflow.
  • having provides/requires for every package import/export is way too verbose and will polute the rpm metadata that much to make it hurt every user with unneeded time to download metadata
  • we don't want multiple providers for the same package - we want  a single, stable and well tested provider. 
This is big enough difference to even make me think of Step 2 where Import-Package headers are used at build time to map them to Require-Bundle. This will not be an easy task but it is possible because at build time we know which bundle exports the needed package. 
Now to not having versions in requires. First of all there is no easy mapping between OSGi version ranges and RPM. OSGi recommends using version ranges while in RPM it's considered a really bad practice to use anything else but setting the lowest version supported. Second, we usually have only one package that provides given osgi bundle. Third , more than one OSGi bundles are usually grouped in a single RPM. All of the above combined makes the requires generated satisfying 90 % or even more of the cases and for the rest one can still add the requires manually (like we do now).
This isn't by any mean a complete solution it covers only the most common case but still it will prevent a number of problems like Eclipse failing to provision because of missing requires for some plugin and etc. Once the current automation makes its way through the whole distribution work on the next integration step can continue because there would be less time spend on simple but timeconsuming things.


10 comments:

Ferry said...

You're pretty much talking about OBR, why not put up an OBR repository for Fedora java RPMs?

And not using the Require-Package completely dilutes the dependency graph and utterly defeats (some of) the main goals of OSGi.

As an example consider an application that wants to use log4j version A and B (this is _not_ hypothetical!).
By your arguments this cannot occur in Fedora because you want only 1 provider, which means that Fedora is pointless w.r.t. OSGi. And I guess you don't want that ;-)

Neil Bartlett said...

Ferry is right. This doesn't sound like it will be useful for the vast majority of OSGi bundles since we use Import-Package rather than Require-Bundle to specify dependencies.... and why would OSGi developers rewrite our bundles to a less correct form in order to satisfy Fedora?

Incidentally I thought that RPM did have the feature of capabilities and requirements for resources, which maps cleanly onto exported/imported packages and is more powerful that "virtual packages" provided by other package managers like Debian.

Alexander Kurtakov said...

Ferry, OBR has nothing to do with the subject because thing will still be installed via RPM and whether there is or isn't an OBR repository wouldn't help anything unless it's integrated to RPM internals. And it will end up to be something similar if not even the same.
As much as your example with log4j is not hypothetical (tell me about it :) this is not something you want to happen in your product (where our product is Fedora). For example Eclipse Classic SDK never ships with two different lucene versions or two different commons-codec versions and etc. Source gets modified to the state where it uses only one version - same for Fedora. It is possible to have multiple versions but people work hard for this to not happen as this is not manageable.

Alexander Kurtakov said...

Neil, I have never mentioned that anyone should change to use Require-Bundle. I said that it doesn't match our workflow and we would not automate this case (for now !). People are free to use whatever is supported by the framework and works best for them, right? And others automate things to the state that works best for them.
I have no idea what do you ment by the capabilities and requirements and the provides generated are more than a virtual packages. See http://koji.fedoraproject.org/koji/rpminfo?rpmID=2969128 for the provides from a single rpm maybe this is what you call capabilities? But still Import-Package is way too verbose and ends up in MBs of metadata generated that will be fetched on every update which is something we can't accept even though we know that we do not automate all the cases but it's most of the cases we hit.
If you have other ideas about RPM integration I'm more than open to hear about it and discuss it.

Ferry said...

It get the difference between RPMs and OBRs :-)

However, OBR was specifically designed to fit perfectly onto the OSGi dependency model. Diluting it is not a good idea.

One of the major powerful features of OSGi is to be able to use multiple versions of the same bundle at the same time.

This is a _real_ situation in some application.

Your example of Eclipse of just an example of an application that doesn't have this issue. It's not proof of the contrary, just as 'A pear is a fruit, therefore all fruits are pears' is invalid.

Your argument that source is modified is obviously only valid if you have control over the source code...

Then your argument about multiple versions not being manageable: OSGi manages this situation just fine (perfectly). It's one of its strengths.

Peter Kriens and Neil Bartlett can provide you with many detailed examples...

Maybe you want to read up on OSGi? ;-)

Alexander Kurtakov said...

Bingo,

>>Your argument that source is modified is obviously only valid if you have control over the source code...

In Fedora we do have that control - it's pure open source. Not only the source code should be available but it should be buildable in an open and reproducible way.

>> Then your argument about multiple versions not being manageable: OSGi manages this situation just fine (perfectly). It's one of its strengths.

Well, there are way more issues than runtime management of dependencies. What about security updates? Very often they are not easily backportable and backporting to different versions requires so much more work that very often it becomes unmanageable. What about QA needed to test few versions before firing an update? What about distribution/mirroring/installation ?

OSGi handles just the runtime part of this and it does it pretty well. But the rest is something that RPM is responsible for in our case. And it also does it damn good. So we need to make them talk together even if the conversation wouldn't be perfect in the beginning.

Ferry said...

>> In Fedora we do have that control - it's pure open source. Not only the source code should be available but it should be buildable in an open and reproducible way.

Bingo back ;-)

Yes _you_ do have that control, others don't; which implies that it's not a good idea to impose that situation on others.
Imposing that on others means that Fedora is not a platform on which the afore mentioned application can be built (through RPM).

People building such application will then always have to use a real OBR repo and completely bypass the RPMs, which makes packaging them pointless...

Alexander Kurtakov said...

>> People building such application will then always have to use a real OBR repo and completely bypass the RPMs, which makes packaging them pointless...

That's not true. The fact that something is installed via RPM doesn't enforce any runtime change on consumers. You still have the very same bundles(jars) in the very same place you have put them in the RPM build. Whether or not you will have them in OBR repo doesn't changes anything with regards to RPM exposing part of the OSGi metadata.

Ferry said...

Ok, 1 final comment then :-)

It sounds like you're trying to solve the dependency problem of non-OSGi java by employing (parts of) OSGi.

See there the mismatch ;-)

1- The non-OSGi java dependency situation is a complete mess and - for all practical purposes - can't be solved. It's the classical 'Maven is downloading the Internet'.

2 - Employing parts of OSGi doesn't solve the situation for OSGi java.

Combine 1 and 2, and you have a solution that doesn't actually solve anything.

Redação iMasters said...

Hi, Alexander!

I’m the web editor at iMasters, one of the largest developer communities in Brazil. I´d like to talk to you about republishing your article at our site. Can you contact me at rina.noronha@imasters.com.br?

Bests,
Rina Noronha
Journalist – web editor
www.imasters.com.br
redacao@imasters.com.br
rina.noronha@imasters.com.br
+55 27 3327-0320 / +55 27 9973-0700