LSB and the GNU build system

Submitted by john.dallman on Tue, 07/01/2008 - 10:27.

This may be stunningly naïve: For a sequence of historical reasons, I have never become closely familiar with the GNU build system. If I'm talking nonsense, just tell me. My employers produce some products that use a lot of LGPL open-source libraries. Many of these are not built as part of the main product builds, which use in-house build systems; instead, they're built with the GNU build system, and the resulting libraries are then stored as binary components in the source-management system. Building these binary components is not something that is done every week, and is not the subject of attention from high-powered people; it's a technician sort of a job. This means that the idea of making these components LSB-compliant in their Linux builds, as a necessary step towards making the products as a whole LSB-compliant, does not have a whole load of traction. It needs somebody to modify all those separate instances of the build system, and to do it again when we get updated source packages. Not an appealing idea to development managers with limited resources to work with, who are still trying to understand why the different flavours of Linux aren't automatically binary-compatible. But there's an obvious way to do an end run round the problem: should the GNU build system have options for building LSB-compliant libraries? -- John Dallman Parasolid Porting Engineer Siemens PLM Software 46 Regent Street, Cambridge, CB2 1DP United Kingdom Tel: +44-1223-371554 john.dallman@siemens.com www.siemens.com/plm

LSB and the GNU build system
Submitted by tytso on Tue, 07/01/2008 - 12:45.

On Tue, Jul 01, 2008 at 12:23:34PM +0200, Dallman, John wrote:
> But there's an obvious way to do an end run round the problem: should
> the GNU build system have options for building LSB-compliant libraries?

There are three answers to your question. The first is that is GNU
build system is too low-level to deal with LSB compliance issues.
It's not just matter of options to the C compiler and linker, but it's
also specific libraries that go far beyond what is shipped by the GNU
project. (For example, things like the Qt libraries, in the next
version of the LSB, the NSS libraries, etc.) So the problem is one of
simply being at the wrong level of abstraction.

The second answer is that making LSB-compliant libraries is a lot more
than just using a different compiler toolchain. Using the LSB SDK
will solve a number of issues flagged by the LSB Application Checker
(currently known as the Application Testkit Manager), but by itself it
is not enough. In some cases, you have to make source-level changes.
For example, certain interfaces such as memalign are not supported in
the LSB; but the simple replacement of that interface with
posix_memalign() is all that is needed to solve that particular
problem. In other cases, because the OpenSSL library does not have a
stable ABI, in LSB 4.0 we will be adding the NSS library to provide
equivalent functionality --- but if your code is currently written to
use the OpenSSL library, it will need to be changed to use the NSS
library.

The third observation is that even if the GNU tools magically gained
the ability to generate LSB-compliant libraries tomorrow, you would
still have to rebuild all of your libraries with the new GNU tools.

With LSB 4.0, we will be shipping a new LSB SDK that should make it
easier to use and integrate into existing build systems, and so what I
would suggest doing is to make plans to start next year to use the LSB
SDK (which is basically a set of wrappers around gcc and ld, plus a
set of header files and other configuration files) instead of using
the GNU toolchain directly. It may be that for your company the
transition to using the LSB is something that will have to be done
gradually, as your various LGPL binary components are gradually
converted over to being compiled with the LSB toolchain, and as each
binary component is modified as necessary for maximum
cross-distribution compatibility and LSB conformance.

> Not an appealing
> idea to development managers with limited resources to work with, who
> are still trying to understand why the different flavours of Linux aren't
> automatically binary-compatible.

One thing to point out to your managers is that very often there are
incompatibilities between Windows 98, ME, XP, 2000, and Vista, even
for applications that only try to restrict themselves to the Win32
API. And of course, it's very easy to accidentally use some new
interface in Windows which only works on newer Windows systems. So
this problem isn't unique to Linux. The LSB is intended to help solve
this problem, but it is not a panacea.

The other thing to point out is that there are advantages to their
being multiple competing flavors of Linux; it means that you don't
have a single monopolistic company (either Microsoft or Apple)
dictating changes unilaterally to customers and ISVs. (It's not just
Microsoft; observe what happened to Adobe when without warning
announced that they were dropping the 64-bit Carbon APIs from Leopard
on the first day of WWDC 2007; as a result the "the fastest way to run
Photoshop CS4 on a Mac will be to run it on Windows[1].)

[1] http://daringfireball.net/2008/04/64000_question

So with the Linux ecosystem, you have to take the bad with the good.
The LSB is intended to be a way to fix up some of the less pleasant
aspects of the fact that there are multiple competing flavors of
Linux.

The one thing I would add in conclusion is that it's great that you
are asking these questions, and if you are interested in a few months
when we have some initial versions of these tools that would be ready
for a beta-test, it would be great if we could include you on the beta
program, and get back your feedback about how easy it is to replace
the use of the raw GNU toolchain with the LSB toolset. We also have a
pre-release version of the next version of the Linux Application
Checker which may be useful as you are trying to determine how
cross-platform portable your applications and libraries are, and what
you might need to do to improve their portability.

Best regards,

- Ted

LSB and the GNU build system
Submitted by john.dallman on Tue, 07/01/2008 - 12:45.

Theodore Tso [mailto:tytso@mit.edu] wrote:

> The first is that is GNU build system is too low-level to deal
> with LSB compliance issues. ... making LSB-compliant libraries
> is a lot more than just using a different compiler toolchain.

OK, that answers the question.

> With LSB 4.0, we will be shipping a new LSB SDK that should make it
> easier to use and integrate into existing build systems, and so what I
> would suggest doing is to make plans to start next year to use the LSB
> SDK (which is basically a set of wrappers around gcc and ld, plus a
> set of header files and other configuration files) instead of using
> the GNU toolchain directly.

OK. The stuff I work on, because it is basically math libraries, is
already built with the LSB 3.1 SDK. But that's only a small part of
the company's products.

How thoroughly will LSB 4.0 support linking in shared libraries that
are built with LSB 3.1?

Will it be possible to install the 4.0 and 3.1 SDKs on the same
machine and use the appropriate one in a script by setting up
environment variables, or will it remain one SDK per machine?

> It may be that for your company the transition to using the LSB
> is something that will have to be done gradually,

Definitely. There is a lot of code in the products that is not
LGPL, but proprietary, and while that's built with GCC on Linux, it
is not built with GNU configure, et al. There's also code that
will likely never be LSB-compliant, because it uses Motif, and
the people who work on it aren't interested in changing to another
implementation of Motif just for LSB compliance. Yes, I know the
reasons for LSB not including Motif. I'm having to work this idea
in gently and in stages.

> One thing to point out to your managers is that very often there are
> incompatibilities between Windows 98, ME, XP, 2000, and Vista, even
> for applications that only try to restrict themselves to the Win32
> API. And of course, it's very easy to accidentally use some new
> interface in Windows which only works on newer Windows systems.

Ah. We make big, heavy pieces of CAD and PLM software. We never
supported Windows 9x; we do our builds and tests on the oldest
version of Windows (currently XP) that is supported. That strategy
provides considerable protection from the vagaries of Microsoft. We
work in a similar way on AIX, Solaris, HP-UX, et al. People are very
used to that kind of way of working, and I still have to explain from
time to time that asking what version number of Linux something
is isn't a meaningful question.

> Microsoft; observe what happened to Adobe when without warning
> announced that they were dropping the 64-bit Carbon APIs from Leopard

Err, you mean Apple?

> The one thing I would add in conclusion is that it's great that you
> are asking these questions, and if you are interested in a few months
> when we have some initial versions of these tools that would be ready
> for a beta-test, it would be great if we could include you on the beta
> program, and get back your feedback about how easy it is to replace
> the use of the raw GNU toolchain with the LSB toolset. We also have a
> pre-release version of the next version of the Linux Application
> Checker which may be useful as you are trying to determine how
> cross-platform portable your applications and libraries are, and what
> you might need to do to improve their portability.

I'd like to say yes, but I doubt I can. I haven't been able to motivate
the other groups strongly about LSB, because of the Motif issue: they
largely regard the incompatibilities between versions as Linux bugs that
Linux needs to fix. I still have to explain, for example, that the
incompatible change from libstdc++.so.5 to libstdc++.so.6 is not a bug,
but a deliberate change, and is not going to be fixed.

The company is not doing Linux because it wants to do Linux. It's doing
it because customers want to run its products on Linux. And while the
customers don't ask for LSB, and LSB has limitations like Motif, LSB
doesn't get much traction.

--
John Dallman
Parasolid Porting Engineer

Siemens PLM Software
46 Regent Street, Cambridge, CB2 1DP
United Kingdom
Tel: +44-1223-371554
john.dallman@siemens.com
www.siemens.com/plm

LSB and the GNU build system
Submitted by Wichmann Mats D on Tue, 07/01/2008 - 15:15.

Dallman, John wrote:

John - Siemens? This is new?

> Theodore Tso [mailto:tytso@mit.edu] wrote:
>
>> The first is that is GNU build system is too low-level to deal
>> with LSB compliance issues. ... making LSB-compliant libraries
>> is a lot more than just using a different compiler toolchain.
>
> OK, that answers the question.

I should note that lsbcc is designed to operate well within the
gnu configure system (less smooth for additional cleverness coming
from libtool, and the somewhat similar pkg-config). configure
depends on performing compilation of small testcases to find out
if features are available or not, allowing decisions to be made
that will make the build go smoothly, instead of deferring the
problems to build time. This means that if the configure tests
are performed using lsbcc you are running these checks in the
same environment as the eventual build, so this often gets you a
long way towards a clean build:

CC=lsbcc ./configure [options]

It's not magic, of course; it only affects those items which the
programmers chose to code configure checks for, and despite how
much control this gives you over portability, you still find lots
of cases where assumptions are made and not checked for.

>> With LSB 4.0, we will be shipping a new LSB SDK that should make it
>> easier to use and integrate into existing build systems, and so what
>> I would suggest doing is to make plans to start next year to use the
>> LSB SDK (which is basically a set of wrappers around gcc and ld,
>> plus a set of header files and other configuration files) instead of
>> using the GNU toolchain directly.
>
> OK. The stuff I work on, because it is basically math libraries, is
> already built with the LSB 3.1 SDK. But that's only a small part of
> the company's products.
>
> How thoroughly will LSB 4.0 support linking in shared libraries that
> are built with LSB 3.1?
>
> Will it be possible to install the 4.0 and 3.1 SDKs on the same
> machine and use the appropriate one in a script by setting up
> environment variables, or will it remain one SDK per machine?

We're just wrapping up the implementation of a multi-version lsbcc,
so it will actually be one sdk with the bits for all the recent
LSB versions (starting at 3.0) installed concurrently. If you keep
up with us on irc (or here, for that matter), I can let you know
when this is stable enough to pull from our development snapshots
and experiment with.

>
>> It may be that for your company the transition to using the LSB
>> is something that will have to be done gradually,
>
> Definitely. There is a lot of code in the products that is not
> LGPL, but proprietary, and while that's built with GCC on Linux, it
> is not built with GNU configure, et al. There's also code that
> will likely never be LSB-compliant, because it uses Motif, and
> the people who work on it aren't interested in changing to another
> implementation of Motif just for LSB compliance. Yes, I know the
> reasons for LSB not including Motif. I'm having to work this idea
> in gently and in stages.
>
>> One thing to point out to your managers is that very often there are
>> incompatibilities between Windows 98, ME, XP, 2000, and Vista, even
>> for applications that only try to restrict themselves to the Win32
>> API. And of course, it's very easy to accidentally use some new
>> interface in Windows which only works on newer Windows systems.
>
> Ah. We make big, heavy pieces of CAD and PLM software. We never
> supported Windows 9x; we do our builds and tests on the oldest
> version of Windows (currently XP) that is supported. That strategy
> provides considerable protection from the vagaries of Microsoft.

Yeah, I don't think this argument will fly with management types,
who have too many conceptions about stuff working fine across every
Windows version (often ignoring a lot of work that went into that,
just like with Linux ports of course). I just hear that comment too
often.

> The company is not doing Linux because it wants to do Linux. It's
> doing it because customers want to run its products on Linux. And
> while the customers don't ask for LSB, and LSB has limitations like
> Motif, LSB doesn't get much traction.

Seems like in the UNIX world, a lot of apps bundle Motif (or used
to, all those years ago when I was paying attention). There are
some reasons why an LSB-friendly Motif lib can't be built right
now, but work towards LSB 4.0 should get us a lot closer to being
able to do that. I don't think we'll ever get to where Motif itself
goes into LSB, since there are distros with issues about including
it as a "mandatory" feature. Would that help?

-- mats

LSB and the GNU build system
Submitted by john.dallman on Tue, 07/01/2008 - 15:30.

> John - Siemens? This is new?

UGS Corp was taken over by Siemens, and is now "Siemens PLM Software".
Same people, same products.

> We're just wrapping up the implementation of a multi-version lsbcc,
> so it will actually be one sdk with the bits for all the recent
> LSB versions (starting at 3.0) installed concurrently. If you keep
> up with us on irc (or here, for that matter), I can let you know
> when this is stable enough to pull from our development snapshots
> and experiment with.

OK, I'll ask in a while. I have a vast preference for e-mail over IRC,
IM, or anything real-time, being a slow and error-prone typist, and
having enough to think about without several different channels
clamouring for my attention.

Will the multi-version stuff handle building 32-bit and 64-bit x86 on
the same 64-bit x86 machine?

> Seems like in the UNIX world, a lot of apps bundle Motif (or used
> to, all those years ago when I was paying attention). There are
> some reasons why an LSB-friendly Motif lib can't be built right
> now, but work towards LSB 4.0 should get us a lot closer to being
> able to do that. I don't think we'll ever get to where Motif itself
> goes into LSB, since there are distros with issues about including
> it as a "mandatory" feature. Would that help?

Possibly. We don't bundle Motif, and when I suggested that we could
do so as a means of overcoming its variations between UNIXes, the
idea was not popular. However, bundling Motif as a Linux-specific
thing might make more sense.

--
John Dallman
Parasolid Porting Engineer

Siemens PLM Software
46 Regent Street, Cambridge, CB2 1DP
United Kingdom
Tel: +44-1223-371554
john.dallman@siemens.com
www.siemens.com/plm

LSB and the GNU build system

Active forum topics