I never clearly understood what an ABI is. Please don't point me to a Wikipedia article. If I could understand it, I wouldn't be here posting such a lengthy post.
This is my mindset about different interfaces:
A TV remote is an interface between the user and the TV. It is an existing entity, but useless (doesn't provide any functionality) by itself. All the functionality for each of those buttons on the remote is implemented in the television set.
Interface: It is an "existing entity" layer between the
functionalityandconsumerof that functionality. An interface by itself doesn't do anything. It just invokes the functionality lying behind.Now depending on who the user is there are different type of interfaces.
Command Line Interface (CLI) commands are the existing entities, the consumer is the user and functionality lies behind.
functionality:my software functionality which solves some purpose to which we are describing this interface.
existing entities:commands
consumer:userGraphical User Interface(GUI) window, buttons, etc. are the existing entities, and again the consumer is the user and functionality lies behind.
functionality:my software functionality which solves some problem to which we are describing this interface.
existing entities:window, buttons etc..
consumer:userApplication Programming Interface(API) functions (or to be more correct) interfaces (in interfaced based programming) are the existing entities, consumer here is another program not a user, and again functionality lies behind this layer.
functionality:my software functionality which solves some problem to which we are describing this interface.
existing entities:functions, Interfaces (array of functions).
consumer:another program/application.Application Binary Interface (ABI) Here is where my problem starts.
functionality:???
existing entities:???
consumer:???
ABIs cover details such as
- data type, size, and alignment;
- the calling convention, which controls how functions' arguments are passed and return values retrieved;
- the system call numbers and how an application should make system calls to the operating system;
Other ABIs standardize details such as
- the C++ name mangling,
- exception propagation, and
- calling convention between compilers on the same platform, but do not require cross-platform compatibility.
Who needs these details? Please don't say the OS. I know assembly programming. I know how linking & loading works. I know exactly what happens inside.
Why did C++ name mangling come in? I thought we are talking at the binary level. Why do languages come in?
Anyway, I've downloaded the [PDF] System V Application Binary Interface Edition 4.1 (1997-03-18) to see what exactly it contains. Well, most of it didn't make any sense.
Why does it contain two chapters (4th & 5th) to describe the ELF file format? In fact, these are the only two significant chapters of that specification. The rest of the chapters are "processor specific". Anyway, I though that it is a completely different topic. Please don't say that ELF file format specifications are the ABI. It doesn't qualify to be an interface according to the definition.
I know, since we are talking at such a low level it must be very specific. But I'm not sure how is it "instruction set architecture (ISA)" specific?
Where can I find Microsoft Windows' ABI?
So, these are the major queries that are bugging me.
This question is related to
compiler-construction
abi
There are various interpretation and strong opinions of the exact layer that define an ABI (application binary interface).
In my view an ABI is a subjective convention of what is considered a given/platform for a specific API. The ABI is the "rest" of conventions that "will not change" for a specific API or that will be addressed by the runtime environment: executors, tools, linkers, compilers, jvm, and OS.
If you want to use a library like joda-time you must declare a dependency on joda-time-<major>.<minor>.<patch>.jar. The library follows best practices and use Semantic Versioning. This defines the API compatibility at three levels:
In order for you to use a new major release of the same library a lot of other conventions are still to be respected:
For example, Java standardized all these conventions, not in a tool, but in a formal JVM specification. The specification allowed other vendors to provide a different set of tools that can output compatible libraries.
Java provides two other interesting case studies for ABI: Scala versions and Dalvik virtual machine.
The Dalvik VM needs a different type of bytecode than the Java bytecode. The Dalvik libraries are obtained by converting the Java bytecode (with same API) for Dalvik. In this way you can get two versions of the same API: defined by the original joda-time-1.7.2.jar. We could call it joda-time-1.7.2.jar and joda-time-1.7.2-dalvik.jar. They use a different ABI one is for the stack-oriented standard Java vms: Oracle's one, IBM's one, open Java or any other; and the second ABI is the one around Dalvik.
Scala doesn't have binary compatibility between minor Scala versions: 2.X . For this reason the same API "io.reactivex" %% "rxscala" % "0.26.5" has three versions (in the future more): for Scala 2.10, 2.11 and 2.12. What is changed? I don't know for now, but the binaries are not compatible. Probably the latest versions adds things that make the libraries unusable on the old virtual machines, probably things related to linking/naming/parameter conventions.
Java has problems with the major releases of the JVM too: 4,5,6,7,8,9. They offer only backward compatibility. Jvm9 knows how to run code compiled/targeted (javac's -target option) for all other versions, while JVM 4 doesn't know how to run code targeted for JVM 5. All these while you have one joda-library. This incompatibility flies bellow the radar thanks to different solutions:
API and ABI are just conventions on how you define compatibility. The lower layers are generic in respect of a plethora of high level semantics. That's why it's easy to make some conventions. The first kind of conventions are about memory alignment, byte encoding, calling conventions, big and little endian encodings, etc. On top of them you get the executable conventions like others described, linking conventions, intermediate byte code like the one used by Java or LLVM IR used by GCC. Third you get conventions on how to find libraries, how to load them (see Java classloaders). As you go higher and higher in concepts you have new conventions that you consider as a given. That's why they didn't made it to the semantic versioning. They are implicit or collapsed in the major version. We could amend semantic versioning with <major>-<minor>-<patch>-<platform/ABI>. This is what is actually happening already: platform is already a rpm, dll, jar (JVM bytecode), war(jvm+web server), apk, 2.11 (specific Scala version) and so on. When you say APK you already talk about a specific ABI part of your API.
The top level of an abstraction (the sources written against the highest API can be recompiled/ported to any other lower level abstraction.
Let's say I have some sources for rxscala. If the Scala tools are changed I can recompile them to that. If the JVM changes I could have automatic conversions from the old machine to the new one without bothering with the high level concepts. While porting might be difficult will help any other client. If a new operating system is created using a totally different assembler code a translator can be created.
There are APIs that are ported in multiple languages like reactive streams. In general they define mappings to specific languages/platforms. I would argue that the API is the master specification formally defined in human language or even a specific programming language. All the other "mappings" are ABI in a sense, else more API than the usual ABI. The same is happening with the REST interfaces.
If you know assembly and how things work at the OS-level, you are conforming to a certain ABI. The ABI govern things like how parameters are passed, where return values are placed. For many platforms there is only one ABI to choose from, and in those cases the ABI is just "how things work".
However, the ABI also govern things like how classes/objects are laid out in C++. This is necessary if you want to be able to pass object references across module boundaries or if you want to mix code compiled with different compilers.
Also, if you have an 64-bit OS which can execute 32-bit binaries, you will have different ABIs for 32- and 64-bit code.
In general, any code you link into the same executable must conform to the same ABI. If you want to communicate between code using different ABIs, you must use some form of RPC or serialization protocols.
I think you are trying too hard to squeeze in different types of interfaces into a fixed set of characteristics. For example, an interface doesn't necessarily have to be split into consumers and producers. An interface is just a convention by which two entities interact.
ABIs can be (partially) ISA-agnostic. Some aspects (such as calling conventions) depend on the ISA, while other aspects (such as C++ class layout) do not.
A well defined ABI is very important for people writing compilers. Without a well defined ABI, it would be impossible to generate interoperable code.
EDIT: Some notes to clarify:
ABI - Application Binary Interface is about a machine code communication in runtime between two binary parts like - application, library, OS... ABI describes how objects are saved in memory, how functions are called(calling convention), mangling...
A good example of API and ABI is iOS ecosystem with Swift language.
Application layer - When you create an application using different languages. For example you can create application using Swift and Objective-C[Mixing Swift and Objective-C]
Application - OS layer - runtime - Swift runtime and standard libraries are parts of OS and they should not be included into each bundle(e.g. app, framework). It is the same as like Objective-C uses
Library layer - Module Stability case - compile time - you will be able to import a framework which was built with another version of Swift's compiler. It means that it is safety to create a closed-source(pre-build) binary which will be consumed by a different version of compiler( .swiftinterface is used with .swiftmodule) and you will not get
Module compiled with _ cannot be imported by the _ compiler
Library layer - Library Evolution case
An application binary interface (ABI) is similar to an API, but the function is not accessible to the caller at source code level. Only a binary representation is accessible/available.
ABIs may be defined at the processor-architecture level or at the OS level. The ABIs are standards to be followed by the code-generator phase of the compiler. The standard is fixed either by the OS or by the processor.
Functionality: Define the mechanism/standard to make function calls independent of the implementation language or a specific compiler/linker/toolchain. Provide the mechanism which allows JNI, or a Python-C interface, etc.
Existing entities: Functions in machine code form.
Consumer: Another function (including one in another language, compiled by another compiler, or linked by another linker).
In order to call code in shared libraries, or call code between compilation units, the object file needs to contain labels for the calls. C++ mangles the names of method labels in order to enforce data hiding and allow for overloaded methods. That is why you cannot mix files from different C++ compilers unless they explicitly support the same ABI.
Functionality: A set of contracts which affect the compiler, assembly writers, the linker, and the operating system. The contracts specify how functions are laid out, where parameters are passed, how parameters are passed, how function returns work. These are generally specific to a (processor architecture, operating system) tuple.
Existing entities: parameter layout, function semantics, register allocation. For instance, the ARM architectures has numerous ABIs (APCS, EABI, GNU-EABI, never mind a bunch of historical cases) - using the a mixed ABI will result in your code simply not working when calling across boundaries.
Consumer: The compiler, assembly writers, operating system, CPU specific architecture.
Who needs these details? The compiler, assembly writers, linkers which do code generation (or alignment requirements), operating system (interrupt handling, syscall interface). If you did assembly programming, you were conforming to an ABI!
C++ name mangling is a special case - its a linker and dynamic linker centered issue - if name mangling is not standardized, then dynamic linking will not work. Henceforth, the C++ ABI is called just that, the C++ ABI. It is not a linker level issue, but instead a code generation issue. Once you have a C++ binary, it is not possible to make it compatible with another C++ ABI (name mangling, exception handling) without recompiling from source.
ELF is a file format for the use of a loader and dynamic linker. ELF is a container format for binary code and data, and as such specifies the ABI of a piece of code. I would not consider ELF to be an ABI in the strict sense, as PE executables are not an ABI.
All ABIs are instruction set specific. An ARM ABI will not make sense on an MSP430 or x86_64 processor.
Windows has several ABIs - for instance, fastcall and stdcall are two common use ABIs. The syscall ABI is different again.
Application binary interface (ABI)
Functionality:
Existing entities:
consumer:
These are needed by whoever has to ensure that build tool-chains work as a whole. If you write one module in assembly language, another in Python, and instead of your own boot-loader want to use an operating system, then your "application" modules are working across "binary" boundaries and require agreement of such "interface".
C++ name mangling because object files from different high-level languages might be required to be linked in your application. Consider using GCC standard library making system calls to Windows built with Visual C++.
ELF is one possible expectation of the linker from an object file for interpretation, though JVM might have some other idea.
For a Windows RT Store app, try searching for ARM ABI if you really wish to make some build tool-chain work together.
The best way to differentiate between ABI and API is to know why and what is it used for:
For x86-64 there is generally one ABI (and for x86 32-bit there is another set):
http://www.x86-64.org/documentation/abi.pdf
http://people.freebsd.org/~obrien/amd64-elf-abi.pdf
Linux + FreeBSD + MacOSX follow it with some slight variations. And Windows x64 have its own ABI:
http://eli.thegreenplace.net/2011/09/06/stack-frame-layout-on-x86-64/
Knowing the ABI and assuming other compiler follows it as well, then the binaries theoretically know how to call each other (libraries API in particular) and pass parameters over the stack or by registers etc. Or what registers will be changed upon calling the functions etc. Essentially these knowledge will help software to integrate with one another. Knowing the order of the registers / stack layout I can easily piece together different software written in assemblies together without much problem.
But API are different:
It is a high level functions names, with argument defined, such that if different software pieces build using these API, MAY be able to call into one another. But an additional requirement of SAME ABI must be adhered to.
For example, Windows used to be POSIX API compliant:
https://en.wikipedia.org/wiki/Windows_Services_for_UNIX
https://en.wikipedia.org/wiki/POSIX
And Linux is POSIX compliant as well. But the binaries cannot be just moved over and run immediately. But because they used the same NAMES in the POSIX compliant API, you can take the same software in C, recompile it in the different OS, and immediately get it running.
API are meant to ease integration of software - pre-compilation stage. So after compilation the software can look totally different - if the ABI are different.
ABI are meant to define exact integration of software at the binary / assembly level.
Let me at least answer a part of your question. With an example of how the Linux ABI affects the systemcalls, and why that is usefull.
A systemcall is a way for a userspace program to ask the kernelspace for something. It works by putting the numeric code for the call and the argument in a certain register and triggering an interrupt. Than a switch occurs to kernelspace and the kernel looks up the numeric code and the argument, handles the request, puts the result back into a register and triggers a switch back to userspace. This is needed for example when the application wants to allocate memory or open a file (syscalls "brk" and "open").
Now the syscalls have short names "brk", etc. and corresponding opcodes, these are defined in a system specific header file. As long as these opcodes stay the same you can run the same compiled userland programs with different updated kernels without having to recompile. So you have an interface used by precompiled binarys, hence ABI.
Linux shared library minimal runnable ABI example
In the context of shared libraries, the most important implication of "having a stable ABI" is that you don't need to recompile your programs after the library changes.
So for example:
if you are selling a shared library, you save your users the annoyance of recompiling everything that depends on your library for every new release
if you are selling closed source program that depends on a shared library present in the user's distribution, you could release and test less prebuilts if you are certain that ABI is stable across certain versions of the target OS.
This is specially important in the case of the C standard library, which many many programs in your system link to.
Now I want to provide a minimal concrete runnable example of this.
main.c
#include <assert.h>
#include <stdlib.h>
#include "mylib.h"
int main(void) {
mylib_mystruct *myobject = mylib_init(1);
assert(myobject->old_field == 1);
free(myobject);
return EXIT_SUCCESS;
}
mylib.c
#include <stdlib.h>
#include "mylib.h"
mylib_mystruct* mylib_init(int old_field) {
mylib_mystruct *myobject;
myobject = malloc(sizeof(mylib_mystruct));
myobject->old_field = old_field;
return myobject;
}
mylib.h
#ifndef MYLIB_H
#define MYLIB_H
typedef struct {
int old_field;
} mylib_mystruct;
mylib_mystruct* mylib_init(int old_field);
#endif
Compiles and runs fine with:
cc='gcc -pedantic-errors -std=c89 -Wall -Wextra'
$cc -fPIC -c -o mylib.o mylib.c
$cc -L . -shared -o libmylib.so mylib.o
$cc -L . -o main.out main.c -lmylib
LD_LIBRARY_PATH=. ./main.out
Now, suppose that for v2 of the library, we want to add a new field to mylib_mystruct called new_field.
If we added the field before old_field as in:
typedef struct {
int new_field;
int old_field;
} mylib_mystruct;
and rebuilt the library but not main.out, then the assert fails!
This is because the line:
myobject->old_field == 1
had generated assembly that is trying to access the very first int of the struct, which is now new_field instead of the expected old_field.
Therefore this change broke the ABI.
If, however, we add new_field after old_field:
typedef struct {
int old_field;
int new_field;
} mylib_mystruct;
then the old generated assembly still accesses the first int of the struct, and the program still works, because we kept the ABI stable.
Here is a fully automated version of this example on GitHub.
Another way to keep this ABI stable would have been to treat mylib_mystruct as an opaque struct, and only access its fields through method helpers. This makes it easier to keep the ABI stable, but would incur a performance overhead as we'd do more function calls.
API vs ABI
In the previous example, it is interesting to note that adding the new_field before old_field, only broke the ABI, but not the API.
What this means, is that if we had recompiled our main.c program against the library, it would have worked regardless.
We would also have broken the API however if we had changed for example the function signature:
mylib_mystruct* mylib_init(int old_field, int new_field);
since in that case, main.c would stop compiling altogether.
Semantic API vs Programming API
We can also classify API changes in a third type: semantic changes.
The semantic API, is usually a natural language description of what the API is supposed to do, usually included in the API documentation.
It is therefore possible to break the semantic API without breaking the program build itself.
For example, if we had modified
myobject->old_field = old_field;
to:
myobject->old_field = old_field + 1;
then this would have broken neither programming API, nor ABI, but main.c the semantic API would break.
There are two ways to programmatically check the contract API:
formal verification. Harder to do, but produces mathematical proof of correctness, essentially unifying documentation and tests into a "human" / machine verifiable manner! As long as there isn't a bug in your formal description of course ;-)
This concept is closely related to the formalization of Mathematics itself: https://math.stackexchange.com/questions/53969/what-does-formal-mean/3297537#3297537
List of everything that breaks C / C++ shared library ABIs
TODO: find / create the ultimate list:
Java minimal runnable example
What is binary compatibility in Java?
Tested in Ubuntu 18.10, GCC 8.2.0.
The term ABI is used to refer to two distinct but related concepts.
When talking about compilers it refers to the rules used to translate from source-level constructs to binary constructs. How big are the data types? how does the stack work? how do I pass parameters to functions? which registers should be saved by the caller vs the callee?
When talking about libraries it refers to the binary interface presented by a compiled library. This interface is the result of a number of factors including the source code of the library, the rules used by the compiler and in some cases definitions picked up from other libraries.
Changes to a library can break the ABI without breaking the API. Consider for example a library with an interface like.
void initfoo(FOO * foo)
int usefoo(FOO * foo, int bar)
void cleanupfoo(FOO * foo)
and the application programmer writes code like
int dostuffwithfoo(int bar) {
FOO foo;
initfoo(&foo);
int result = usefoo(&foo,bar)
cleanupfoo(&foo);
return result;
}
The application programmer doesn't care about the size or layout of FOO, but the application binary ends up with a hardcoded size of foo. If the library programmer adds an extra field to foo and someone uses the new library binary with the old application binary then the library may make out of bounds memory accesses.
OTOH if the library author had designed their API like.
FOO * newfoo(void)
int usefoo(FOO * foo, int bar)
void deletefoo((FOO * foo, int bar))
and the application programmer writes code like
int dostuffwithfoo(int bar) {
FOO * foo;
foo = newfoo();
int result = usefoo(foo,bar)
deletefoo(foo);
return result;
}
Then the application binary does not need to know anything about the structure of FOO, that can all be hidden inside the library. The price you pay for that though is that heap operations are involved.
In short and in philosophy, only things of a kind can get along well, and the ABI could be seen as the kind of which software stuff work together.
You actually don't need an ABI at all if--
An oversimplified summary:
API: "Here are all the functions you may call."
ABI: "This is how to call a function."
The ABI is set of rules that compilers and linkers adhere to in order to compile your program so that will work properly. ABIs cover multiple topics:
Taking a deeper look at calling convention, which I consider to be the core of an ABI:
The machine itself has no concept of "functions". When you write a function in a high-level language like c, the compiler generates a line of assembly code like _MyFunction1:. This is a label, which will eventually get resolved into an address by the assembler. This label marks the "start" of your "function" in the assembly code. In high-level code, when you "call" that function, what you're really doing is causing the CPU to jump to the address of that label and continue executing there.
In preparation for the jump, the compiler must do a bunch of important stuff. The calling convention is like a checklist that the compiler follows to do all this stuff:
_MyFunction1:). At this point, you can consider the CPU to be "in" your "function".There are many different ABIs / calling conventions. Some main ones are:
Here is a great page that actually shows the differences in the assembly generated when compiling for different ABIs.
Another thing to mention is that an ABI isn't only relevant inside your program's executable module. It's also used by the linker to make sure your program calls library functions correctly. You have multiple shared libraries running on your computer, and as long as your compiler knows what ABI they each use, it can call functions from them properly without blowing up the stack.
Your compiler understanding how to call library functions is extremely important. On a hosted platform (that is, one where an OS loads programs), your program can't even blink without making a kernel call.
The ABI needs to be consistent between caller and callee to be certain that the call succeeds. Stack use, register use, end-of-routine stack pop. All these are the most important parts of the ABI.
I was also trying to understand ABI and JesperE’s answer was very helpful.
From a very simple perspective, we may try to understand ABI by considering binary compatibility.
KDE wiki defines a library as binary compatible “if a program linked dynamically to a former version of the library continues running with newer versions of the library without the need to recompile.” For more on dynamic linking, refer Static linking vs dynamic linking
Now, let’s try to look at just the most basic aspects needed for a library to be binary compatibility (assuming there are no source code changes to the library):
Sure, there are many other details but this is mostly what the ABI also covers.
More specifically to answer your question, from the above, we can deduce:
ABI functionality: binary compatibility
existing entities: existing program/libraries/OS
consumer: libraries, OS
Hope this helps!
Source: Stackoverflow.com