[sql-server] What are named pipes?

What are they and how do they work?

Context happens to be SQL Server

Unix and Windows both have things called "Named pipes", but they behave differently. On Unix, a named pipe is a one-way street which typically has just one reader and one writer - the writer writes, and the reader reads, you get it?

On Windows, the thing called a "Named pipe" is an IPC object more like a TCP socket - things can flow both ways and there is some metadata (You can obtain the credentials of the thing on the other end etc).

Unix named pipes appear as a special file in the filesystem and can be accessed with normal file IO commands including the shell. Windows ones don't, and need to be opened with a special system call (after which they behave mostly like a normal win32 handle).

Even more confusing, Unix has something called a "Unix socket" or AF_UNIX socket, which works more like (but not completely like) a win32 "named pipe", being bidirectional.

Pipes are a way of streaming data between applications. Under Linux I use this all the time to stream the output of one process into another. This is anonymous because the destination app has no idea where that input-stream comes from. It doesn't need to.

A named pipe is just a way of actively hooking onto an existing pipe and hoovering-up its data. It's for situations where the provider doesn't know what clients will be eating the data.

Linux Pipes
First In First Out (FIFO) interproccess communication mechanism.

Unnamed Pipes
On the command line, represented by a "|" between two commands.

Named Pipes
A FIFO special file. Once created, you can use the pipe just like a normal file(open, close, write, read, etc).

To create a named pipe, called "myPipe", from the command line (man page):

mkfifo myPipe  

To create a named pipe from c, where "pathname" is the name you would like the pipe to have and "mode" contains the permissions you want the pipe to have (man page):

#include <sys/types.h>
#include <sys/stat.h>
int mkfifo(const char *pathname, mode_t mode);

According to Wikipedia:

[...] A traditional pipe is "unnamed" because it exists anonymously and persists only for as long as the process is running. A named pipe is system-persistent and exists beyond the life of the process and must be "unlinked" or deleted once it is no longer being used. Processes generally attach to the named pipe (usually appearing as a file) to perform IPC (inter-process communication).

Pipes are a way of streaming data between applications. Under Linux I use this all the time to stream the output of one process into another. This is anonymous because the destination app has no idea where that input-stream comes from. It doesn't need to.

A named pipe is just a way of actively hooking onto an existing pipe and hoovering-up its data. It's for situations where the provider doesn't know what clients will be eating the data.

Named pipes is a windows system for inter-process communication. In the case of SQL server, if the server is on the same machine as the client, then it is possible to use named pipes to tranfer the data, as opposed to TCP/IP.

Inter-process communication (mostly) for Windows Applications. Similar to using sockets to communicate between applications in Unix.

MSDN

Compare

echo "test" | wc

to

mkdnod apipe p
wc apipe

wc will block until

echo "test" > apipe

executes

This is an exeprt from Technet (so not sure why the marked answer says named pipes are faster??):

Named Pipes vs. TCP/IP Sockets

In a fast local area network (LAN) environment, Transmission Control Protocol/Internet Protocol (TCP/IP) Sockets and Named Pipes clients are comparable with regard to performance. However, the performance difference between the TCP/IP Sockets and Named Pipes clients becomes apparent with slower networks, such as across wide area networks (WANs) or dial-up networks. This is because of the different ways the interprocess communication (IPC) mechanisms communicate between peers.

For named pipes, network communications are typically more interactive. A peer does not send data until another peer asks for it using a read command. A network read typically involves a series of peek named pipes messages before it starts to read the data. These can be very costly in a slow network and cause excessive network traffic, which in turn affects other network clients.

It is also important to clarify if you are talking about local pipes or network pipes. If the server application is running locally on the computer that is running an instance of SQL Server, the local Named Pipes protocol is an option. Local named pipes runs in kernel mode and is very fast.

For TCP/IP Sockets, data transmissions are more streamlined and have less overhead. Data transmissions can also take advantage of TCP/IP Sockets performance enhancement mechanisms such as windowing, delayed acknowledgements, and so on. This can be very helpful in a slow network. Depending on the type of applications, such performance differences can be significant.

TCP/IP Sockets also support a backlog queue. This can provide a limited smoothing effect compared to named pipes that could lead to pipe-busy errors when you are trying to connect to SQL Server.

Generally, TCP/IP is preferred in a slow LAN, WAN, or dial-up network, whereas named pipes can be a better choice when network speed is not the issue, as it offers more functionality, ease of use, and configuration options.

This is an exeprt from Technet (so not sure why the marked answer says named pipes are faster??):

Named Pipes vs. TCP/IP Sockets

In a fast local area network (LAN) environment, Transmission Control Protocol/Internet Protocol (TCP/IP) Sockets and Named Pipes clients are comparable with regard to performance. However, the performance difference between the TCP/IP Sockets and Named Pipes clients becomes apparent with slower networks, such as across wide area networks (WANs) or dial-up networks. This is because of the different ways the interprocess communication (IPC) mechanisms communicate between peers.

For named pipes, network communications are typically more interactive. A peer does not send data until another peer asks for it using a read command. A network read typically involves a series of peek named pipes messages before it starts to read the data. These can be very costly in a slow network and cause excessive network traffic, which in turn affects other network clients.

It is also important to clarify if you are talking about local pipes or network pipes. If the server application is running locally on the computer that is running an instance of SQL Server, the local Named Pipes protocol is an option. Local named pipes runs in kernel mode and is very fast.

For TCP/IP Sockets, data transmissions are more streamlined and have less overhead. Data transmissions can also take advantage of TCP/IP Sockets performance enhancement mechanisms such as windowing, delayed acknowledgements, and so on. This can be very helpful in a slow network. Depending on the type of applications, such performance differences can be significant.

TCP/IP Sockets also support a backlog queue. This can provide a limited smoothing effect compared to named pipes that could lead to pipe-busy errors when you are trying to connect to SQL Server.

Generally, TCP/IP is preferred in a slow LAN, WAN, or dial-up network, whereas named pipes can be a better choice when network speed is not the issue, as it offers more functionality, ease of use, and configuration options.

Inter-process communication (mostly) for Windows Applications. Similar to using sockets to communicate between applications in Unix.

MSDN

Named pipes in a unix/linux context can be used to make two different shells to communicate since a shell just can't share anything with another.

Furthermore, one script instantiated twice in the same shell can't share anything through the two instances. I found a use for named pipes when coding a daemon that contains the start() and stop() function, and I wanted to use the same script to perform the two actions.

Without named pipes (or any kind of semaphore) starting the script in the background is not a problem. The thing is when it finishes you just can't access the instance in background.

So when you want to send him the stop command you just can't: running the same script without named pipes and calling the stop() function won't do anything since you are actually running another instance.

The solution was to implement two pipes, one READ and the other WRITE when you start the daemon. Then make him, among its other tasks, listen to the READ pipe. Then the Stop() function contains a command that will write a message in the pipe, that will be handled by the background running script that will perform an exit 0. This way our second instance of the same script has only on task to do: tell the first instance to stop.

This way one and only one script can start and stop itself.

Of course you have different ways to do it by triggering the stop via a touch for example. But this one is nice and interesting to code.

Compare

echo "test" | wc

to

mkdnod apipe p
wc apipe

wc will block until

echo "test" > apipe

executes

Unix and Windows both have things called "Named pipes", but they behave differently. On Unix, a named pipe is a one-way street which typically has just one reader and one writer - the writer writes, and the reader reads, you get it?

On Windows, the thing called a "Named pipe" is an IPC object more like a TCP socket - things can flow both ways and there is some metadata (You can obtain the credentials of the thing on the other end etc).

Unix named pipes appear as a special file in the filesystem and can be accessed with normal file IO commands including the shell. Windows ones don't, and need to be opened with a special system call (after which they behave mostly like a normal win32 handle).

Even more confusing, Unix has something called a "Unix socket" or AF_UNIX socket, which works more like (but not completely like) a win32 "named pipe", being bidirectional.

Linux Pipes
First In First Out (FIFO) interproccess communication mechanism.

Unnamed Pipes
On the command line, represented by a "|" between two commands.

Named Pipes
A FIFO special file. Once created, you can use the pipe just like a normal file(open, close, write, read, etc).

To create a named pipe, called "myPipe", from the command line (man page):

mkfifo myPipe  

To create a named pipe from c, where "pathname" is the name you would like the pipe to have and "mode" contains the permissions you want the pipe to have (man page):

#include <sys/types.h>
#include <sys/stat.h>
int mkfifo(const char *pathname, mode_t mode);

Named pipes is a windows system for inter-process communication. In the case of SQL server, if the server is on the same machine as the client, then it is possible to use named pipes to tranfer the data, as opposed to TCP/IP.

Inter-process communication (mostly) for Windows Applications. Similar to using sockets to communicate between applications in Unix.

MSDN

Pipes are a way of streaming data between applications. Under Linux I use this all the time to stream the output of one process into another. This is anonymous because the destination app has no idea where that input-stream comes from. It doesn't need to.

A named pipe is just a way of actively hooking onto an existing pipe and hoovering-up its data. It's for situations where the provider doesn't know what clients will be eating the data.

Named pipes in a unix/linux context can be used to make two different shells to communicate since a shell just can't share anything with another.

Furthermore, one script instantiated twice in the same shell can't share anything through the two instances. I found a use for named pipes when coding a daemon that contains the start() and stop() function, and I wanted to use the same script to perform the two actions.

Without named pipes (or any kind of semaphore) starting the script in the background is not a problem. The thing is when it finishes you just can't access the instance in background.

So when you want to send him the stop command you just can't: running the same script without named pipes and calling the stop() function won't do anything since you are actually running another instance.

The solution was to implement two pipes, one READ and the other WRITE when you start the daemon. Then make him, among its other tasks, listen to the READ pipe. Then the Stop() function contains a command that will write a message in the pipe, that will be handled by the background running script that will perform an exit 0. This way our second instance of the same script has only on task to do: tell the first instance to stop.

This way one and only one script can start and stop itself.

Of course you have different ways to do it by triggering the stop via a touch for example. But this one is nice and interesting to code.

Named pipes is a windows system for inter-process communication. In the case of SQL server, if the server is on the same machine as the client, then it is possible to use named pipes to tranfer the data, as opposed to TCP/IP.

Unix and Windows both have things called "Named pipes", but they behave differently. On Unix, a named pipe is a one-way street which typically has just one reader and one writer - the writer writes, and the reader reads, you get it?

On Windows, the thing called a "Named pipe" is an IPC object more like a TCP socket - things can flow both ways and there is some metadata (You can obtain the credentials of the thing on the other end etc).

Unix named pipes appear as a special file in the filesystem and can be accessed with normal file IO commands including the shell. Windows ones don't, and need to be opened with a special system call (after which they behave mostly like a normal win32 handle).

Even more confusing, Unix has something called a "Unix socket" or AF_UNIX socket, which works more like (but not completely like) a win32 "named pipe", being bidirectional.

According to Wikipedia:

[...] A traditional pipe is "unnamed" because it exists anonymously and persists only for as long as the process is running. A named pipe is system-persistent and exists beyond the life of the process and must be "unlinked" or deleted once it is no longer being used. Processes generally attach to the named pipe (usually appearing as a file) to perform IPC (inter-process communication).

Named pipes is a windows system for inter-process communication. In the case of SQL server, if the server is on the same machine as the client, then it is possible to use named pipes to tranfer the data, as opposed to TCP/IP.

Compare

echo "test" | wc

to

mkdnod apipe p
wc apipe

wc will block until

echo "test" > apipe

executes

According to Wikipedia:

[...] A traditional pipe is "unnamed" because it exists anonymously and persists only for as long as the process is running. A named pipe is system-persistent and exists beyond the life of the process and must be "unlinked" or deleted once it is no longer being used. Processes generally attach to the named pipe (usually appearing as a file) to perform IPC (inter-process communication).

Unix and Windows both have things called "Named pipes", but they behave differently. On Unix, a named pipe is a one-way street which typically has just one reader and one writer - the writer writes, and the reader reads, you get it?

On Windows, the thing called a "Named pipe" is an IPC object more like a TCP socket - things can flow both ways and there is some metadata (You can obtain the credentials of the thing on the other end etc).

Unix named pipes appear as a special file in the filesystem and can be accessed with normal file IO commands including the shell. Windows ones don't, and need to be opened with a special system call (after which they behave mostly like a normal win32 handle).

Even more confusing, Unix has something called a "Unix socket" or AF_UNIX socket, which works more like (but not completely like) a win32 "named pipe", being bidirectional.

Pipes are a way of streaming data between applications. Under Linux I use this all the time to stream the output of one process into another. This is anonymous because the destination app has no idea where that input-stream comes from. It doesn't need to.

A named pipe is just a way of actively hooking onto an existing pipe and hoovering-up its data. It's for situations where the provider doesn't know what clients will be eating the data.

Linux Pipes
First In First Out (FIFO) interproccess communication mechanism.

Unnamed Pipes
On the command line, represented by a "|" between two commands.

Named Pipes
A FIFO special file. Once created, you can use the pipe just like a normal file(open, close, write, read, etc).

To create a named pipe, called "myPipe", from the command line (man page):

mkfifo myPipe  

To create a named pipe from c, where "pathname" is the name you would like the pipe to have and "mode" contains the permissions you want the pipe to have (man page):

#include <sys/types.h>
#include <sys/stat.h>
int mkfifo(const char *pathname, mode_t mode);