[LeapList]Warning: ridiculously long PATH truncated

[John Simpson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wednesday 28 August 2002 16:51, Vernon Singleton wrote:
>
> What is/controls the magic number of characters which can be forced int=
o
> a PATH environment variable?
> Are there any quick tricks around that magic number?

funny... i've run into problems with long command lines before but always=
=20
assumed it was the shell that had a limit on the length of a command line=
=2E

however, your question made me curious about what the real answer is.

warning: MASSIVELY HARDCORE technical stuff follows. skip to the end if y=
ou=20
just want the answer and don't feel the urge to follow me on a trip throu=
gh=20
the kernel source code...



now that they're all gone... (=3D

i did some testing with tcsh (the shell that i normally use) setting an=20
environment variable (using "setenv") to a value of almost 300K of text. =
as=20
soon as i did this, EVERY command i ran died with "Argument list too long=
"...=20
but built-in shell commands still run with no problems.

it turns out bash does the same thing. setting a variable to a huge value=
=20
seems to work okay, but as soon as you "export" that variable (so child=20
processes inherit the value) no external programs work, all complaining o=
f=20
"Argument list too long".

the pattern? external commands can't execute, but anything totally within=
 the=20
shell works. looks like exec() is having issues, or it's something in the=
=20
initialization stuff for glibc.

a little digging about shows that the issue is how the kernel is compiled=
=2E=20
remember the thread about the maximum number of open files for a process =
or=20
for the system as a whole? those limits are controlled by the file=20
"include/linux/limits.h" in your kernel source code.

and guess what else is in there?

#define ARG_MAX         131072  /* # bytes of args + environ for exec() *=
/

maybe when the kernel handles exec() (to start running a new process), it=
=20
needs to set aside a certain amount of memory (in this case 128K) to hold=
 the=20
argument list and environment that the parent is passing to the child?

grepping thorugh the kernel's source only shows the only place this value=
 gets=20
used is in the sparc/sparc64 interface stuff (the extra hooks needed to r=
un=20
"normal" 32-bit programs on top of a 64-bit processor.)

however, looking through fs/exec.c (which is where the do_execve() functi=
on=20
actually lives) i find the function setup_arg_pages(), which seems to be=20
responsible for setting up the initial memory space for a new process. it=
=20
contains this line (fs/exec.c:300)

=09stack_base =3D STACK_TOP - MAX_ARG_PAGES*PAGE_SIZE ;

MAX_ARG_PAGES and PAGE_SIZE are defined elsewhere in the kernel source co=
de-=20
on ix86 and sparc64 these values are 16 and 4096 respectively. this is=20
because both processors do paging using 4K page sizes.

remember that exec() OVERWRITES an existing process with the new one. the=
 only=20
things that get saved are the arguments and an environment (and a few oth=
er=20
miscellaneous items not involving memory.)

when exec() sets up the memory space for a new process, it has to put cer=
tain=20
things in certain places. the stack is located at the "top" of each proce=
ss=20
(i.e. the highest memory addresses.) the only "safe" place to put the arg=
s=20
and environment (to be sure the new process won't overwrite it by acciden=
t)=20
is above the top of the stack. (a stack starts at the top and works its w=
ay=20
down as it gets used.)

setup_arg_pages() reserves a block of memory at the very top of the new=20
process's memory space to hold the args and environment, and starts the s=
tack=20
just below this block. do_execve() needs to know how much memory is reser=
ved=20
so it can set the initial stack pointer correctly, and also because it ha=
s to=20
map the pages containing the stored args and environment into the top end=
 of=20
the new process's memory space (so the new process can read its args and=20
environment.)

to make sure i'm at least on the right track, i compiled this little prog=
ram=20
on both machines (i686 redhat 7.3, sparc64 gentoo, both running 2.4.18=20
kernels)

#include <stdio.h>
#include <stdlib.h>

int main ( int argc , char *argv[] , char *envp[] )
{
        int a , b ;
        void *c , *d ;

        c =3D malloc ( 1024 ) ;
        d =3D malloc ( 1048576 ) ;

        printf ( "argv=3D%08X\n" , argv ) ;
        printf ( "envp=3D%08X\n" , envp ) ;
        printf ( "&a=3D%08X\n" , &a ) ;
        printf ( "&b=3D%08X\n" , &b ) ;
        printf ( "c=3D%08X\n" , c ) ;
        printf ( "d=3D%08X\n" , d ) ;

        return 0 ;
}

the results:

(i686)
argv=3DBFFFF914
envp=3DBFFFF91C
&a=3DBFFFF8A4
&b=3DBFFFF8A0
c=3D080496E0
d=3D40014008

(sparc64)
argv=3DEFFFFCE4
envp=3DEFFFFCEC
&a=3DEFFFFC14
&b=3DEFFFFC10
c=3D00020660
d=3D70170008

this tells me that:

- - the i686 looks like it has a process limit of 3GB (highest address is=
=20
BFFFFFFF), and sparc64 has a 3.75GB limit (highest address EFFFFFFF)

- - the "c" and "d" addresses show that both libc's handle allocations of=
 small=20
and large blocks differently (this is a carry-on of something i discovere=
d at=20
the last installfest, on steve litt's beater box.)

- - the "a" and "b" addresses are on the stack. on both architectures the=
 stack=20
is LOWER THAN the argv/envp addresses.



SO THE SIMPLE ANSWER IS THIS: the limit isn't so much your PATH as your e=
ntire=20
environment block. for ix86 and sparc64 processors, the limit is 128K, an=
d it=20
looks like it's also 128K on all of the other architectures as well.

the only way around it looks like re-compiling your kernel, making sure t=
o=20
change ARG_MAX in "include/linux/limits.h" as well as MAX_ARG_PAGES where=
ver=20
it's defined (different for each processor type.) there's probably more t=
o it=20
than that, i'm not about to go and mess with it.

which brings up my question- what possible reason could you have for need=
ing=20
such a long PATH? did i miss something at the beginning of this thread?

- --=20
- ----------------------------------------
| John Simpson     Programmer at Large |
| <jms1@jms1.net> http://www.jms1.net/ |
- ----------------------------------------
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)

iD8DBQE9baQvEB9RczMG/PsRAngzAJ9EG5h8qJRBhd7Ysd9AlbWCTEo+YgCfbF6q
LYHsVd0VHG9sUgnm/NIQLhw=3D
=3DqJtt
-----END PGP SIGNATURE-----