Tokenizing strings in C

Question

I have been trying to tokenize a string using SPACE as delimiter but it doesn t work  Does any one have suggestion on why it doesn t work   Edit  tokenizing using   strtok string          The code is like the following  pch   strtok  str       while  pch    NULL      printf    s n  pch     pch   strtok  NULL

User · Answer

Do it like this   char s 256   strcpy s   one two three    char  token   strtok s        while  token        printf  token   s n   token       token   strtok NULL            Note  strtok modifies the string its tokenising  so it cannot be a const char

User · Answer

When reading the strtok documentation  I see you need to pass in a NULL pointer after the first  initializing  call   Maybe you didn t do that   Just a guess of course

User · Answer

I ve made some string functions in order to split values  by using less pointers as I could because this code is intended to run on PIC18F processors  Those processors does not handle really good with pointers when you have few free RAM available    include  lt stdio h gt   include  lt string h gt   char POSTREQ 255     pwd 123456 amp apply Apply amp d1 88 amp d2 100 amp pwr 1 amp mpx Internal amp stmo Stereo amp proc Processor amp cmp Compressor amp ip1 192 amp ip2 168 amp ip3 10 amp ip4 131 amp gw1 192 amp gw2 168 amp gw3 10 amp gw4 192 amp pt 80 amp lic  amp A A    int findchar char  string  int Start  char C        while  string Start     0     Start    if string Start     C  return Start        return -1     int findcharn char  string  int Times  char C       int i   0  pos   0  fnd   0       while i  lt  Times           fnd   findchar string  pos  C           if fnd  lt  0  return -1          if fnd  gt  0  pos   fnd         i            return fnd     void mid char  in  char  out  int start  int end        int i   0      int size   end - start       for i   0  i  lt  size  i             out i    in start   i   1             out size    0     void getvalue char  out  int index        mid POSTREQ  out  findcharn POSTREQ  index         findcharn POSTREQ  index    amp    - 1       void main        char n pwd 7      char n d1 7       getvalue n d1  1       printf  Value   s n   n d1

User · Answer

Do it like this   char s 256   strcpy s   one two three    char  token   strtok s        while  token        printf  token   s n   token       token   strtok NULL            Note  strtok modifies the string its tokenising  so it cannot be a const char

User · Answer

Here s an example of strtok usage  keep in mind that strtok is destructive of its input string  and therefore can t ever be used on a string constant  char  p   strtok str        while p    NULL        printf   s n   p       p   strtok NULL            Basically the thing to note is that passing a NULL as the first parameter to strtok tells it to get the next token from the string it was previously tokenizing

User · Answer

You can simplify the code by introducing an extra variable    include  lt string h gt   include  lt stdio h gt   int main         char str 100    s   str   t   NULL       strcpy str   a space delimited string        while   t   strtok s           NULL            s   NULL          printf    s  n   t             return 0

User · Answer

You can simplify the code by introducing an extra variable    include  lt string h gt   include  lt stdio h gt   int main         char str 100    s   str   t   NULL       strcpy str   a space delimited string        while   t   strtok s           NULL            s   NULL          printf    s  n   t             return 0

User · Answer

strtok can be very dangerous   It is not thread safe   Its intended use is to be called over and over in a loop  passing in the output from the previous call   The strtok function has an internal variable that stores the state of the strtok call   This state is not unique to each thread - it is global   If any other code uses strtok in another thread  you get problems   Not the kind of problems you want to track down either   I d recommend looking for a regex implementation  or using sscanf to pull apart the string   Try this   char strprint 256   char text 256   strcpy text   My string to test    while   sscanf  text    s  s   strprint  text   gt  0        printf  token   s n   strprint       Note  The  text  string is destroyed as it s separated  This may not be the preferred behaviour

User · Answer

strtok can be very dangerous   It is not thread safe   Its intended use is to be called over and over in a loop  passing in the output from the previous call   The strtok function has an internal variable that stores the state of the strtok call   This state is not unique to each thread - it is global   If any other code uses strtok in another thread  you get problems   Not the kind of problems you want to track down either   I d recommend looking for a regex implementation  or using sscanf to pull apart the string   Try this   char strprint 256   char text 256   strcpy text   My string to test    while   sscanf  text    s  s   strprint  text   gt  0        printf  token   s n   strprint       Note  The  text  string is destroyed as it s separated  This may not be the preferred behaviour

User · Answer

Here s an example of strtok usage  keep in mind that strtok is destructive of its input string  and therefore can t ever be used on a string constant  char  p   strtok str        while p    NULL        printf   s n   p       p   strtok NULL            Basically the thing to note is that passing a NULL as the first parameter to strtok tells it to get the next token from the string it was previously tokenizing

User · Answer

You can simplify the code by introducing an extra variable    include  lt string h gt   include  lt stdio h gt   int main         char str 100    s   str   t   NULL       strcpy str   a space delimited string        while   t   strtok s           NULL            s   NULL          printf    s  n   t             return 0

User · Answer

Here is another strtok   implementation  which has the ability to recognize consecutive delimiters  standard library s strtok   does not have this   The function is a part of BSD licensed string library  called zString  You are more than welcome to contribute     https   github com fnoyanisi zString  char  zstring strtok char  str  const char  delim        static char  static str 0          var to store last address        int index 0  strlength 0           integers for indexes        int found   0                      check if delim is found            delimiter cannot be NULL       if no more char left  return NULL as well            if  delim  0     str    0  amp  amp  static str    0           return 0       if  str    0          str   static str          get length of string        while str strlength           strlength            find the first occurance of delim        for  index 0 index lt strlength index            if  str index   delim 0                 found 1              break                    if delim is not contained in str  return str        if   found            static str   0          return str                check for consecutive delimiters      if first char is delim  return delim            if  str 0   delim 0             static str    str   1           return  char   delim                terminate the string       this assignmetn requires char    so str has to       be char   rather than  char            str index      0           save the rest of the string        if   str   index   1   0          static str    str   index   1       else         static str   0           return str      As mentioned in previous posts  since strtok    or the one I implmented above  relies on a static  char variable to preserve the location of last delimiter between consecutive calls  extra care should be taken while dealing with multi-threaded aplications

User · Answer

Here s an example of strtok usage  keep in mind that strtok is destructive of its input string  and therefore can t ever be used on a string constant  char  p   strtok str        while p    NULL        printf   s n   p       p   strtok NULL            Basically the thing to note is that passing a NULL as the first parameter to strtok tells it to get the next token from the string it was previously tokenizing

User · Answer

When reading the strtok documentation  I see you need to pass in a NULL pointer after the first  initializing  call   Maybe you didn t do that   Just a guess of course

User · Answer

strtok can be very dangerous   It is not thread safe   Its intended use is to be called over and over in a loop  passing in the output from the previous call   The strtok function has an internal variable that stores the state of the strtok call   This state is not unique to each thread - it is global   If any other code uses strtok in another thread  you get problems   Not the kind of problems you want to track down either   I d recommend looking for a regex implementation  or using sscanf to pull apart the string   Try this   char strprint 256   char text 256   strcpy text   My string to test    while   sscanf  text    s  s   strprint  text   gt  0        printf  token   s n   strprint       Note  The  text  string is destroyed as it s separated  This may not be the preferred behaviour

User · Answer

When reading the strtok documentation  I see you need to pass in a NULL pointer after the first  initializing  call   Maybe you didn t do that   Just a guess of course

User · Answer

Do it like this   char s 256   strcpy s   one two three    char  token   strtok s        while  token        printf  token   s n   token       token   strtok NULL            Note  strtok modifies the string its tokenising  so it cannot be a const char

User · Answer

Here s an example of strtok usage  keep in mind that strtok is destructive of its input string  and therefore can t ever be used on a string constant  char  p   strtok str        while p    NULL        printf   s n   p       p   strtok NULL            Basically the thing to note is that passing a NULL as the first parameter to strtok tells it to get the next token from the string it was previously tokenizing

User · Answer

Here is another strtok   implementation  which has the ability to recognize consecutive delimiters  standard library s strtok   does not have this   The function is a part of BSD licensed string library  called zString  You are more than welcome to contribute     https   github com fnoyanisi zString  char  zstring strtok char  str  const char  delim        static char  static str 0          var to store last address        int index 0  strlength 0           integers for indexes        int found   0                      check if delim is found            delimiter cannot be NULL       if no more char left  return NULL as well            if  delim  0     str    0  amp  amp  static str    0           return 0       if  str    0          str   static str          get length of string        while str strlength           strlength            find the first occurance of delim        for  index 0 index lt strlength index            if  str index   delim 0                 found 1              break                    if delim is not contained in str  return str        if   found            static str   0          return str                check for consecutive delimiters      if first char is delim  return delim            if  str 0   delim 0             static str    str   1           return  char   delim                terminate the string       this assignmetn requires char    so str has to       be char   rather than  char            str index      0           save the rest of the string        if   str   index   1   0          static str    str   index   1       else         static str   0           return str      As mentioned in previous posts  since strtok    or the one I implmented above  relies on a static  char variable to preserve the location of last delimiter between consecutive calls  extra care should be taken while dealing with multi-threaded aplications

User · Answer

int not in delimiter char c  char  delim        while  delim      0                if c     delim  return 0              delim              return 1     char  token separater char  source  char  delimiter  char   last    char  begin   next token  char  sbegin     Get the start of the token    if source    begin   source  else   begin    last   sbegin   begin     Scan through the string till we find character in delimiter     while  begin      0   amp  amp  not in delimiter  begin  delimiter           begin          Check if we have reached at of the string    if  begin      0        We dont need to come further  hence return NULL       last   NULL      return sbegin       Scan the string till we find a character which is not in delimiter     next token    begin   while next token      0   amp  amp   not in delimiter  next token  delimiter            next token           If we have not reached at the end of the string     if  next token      0       last   next token--     next token     0      return sbegin        void main         char string 10     abcb dccc       char delim 10             char  token   NULL      char  last            token    token separater string  delim   amp last       printf   s n   token       while last               token    token separater NULL  delim   amp last               printf   s n   token              You can read detail analysis at blog mentioned in my profile

User · Answer

You can simplify the code by introducing an extra variable    include  lt string h gt   include  lt stdio h gt   int main         char str 100    s   str   t   NULL       strcpy str   a space delimited string        while   t   strtok s           NULL            s   NULL          printf    s  n   t             return 0

User · Answer

int not in delimiter char c  char  delim        while  delim      0                if c     delim  return 0              delim              return 1     char  token separater char  source  char  delimiter  char   last    char  begin   next token  char  sbegin     Get the start of the token    if source    begin   source  else   begin    last   sbegin   begin     Scan through the string till we find character in delimiter     while  begin      0   amp  amp  not in delimiter  begin  delimiter           begin          Check if we have reached at of the string    if  begin      0        We dont need to come further  hence return NULL       last   NULL      return sbegin       Scan the string till we find a character which is not in delimiter     next token    begin   while next token      0   amp  amp   not in delimiter  next token  delimiter            next token           If we have not reached at the end of the string     if  next token      0       last   next token--     next token     0      return sbegin        void main         char string 10     abcb dccc       char delim 10             char  token   NULL      char  last            token    token separater string  delim   amp last       printf   s n   token       while last               token    token separater NULL  delim   amp last               printf   s n   token              You can read detail analysis at blog mentioned in my profile

User · Answer

When reading the strtok documentation  I see you need to pass in a NULL pointer after the first  initializing  call   Maybe you didn t do that   Just a guess of course

User · Answer

Do it like this   char s 256   strcpy s   one two three    char  token   strtok s        while  token        printf  token   s n   token       token   strtok NULL            Note  strtok modifies the string its tokenising  so it cannot be a const char

User · Answer

I ve made some string functions in order to split values  by using less pointers as I could because this code is intended to run on PIC18F processors  Those processors does not handle really good with pointers when you have few free RAM available    include  lt stdio h gt   include  lt string h gt   char POSTREQ 255     pwd 123456 amp apply Apply amp d1 88 amp d2 100 amp pwr 1 amp mpx Internal amp stmo Stereo amp proc Processor amp cmp Compressor amp ip1 192 amp ip2 168 amp ip3 10 amp ip4 131 amp gw1 192 amp gw2 168 amp gw3 10 amp gw4 192 amp pt 80 amp lic  amp A A    int findchar char  string  int Start  char C        while  string Start     0     Start    if string Start     C  return Start        return -1     int findcharn char  string  int Times  char C       int i   0  pos   0  fnd   0       while i  lt  Times           fnd   findchar string  pos  C           if fnd  lt  0  return -1          if fnd  gt  0  pos   fnd         i            return fnd     void mid char  in  char  out  int start  int end        int i   0      int size   end - start       for i   0  i  lt  size  i             out i    in start   i   1             out size    0     void getvalue char  out  int index        mid POSTREQ  out  findcharn POSTREQ  index         findcharn POSTREQ  index    amp    - 1       void main        char n pwd 7      char n d1 7       getvalue n d1  1       printf  Value   s n   n d1

User · Answer

strtok can be very dangerous   It is not thread safe   Its intended use is to be called over and over in a loop  passing in the output from the previous call   The strtok function has an internal variable that stores the state of the strtok call   This state is not unique to each thread - it is global   If any other code uses strtok in another thread  you get problems   Not the kind of problems you want to track down either   I d recommend looking for a regex implementation  or using sscanf to pull apart the string   Try this   char strprint 256   char text 256   strcpy text   My string to test    while   sscanf  text    s  s   strprint  text   gt  0        printf  token   s n   strprint       Note  The  text  string is destroyed as it s separated  This may not be the preferred behaviour

[c] Tokenizing strings in C

Examples related to c

Examples related to string

Examples related to tokenize