How do I tokenize a string in C

Question

Java has a convenient split method   String str    The quick brown fox   String   results   str split         Is there an easy way to do this in C

User · Answer

If you re willing to use C  you can use the strtok function  You should pay attention to multi-threading issues when using it

User · Answer

Check this example  It might help you     include  lt iostream gt   include  lt sstream gt   using namespace std   int main          string tmps      istringstream is   the dellimiter is the space        while  is good               is  gt  gt  tmps          cout  lt  lt  tmps  lt  lt    n             return 0

User · Answer

I made a lexer tokenizer before with the use of only standard libraries  Here s the code    include  lt iostream gt   include  lt string gt   include  lt vector gt   include  lt sstream gt   using namespace std   string seps string amp  s        if   s size    return         stringstream ss      ss  lt  lt  s 0       for  int i   1  i  lt  s size    i              ss  lt  lt       lt  lt  s i             return ss str       void Tokenize string amp  str  vector lt string gt  amp  tokens  const string amp  delimiters              seps str           Skip delimiters at beginning      string  size type lastPos   str find first not of delimiters  0          Find first  non-delimiter       string  size type pos   str find first of delimiters  lastPos        while  string  npos    pos    string  npos    lastPos                   Found a token  add it to the vector          tokens push back str substr lastPos  pos - lastPos               Skip delimiters   Note the  not of          lastPos   str find first not of delimiters  pos              Find next  non-delimiter          pos   str find first of delimiters  lastPos            int main int argc  char  argv          vector lt string gt  t      string s    Tokens for everyone         Tokenize s  t             for  auto c   t          cout  lt  lt  c  lt  lt  endl       system  pause         return 0

User · Answer

For simple stuff I just use the following   unsigned TokenizeString const std  string amp  i source                          const std  string amp  i seperators                          bool i discard empty tokens                          std  vector lt std  string gt  amp  o tokens        unsigned prev pos   0      unsigned pos   0      unsigned number of tokens   0      o tokens clear        pos   i source find first of i seperators  pos       while  pos    std  string  npos                std  string token   i source substr prev pos  pos - prev pos           if   i discard empty tokens    token                              o tokens push back i source substr prev pos  pos - prev pos                number of tokens                       pos            prev pos   pos          pos   i source find first of i seperators  pos              if  prev pos  lt  i source length                  o tokens push back i source substr prev pos            number of tokens               return number of tokens      Cowardly disclaimer  I write real-time data processing software where the data comes in through binary files  sockets  or some API call  I O cards  camera s   I never use this function for something more complicated or time-critical than reading external configuration files on startup

User · Answer

Here s my Swiss   Army Knife of string-tokenizers for splitting up strings by whitespace  accounting for single and double-quote wrapped strings as well as stripping those characters from the results  I used RegexBuddy 4 x to generate most of the code-snippet  but I added custom handling for stripping quotes and a few other things    include  lt string gt   include  lt locale gt   include  lt regex gt   std  vector lt std  wstring gt  tokenize string std  wstring string to tokenize        std  vector lt std  wstring gt  tokens       std  wregex re LR                               std  regex constants  collate        std  wsregex iterator next  string to tokenize begin                                    string to tokenize end                                    re                                  std  regex constants  match not null         std  wsregex iterator end      const wchar t single quote   L          const wchar t double quote   L          while   next    end             std  wsmatch match    next          const std  wstring token   match str  0            next             if  token length    gt  2  amp  amp   token front      double quote    token front      single quote               tokens emplace back  std  wstring token begin   1  token begin   token length  -1             else             tokens emplace back token             return tokens

User · Answer

boost  tokenizer is your friend  but consider making your code portable with reference to internationalization  i18n  issues by using wstring wchar t instead of the legacy string char types    include  lt iostream gt   include  lt boost tokenizer hpp gt   include  lt string gt   using namespace std  using namespace boost   typedef tokenizer lt char separator lt wchar t gt                     wstring  const iterator  wstring gt  Tok   int main       wstring s    while  getline wcin  s         char separator lt wchar t gt  sep L         list of separator characters     Tok tok s  sep       for  Tok  iterator beg   tok begin    beg    tok end      beg          wcout  lt  lt   beg  lt  lt  L  t      output  or store in vector            wcout  lt  lt  L  n         return 0

User · Answer

If the maximum length of the input string to be tokenized is known  one can exploit this and implement a very fast version  I am sketching the basic idea below  which was inspired by both strtok   and the  suffix array -data structure described Jon Bentley s  Programming Perls  2nd edition  chapter 15  The C   class in this case only gives some organization and convenience of use  The implementation shown can be easily extended for removing leading and trailing whitespace characters in the tokens   Basically one can replace the separator characters with string-terminating   0 -characters and set pointers to the tokens withing the modified string  In the extreme case when the string consists only of separators  one gets string-length plus 1 resulting empty tokens  It is practical to duplicate the string to be modified   Header file   class TextLineSplitter   public       TextLineSplitter  const size t max line len          TextLineSplitter         void            SplitLine  const char  line                                 const char sep char                                             inline size t   NumTokens  void   const               return mNumTokens             const char      GetToken  const size t token idx   const               assert  token idx  lt  mNumTokens            return mTokens  token idx           private      const size t    mStorageSize       char            mBuff      char            mTokens      size t          mNumTokens       inline void     ResetContent  void                 memset  mBuff  0  mStorageSize               mark all items as empty          memset  mTokens  0  mStorageSize   sizeof  char                  reset counter for found items          mNumTokens   0L             Implementattion file   TextLineSplitter  TextLineSplitter  const size t max line len        mStorageSize   max line len   1L            allocate memory     mBuff     new char    mStorageSize        mTokens   new char    mStorageSize         ResetContent       TextLineSplitter   TextLineSplitter         delete    mBuff      delete    mTokens      void TextLineSplitter  SplitLine  const char  line                                    const char sep char                                                        assert  sep char      0          ResetContent        strncpy  mBuff  line  mMaxLineLen         size t idx         0L     running index for characters      do               assert  idx  lt  mStorageSize             const char chr   line  idx       retrieve current character          if  mTokens  mNumTokens      NULL                         mTokens  mNumTokens      amp mBuff  idx                 if          if  chr    sep char    chr      0                 item or line finished                overwrite separator with a 0-terminating character              mBuff  idx       0                  count-up items              mNumTokens                  if        while  line  idx            A scenario of usage would be      create an instance capable of splitting strings up to 1000 chars long  TextLineSplitter spl  1000    spl SplitLine   Item1  Item2 Item3     for  size t i   0  i  lt  spl NumTokens    i           printf    s n   spl GetToken  i          output   Item1  Item2 Item3

User · Answer

Adam Pierce s answer provides an hand-spun tokenizer taking in a const char   It s a bit more problematic to do with iterators because incrementing a string s end iterator is undefined  That said  given string str   The quick brown fox    we can certainly accomplish this   auto start   find cbegin str   cend str         vector lt string gt  tokens  string cbegin str   start      while  start    cend str         const auto finish   find   start  cend str              tokens push back string start  finish        start   finish      Live Example    If you re looking to abstract complexity by using standard functionality  as On Freund suggests strtok is a simple option    vector lt string gt  tokens   for  auto i   strtok data str         i    nullptr  i   strtok nullptr        tokens push back i     If you don t have access to C  17 you ll need to substitute data str  as in this example  http   ideone com 8kAGoa  Though not demonstrated in the example  strtok need not use the same delimiter for each token  Along with this advantage though  there are several drawbacks    strtok cannot be used on multiple strings at the same time  Either a nullptr must be passed to continue tokenizing the current string or a new char  to tokenize must be passed  there are some non-standard implementations which do support this however  such as  strtok s  For the same reason strtok cannot be used on multiple threads simultaneously  this may however be implementation defined  for example  Visual Studio s implementation is thread safe  Calling strtok modifies the string it is operating on  so it cannot be used on const strings  const char s  or literal strings  to tokenize any of these with strtok or to operate on a string who s contents need to be preserved  str would have to be copied  then the copy could be operated on     c  20 provides us with split view to tokenize strings  in a non-destructive manner  https   topanswers xyz cplusplus q 749 a874    The previous methods cannot generate a tokenized vector in-place  meaning without abstracting them into a helper function they cannot initialize const vector lt string gt  tokens  That functionality and the ability to accept any white-space delimiter can be harnessed using an istream iterator  For example given  const string str   The  quick  tbrown  nfox    we can do this   istringstream is  str    const vector lt string gt  tokens  istream iterator lt string gt  is   istream iterator lt string gt         Live Example  The required construction of an istringstream for this option has far greater cost than the previous 2 options  however this cost is typically hidden in the expense of string allocation     If none of the above options are flexable enough for your tokenization needs  the most flexible option is using a regex token iterator of course with this flexibility comes greater expense  but again this is likely hidden in the string allocation cost  Say for example we want to tokenize based on non-escaped commas  also eating white-space  given the following input  const string str   The  qu   ick   tbrown  fox    we can do this   const regex re     s                         s             const vector lt string gt  tokens  sregex token iterator cbegin str   cend str   re  1   sregex token iterator        Live Example

User · Answer

Here s a real simple one    include  lt vector gt   include  lt string gt  using namespace std   vector lt string gt  split const char  str  char c              vector lt string gt  result       do               const char  begin   str           while  str    c  amp  amp   str              str             result push back string begin  str          while  0     str          return result

User · Answer

Boost has a strong split function  boost  algorithm  split   Sample program    include  lt vector gt   include  lt boost algorithm string hpp gt   int main         auto s    a b  c   e f        std  vector lt std  string gt  fields      boost  split fields  s  boost  is any of            for  const auto amp  field   fields          std  cout  lt  lt        lt  lt  field  lt  lt      n       return 0      Output    a   b    c       e   f

User · Answer

I posted this answer for similar question  Don t reinvent the wheel   I ve used a number of libraries and the fastest and most flexible I have come across is   C   String Toolkit Library     Here is an example of how to use it that I ve posted else where on the stackoverflow    include  lt iostream gt   include  lt vector gt   include  lt string gt   include  lt strtk hpp gt   const char  whitespace       t r n f   const char  whitespace and punctuation       t r n f       int main                normal parsing of a string into a vector of strings        std  string s  Somewhere down the road           std  vector lt std  string gt  result         if  strtk  parse  s  whitespace  result                         for size t i   0  i  lt  result size      i               std  cout  lt  lt  result i   lt  lt  std  endl                            parsing a string into a vector of floats with other separators           besides spaces         std  string s  3 0  3 14  4 0           std  vector lt float gt  values         if  strtk  parse  s  whitespace and punctuation  values                         for size t i   0  i  lt  values size      i               std  cout  lt  lt  values i   lt  lt  std  endl                            parsing a string into specific variables         std  string s  angle   45  radius   9 9           std  string w1  w2         float v1  v2         if  strtk  parse  s  whitespace and punctuation  w1  v1  w2  v2                        std  cout  lt  lt   word    lt  lt  w1  lt  lt     value    lt  lt  v1  lt  lt  std  endl             std  cout  lt  lt   word    lt  lt  w2  lt  lt     value    lt  lt  v2  lt  lt  std  endl                      return 0

User · Answer

You can simply use a regular expression library and solve that using regular expressions   Use expression   w   and the variable in  1  or  1 depending on the library implementation of regular expressions

User · Answer

Many overly complicated suggestions here   Try this simple std  string solution   using namespace std   string someText        string  size type tokenOff   0  sepOff   tokenOff  while  sepOff    string  npos        sepOff   someText find      sepOff       string  size type tokenLen    sepOff    string  npos    sepOff   sepOff   - tokenOff      string token   someText substr tokenOff  tokenLen       if   token empty               do something with token         tokenOff   sepOff

User · Answer

I know you asked for a C   solution  but you might consider this helpful   Qt   include  lt QString gt        QString str    The quick brown fox    QStringList results   str split          The advantage over Boost in this example is that it s a direct one to one mapping to your post s code   See more at Qt documentation

User · Answer

I thought that was what the  gt  gt  operator on string streams was for   string word  sin  gt  gt  word

User · Answer

pystring is a small library which implements a bunch of Python s string functions  including the split method    include  lt string gt   include  lt vector gt   include  pystring h   std  vector lt std  string gt  chunks  pystring  split  this string   chunks       also can specify a separator pystring  split  this-string   chunks   -

User · Answer

split a string into multiple sub strings  based on a separator string     for example  if separator               s    abc  - gt   abc          s    abc  def xy  st   - gt   abc    def xy  and  st            s      abc    - gt   abc          s        - gt  NO sub strings found         s      - gt  NO sub strings found         then append the sub-strings to the end of the vector v           the idea comes from the findUrls   function of  Accelerated C     chapt7      findurls cpp     void split const string amp  s  const string amp  sep  vector lt string gt  amp  v        typedef string  const iterator iter      iter b   s begin    e   s end    i      iter sep b   sep begin    sep e   sep end            search through s     while  b    e           i   search b  e  sep b  sep e               no more separator found         if  i    e                  it s not an empty string             if  b    e                  v push back string b  e                break                    else if  i    b                  the separator is found and right at the beginning                in this case  we need to move on and search for the                next separator             b   i   sep length                      else                 found the separator             v push back string b  i                b   i                      The boost library is good  but they are not always available  Doing this sort of things by hand is also a good brain exercise  Here we just use the std  search   algorithm from the STL  see the above code

User · Answer

MFC ATL has a very nice tokenizer   From MSDN   CAtlString str    First Second Third     CAtlString resToken  int curPos  0   resToken  str Tokenize       curPos   while  resToken             printf  Resulting token   s n   resToken      resToken  str Tokenize       curPos       Output  Resulting Token  First Resulting Token  Second Resulting Token  Third

User · Answer

you can take advantage of boost  make find iterator  Something similar to this   template lt typename CH gt  inline vector lt  basic string lt CH gt   gt  tokenize      const basic string lt CH gt   amp Input      const basic string lt CH gt   amp Delimiter      bool remove empty token              typedef typename basic string lt CH gt   const iterator string iterator t      typedef boost  find iterator lt  string iterator t  gt  string find iterator t       vector lt  basic string lt CH gt   gt  Result      string iterator t it   Input begin        string iterator t it end   Input end        for string find iterator t i   boost  make find iterator Input  boost  first finder Delimiter  boost  is equal              i    string find iterator t              i            if remove empty token               if it    i- gt begin                    Result push back basic string lt CH gt  it i- gt begin                        else             Result push back basic string lt CH gt  it i- gt begin              it   i- gt end              if it    it end          Result push back basic string lt CH gt  it it end         return Result

User · Answer

Another quick way is to use getline  Something like   stringstream ss  bla bla    string s   while  getline ss  s           cout  lt  lt  s  lt  lt  endl      If you want  you can make a simple split   method returning a vector lt string gt   which is  really useful

User · Answer

No offense folks  but for such a simple problem  you are making things way too complicated  There are a lot of reasons to use Boost  But for something this simple  it s like hitting a fly with a 20  sledge   void split  vector lt string gt   amp  theStringVector      Altered returned value           const  string   amp  theString         const  string   amp  theDelimiter        UASSERT  theDelimiter size     gt   0      My own ASSERT macro       size t  start   0  end   0       while   end    string  npos                end   theString find  theDelimiter  start               If at end  use length maxLength   Else use length end-start          theStringVector push back  theString substr  start                          end    string  npos    string  npos   end - start                If at end  use start maxSize   Else use start end delimiter          start         end  gt   string  npos - theDelimiter size                           string  npos     end   theDelimiter size               For example  for Doug s case     define SHOW I X    cout  lt  lt       lt  lt   I   lt  lt     t     X          lt  lt   X   lt  lt        lt  lt  endl  int main         vector lt string gt  v       split  v   A PEP 909 Inventory Item               for  unsigned int i   0   i  lt  v size      i            SHOW  i  v i         And yes  we could have split   return a new vector rather than passing one in  It s trivial to wrap and overload  But depending on what I m doing  I often find it better to re-use pre-existing objects rather than always creating new ones   Just as long as I don t forget to empty the vector in between    Reference  http   www cplusplus com reference string string     I was originally writing a response to Doug s question  C   Strings Modifying and Extracting based on Separators  closed   But since Martin York closed that question with a pointer over here    I ll just generalize my code

User · Answer

I know this question is already answered but I want to contribute  Maybe my solution is a bit simple but this is what I came up with   vector lt string gt  get words string const amp  text  string const amp  separator        vector lt string gt  result      string tmp   text       size t first pos   0      size t second pos   tmp find separator        while  second pos    string  npos                if  first pos    second pos                        string word   tmp substr first pos  second pos - first pos               result push back word                     tmp   tmp substr second pos   separator length             second pos   tmp find separator              result push back tmp        return result      Please comment if there is a better approach to something in my code or if something is wrong   UPDATE  added generic separator

User · Answer

This a simple loop to tokenise with only standard library files   include  lt iostream h gt   include  lt stdio h gt   include  lt string h gt   include  lt math h gt   include  lt conio h gt  class word            public       char w 20        word                   for int j 0 j lt  20 j             w j    0                       void main           int i 1 n 0 j 0 k 0 m 1      char input 100       word ww 100       gets input        n strlen input         for i 0 i lt  m i                    if context i                               ww k  w j  context i               j                         else                    k             j 0           m

User · Answer

The Boost tokenizer class can make this sort of thing quite simple    include  lt iostream gt   include  lt string gt   include  lt boost foreach hpp gt   include  lt boost tokenizer hpp gt   using namespace std  using namespace boost   int main int  char          string text    token  test   string        char separator lt char gt  sep            tokenizer lt  char separator lt char gt   gt  tokens text  sep       BOOST FOREACH  const string amp  t  tokens            cout  lt  lt  t  lt  lt       lt  lt  endl            Updated for C  11    include  lt iostream gt   include  lt string gt   include  lt boost tokenizer hpp gt   using namespace std  using namespace boost   int main int  char          string text    token  test   string        char separator lt char gt  sep            tokenizer lt char separator lt char gt  gt  tokens text  sep       for  const auto amp  t   tokens            cout  lt  lt  t  lt  lt       lt  lt  endl

User · Answer

You can use streams  iterators  and the copy algorithm to do this fairly directly    include  lt string gt   include  lt vector gt   include  lt iostream gt   include  lt istream gt   include  lt ostream gt   include  lt iterator gt   include  lt sstream gt   include  lt algorithm gt   int main       std  string str    The quick brown fox         construct a stream from the string   std  stringstream strstr str         use stream iterators to copy the stream to the vector as whitespace separated strings   std  istream iterator lt std  string gt  it strstr     std  istream iterator lt std  string gt  end    std  vector lt std  string gt  results it  end         send the vector to stdout    std  ostream iterator lt std  string gt  oit std  cout     std  copy results begin    results end    oit

User · Answer

Simple C   code  standard C  98   accepts multiple delimiters  specified in a std  string   uses only vectors  strings and iterators    include  lt iostream gt   include  lt vector gt   include  lt string gt   include  lt stdexcept gt    std  vector lt std  string gt   split const std  string amp  str  const std  string amp  delim       std  vector lt std  string gt  result      if  str empty            throw std  runtime error  Can not tokenize an empty string         std  string  const iterator begin  str it      begin   str it   str begin         do           while  delim find  str it     std  string  npos  amp  amp  str it    str end                str it       find the position of the first delimiter in str         std  string token   std  string begin  str it      grab the token         if   token empty       empty token only when str starts with a delimiter             result push back token      push the token into a vector lt string gt          while  delim find  str it     std  string  npos  amp  amp  str it    str end                str it       ignore the additional consecutive delimiters         begin   str it     process the remaining tokens           while  str it    str end         return result     int main         std  string test string     this is a     simple  test   END       std  string delim              string containing the delimiters     std  vector lt std  string gt  tokens   split test string  delim                  for  std  vector lt std  string gt   const iterator it   tokens begin             it    tokens end    it                std  cout  lt  lt   it  lt  lt  std  endl

User · Answer

This is a simple STL-only solution   5 lines   using std  find and std  find first not of that handles repetitions of the delimiter  like spaces or periods for instance   as well leading and trailing delimiters    include  lt string gt   include  lt vector gt   void tokenize std  string str  std  vector lt string gt   amp token v       size t start   str find first not of DELIMITER   end start       while  start    std  string  npos              Find next occurence of delimiter         end   str find DELIMITER  start              Push back the token found into vector         token v push back str substr start  end-start               Skip all occurences of the delimiter to find new start         start   str find first not of DELIMITER  end             Try it out live

User · Answer

I ve been searching for a way to split a string by a separator of any length  so I started writing it from scratch  as existing solutions didn t suit me   Here is my little algorithm  using only STL     use like this   std  vector lt std  wstring gt  vec   Split lt std  wstring gt   L Hello  world      L        template  lt typename valueType gt  static std  vector  lt valueType gt  Split  valueType text  const valueType amp  delimiter        std  vector  lt valueType gt  tokens      size t pos   0      valueType token       while   pos   text find delimiter      valueType  npos                 token   text substr 0  pos           tokens push back  token           text erase 0  pos   delimiter length               tokens push back  text        return tokens      It can be used with separator of any length and form  as far as I ve tested  Instantiate with either string or wstring type   All the algorithm does is it searches for the delimiter  gets the part of the string that is up to the delimiter  deletes the delimiter and searches again until it finds it no more   Hope it helps

User · Answer

Here s an approach that allows you control over whether empty tokens are included  like strsep  or excluded  like strtok      include  lt string h gt     for strchr and strlen        want empty tokens  true    include empty tokens  like strsep      want empty tokens  false   exclude empty tokens  like strtok       std  vector lt std  string gt  tokenize const char  src                                    char delim                                    bool want empty tokens      std  vector lt std  string gt  tokens     if  src and  src      0      defensive     while  true            const char  d   strchr src  delim         size t len    d   d-src   strlen src          if  len or want empty tokens          tokens push back  std  string src  len        capture token        if  d  src    len 1  else break           return tokens

User · Answer

If you re using C   ranges - the full ranges-v3 library  not the limited functionality accepted into C  20 - you could do it this way  auto results   str   ranges  views  tokenize  quot   quot  1        and this is lazily-evaluated  i e  O 1  time and space  You can alternatively set a vector to this range  auto results   str   ranges  views  tokenize  quot   quot  1    to lt std  vector gt      this will take O m  space and O n  time if str has n characters making up m words  See also the library s own tokenization example  here

User · Answer

C   standard library algorithms are pretty universally based around iterators rather than concrete containers  Unfortunately this makes it hard to provide a Java-like split function in the C   standard library  even though nobody argues that this would be convenient  But what would its return type be  std  vector lt std  basic string lt     gt  gt   Maybe  but then we   re forced to perform  potentially redundant and costly  allocations   Instead  C   offers a plethora of ways to split strings based on arbitrarily complex delimiters  but none of them is encapsulated as nicely as in other languages  The numerous ways fill whole blog posts   At its simplest  you could iterate using std  string  find until you hit std  string  npos  and extract the contents using std  string  substr   A more fluid  and idiomatic  but basic  version for splitting on whitespace would use a std  istringstream   auto iss   std  istringstream  The quick brown fox    auto str   std  string     while  iss  gt  gt  str        process str       Using std  istream iterators  the contents of the string stream could also be copied into a vector using its iterator range constructor   Multiple libraries  such as Boost Tokenizer  offer specific tokenisers   More advanced splitting require regular expressions  C   provides the std  regex token iterator for this purpose in particular   auto const str    The quick brown fox s  auto const re   std  regex R   s      auto const vec   std  vector lt std  string gt       std  sregex token iterator begin str   end str   re  -1       std  sregex token iterator

User · Answer

Here is a sample tokenizer class that might do what you want     Header file class Tokenizer        public          static const std  string DELIMITERS          Tokenizer const std  string amp  str           Tokenizer const std  string amp  str  const std  string amp  delimiters           bool NextToken            bool NextToken const std  string amp  delimiters           const std  string GetToken   const          void Reset        protected          size t m offset          const std  string m string          std  string m token          std  string m delimiters        CPP file const std  string Tokenizer  DELIMITERS    t n r     Tokenizer  Tokenizer const std  string amp  s        m string s        m offset 0        m delimiters DELIMITERS      Tokenizer  Tokenizer const std  string amp  s  const std  string amp  delimiters        m string s        m offset 0        m delimiters delimiters      bool Tokenizer  NextToken          return NextToken m delimiters      bool Tokenizer  NextToken const std  string amp  delimiters         size t i   m string find first not of delimiters  m offset       if  std  string  npos    i                 m offset   m string length            return false             size t j   m string find first of delimiters  i       if  std  string  npos    j                 m token   m string substr i           m offset   m string length            return true             m token   m string substr i  j - i       m offset   j      return true      Example   std  vector  lt std  string gt  v  Tokenizer s  split this string         while  s NextToken          v push back s GetToken

User · Answer

Use strtok   In my opinion  there isn t a need to build a class around tokenizing unless strtok doesn t provide you with what you need   It might not  but in 15  years of writing various parsing code in C and C    I ve always used strtok   Here is an example  char myString      The quick brown fox   char  p   strtok myString        while  p        printf   Token   s n   p       p   strtok NULL            A few caveats  which might not suit your needs    The string is  destroyed  in the process  meaning that EOS characters are placed inline in the delimter spots   Correct usage might require you to make a non-const version of the string   You can also change the list of delimiters mid parse   In my own opinion  the above code is far simpler and easier to use than writing a separate class for it   To me  this is one of those functions that the language provides and it does it well and cleanly   It s simply a  C based  solution   It s appropriate  it s easy  and you don t have to write a lot of extra code  -

User · Answer

Seems odd to me that with all us speed conscious nerds here on SO no one has presented a version that uses a compile time generated look up table for the delimiter  example implementation further down   Using a look up table and iterators should beat std  regex in efficiency  if you don t need to beat regex  just use it  its standard as of C  11 and super flexible   Some have suggested regex already but for the noobs here is a packaged example that should do exactly what the OP expects   std  vector lt std  string gt  split std  string  const iterator it  std  string  const iterator end  std  regex e   std  regex    w          std  smatch m        std  vector lt std  string gt  ret        while  std  regex search  it end m e             ret emplace back m str                           std  advance it  m position     m length       next start position   match position   match length           return ret    std  vector lt std  string gt  split const std  string  amp s  std  regex e   std  regex    w         comfort version calls flexible version     return split s cbegin    s cend    std  move e      int main          std  string str   Some people  excluding those present  have been compile time constants - since puberty         auto v   split str       for const auto amp s v           std  cout  lt  lt  s  lt  lt  std  endl            std  cout  lt  lt   crazy version    lt  lt  std  endl      v   split str  std  regex    e          using e as delim shows flexibility     for const auto amp s v           std  cout  lt  lt  s  lt  lt  std  endl            return 0      If we need to be faster and accept the constraint that all chars must be 8 bits we can make a look up table at compile time using metaprogramming   template lt bool    gt  struct BoolSequence             just here to hold bools template lt char    gt  struct CharSequence             just here to hold chars template lt typename T  char C gt  struct Contains      generic template lt char First  char    Cs  char Match gt       not first specialization struct Contains lt CharSequence lt First  Cs    gt  Match gt        Contains lt CharSequence lt Cs    gt   Match gt           strip first and increase index template lt char First  char    Cs gt                   is first specialization struct Contains lt CharSequence lt First  Cs    gt  First gt   std  true type      template lt char Match gt                               not found specialization struct Contains lt CharSequence lt  gt  Match gt   std  false type     template lt int I  typename T  typename U gt   struct MakeSequence                               generic template lt int I  bool    Bs  typename U gt   struct MakeSequence lt I BoolSequence lt Bs    gt   U gt      not last     MakeSequence lt I-1  BoolSequence lt Contains lt U I-1 gt   value Bs    gt   U gt     template lt bool    Bs  typename U gt   struct MakeSequence lt 0 BoolSequence lt Bs    gt  U gt       last       using Type   BoolSequence lt Bs    gt      template lt typename T gt  struct BoolASCIITable  template lt bool    Bs gt  struct BoolASCIITable lt BoolSequence lt Bs    gt  gt          could be made constexpr but not yet supported by MSVC        static bool isDelim const char c           static const bool table 256     Bs              return table static cast lt int gt  c                using Delims   CharSequence lt                   n  gt      list your custom delimiters here using Table   BoolASCIITable lt typename MakeSequence lt 256 BoolSequence lt  gt  Delims gt   Type gt     With that in place making a getNextToken function is easy   template lt typename T It gt  std  pair lt T It T It gt  getNextToken T It begin T It end       begin   std  find if begin end std  not1 Table        find first non delim or end     auto second   std  find if begin end Table            find first delim or end     return std  make pair begin second       Using it is also easy   int main         std  string s  Some people  excluding those present  have been compile time constants - since puberty         auto it   std  begin s       auto end   std  end s       while it    std  end s            auto token   getNextToken it end           std  cout  lt  lt  std  string token first token second   lt  lt  std  endl          it   token second            return 0      Here is a live example  http   ideone com GKtkLQ

User · Answer

A solution using regex token iterators    include  lt iostream gt   include  lt regex gt   include  lt string gt   using namespace std   int main         string str  The quick brown fox         regex reg    s          sregex token iterator iter str begin    str end    reg  -1       sregex token iterator end       vector lt string gt  vec iter  end        for  auto a   vec                cout  lt  lt  a  lt  lt  endl

[c++] How do I tokenize a string in C++?

Examples related to c++

Examples related to string

Examples related to split

Examples related to tokenize