[ACCEPTED]-How to parse strings to look like sys.argv-argv

Accepted answer
Score: 95

I believe you want the shlex module.

>>> import shlex
>>> shlex.split('-o 1 --long "Some long string"')
['-o', '1', '--long', 'Some long string']

0

Score: 3

Before I was aware of shlex.split, I made the following:

import sys

_WORD_DIVIDERS = set((' ', '\t', '\r', '\n'))

_QUOTE_CHARS_DICT = {
    '\\':   '\\',
    ' ':    ' ',
    '"':    '"',
    'r':    '\r',
    'n':    '\n',
    't':    '\t',
}

def _raise_type_error():
    raise TypeError("Bytes must be decoded to Unicode first")

def parse_to_argv_gen(instring):
    is_in_quotes = False
    instring_iter = iter(instring)
    join_string = instring[0:0]

    c_list = []
    c = ' '
    while True:
        # Skip whitespace
        try:
            while True:
                if not isinstance(c, str) and sys.version_info[0] >= 3:
                    _raise_type_error()
                if c not in _WORD_DIVIDERS:
                    break
                c = next(instring_iter)
        except StopIteration:
            break
        # Read word
        try:
            while True:
                if not isinstance(c, str) and sys.version_info[0] >= 3:
                    _raise_type_error()
                if not is_in_quotes and c in _WORD_DIVIDERS:
                    break
                if c == '"':
                    is_in_quotes = not is_in_quotes
                    c = None
                elif c == '\\':
                    c = next(instring_iter)
                    c = _QUOTE_CHARS_DICT.get(c)
                if c is not None:
                    c_list.append(c)
                c = next(instring_iter)
            yield join_string.join(c_list)
            c_list = []
        except StopIteration:
            yield join_string.join(c_list)
            break

def parse_to_argv(instring):
    return list(parse_to_argv_gen(instring))

This 13 works with Python 2.x and 3.x. On Python 12 2.x, it works directly with byte strings 11 and Unicode strings. On Python 3.x, it only accepts 10 [Unicode] strings, not bytes objects.

This doesn't 9 behave exactly the same as shell argv splitting—it 8 also allows quoting of CR, LF and TAB characters 7 as \r, \n and \t, converting them to real CR, LF, TAB 6 (shlex.split doesn't do that). So writing my own function 5 was useful for my needs. I guess shlex.split is better 4 if you just want plain shell-style argv 3 splitting. I'm sharing this code in case 2 it's useful as a baseline for doing something 1 slightly different.

More Related questions