Add params to given URL in Python

Question

Suppose I was given a URL  It might already have GET parameters  e g  http   example com search q question  or it might not  e g  http   example com     And now I need to add some parameters to it like   lang   en   tag   python    In the first case I m going to have http   example com search q question amp lang en amp tag python and in the second     http   example com search lang en amp tag python   Is there any standard way to do this

User · Answer

Yes  use urllib   From the examples in the documentation    gt  gt  gt  import urllib  gt  gt  gt  params   urllib urlencode   spam   1   eggs   2   bacon   0    gt  gt  gt  f   urllib urlopen  http   www musi-cal com cgi-bin query  s    params   gt  gt  gt  print f geturl     Prints the final URL with parameters   gt  gt  gt  print f read     Prints the contents

User · Answer

You want to use URL encoding if the strings can have arbitrary data  for example  characters such as ampersands  slashes  etc  will need to be encoded    Check out urllib urlencode    gt  gt  gt  import urllib  gt  gt  gt  urllib urlencode   lang   en   tag   python     lang en amp tag python    In python3   from urllib import parse parse urlencode   lang   en   tag   python

User · Answer

You can also use the furl module https   github com gruns furl   gt  gt  gt  from furl import furl  gt  gt  gt  print furl  http   example com search q question   add   lang   en   tag   python    url http   example com search q question amp lang en amp tag python

User · Answer

Outsource it to the battle tested requests library   This is how I will do it   from requests models import PreparedRequest url    http   example com search q question  params     lang   en   tag   python   req   PreparedRequest   req prepare url url  params  print req url

User · Answer

Use the various urlparse functions to tear apart the existing URL  urllib urlencode   on the combined dictionary  then urlparse urlunparse   to put it all back together again   Or just take the result of urllib urlencode   and concatenate it to the URL appropriately

User · Answer

Based on this answer  one-liner for simple cases  Python 3 code    from urllib parse import urlparse  urlencode   url    https   stackoverflow com search q question  params     lang   en   tag   python    url       amp   if urlparse url  query else        urlencode params    or   url       amp         urlparse url  query          urlencode params

User · Answer

In python 2 5  import cgi import urllib import urlparse  def add url param url    params       n 3     parts   list urlparse urlsplit url       d   dict cgi parse qsl parts n      use cgi parse qs for list values     d update params      parts n  urllib urlencode d      return urlparse urlunsplit parts   url    http   stackoverflow com search q question  add url param url  lang  en       http   stackoverflow com search q question amp lang en

User · Answer

Here is how I implemented it   import urllib  params   urllib urlencode   lang   en   tag   python    url      if request GET     url   request url     amp     params else     url   request url         params       Worked like a charm  However  I would have liked a more cleaner way to implement this   Another way of implementing the above is put it in a method   import urllib  def add url param request    params      new url          params   dict   params      params   urllib urlencode  params      if  params        if request GET           new url   request url     amp      params       else           new url   request url          params    else        new url   request url     return new ur

User · Answer

There are a couple of quirks with the urllib and urlparse modules  Here s a working example   try      import urlparse     from urllib import urlencode except    For Python 3     import urllib parse as urlparse     from urllib parse import urlencode  url    http   stackoverflow com search q question  params     lang   en   tag   python    url parts   list urlparse urlparse url   query   dict urlparse parse qsl url parts 4    query update params   url parts 4    urlencode query   print urlparse urlunparse url parts     ParseResult  the result of urlparse    is read-only and we need to convert it to a list before we can attempt to modify its data

User · Answer

Why  I ve been not satisfied with all the solutions on this page  come on  where is our favorite copy-paste thing   so I wrote my own based on answers here  It tries to be complete and more Pythonic  I ve added a handler for dict and bool values in arguments to be more consumer-side  JS  friendly  but they are yet optional  you can drop them   How it works  Test 1  Adding new arguments  handling Arrays and Bool values   url    http   stackoverflow com test  new params     answers   False   data     some   values     add url params url  new params            http   stackoverflow com test data some amp data values amp answers false    Test 2  Rewriting existing args  handling DICT values   url    http   stackoverflow com test  question false  new params     question       X       Y       add url params url  new params            http   stackoverflow com test  question  7B 22  X   22 3A  22  Y   22 7D    Talk is cheap  Show me the code   Code itself  I ve tried to describe it in details   from json import dumps  try      from urllib import urlencode  unquote     from urlparse import urlparse  parse qsl  ParseResult except ImportError        Python 3 fallback     from urllib parse import           urlencode  unquote  urlparse  parse qsl  ParseResult         def add url params url  params           Add GET params to provided URL being aware of existing        param url  string of target URL      param params  dict containing requested params to be added      return  string with updated URL       gt  gt  url    http   stackoverflow com test answers true       gt  gt  new params     answers   False   data     some   values         gt  gt  add url params url  new params       http   stackoverflow com test data some amp data values amp answers false                Unquoting URL first so we don t loose existing args     url   unquote url        Extracting url info     parsed url   urlparse url        Extracting URL arguments from parsed URL     get args   parsed url query       Converting URL arguments to dict     parsed get args   dict parse qsl get args         Merging URL arguments dict with new params     parsed get args update params         Bool and Dict values should be converted to json-friendly values       you may throw this part away if you don t like it        parsed get args update           k  dumps v  for k  v in parsed get args items            if isinstance v   bool  dict                 Converting URL argument to proper query string     encoded get args   urlencode parsed get args  doseq True        Creating new parsed result object based on provided with new       URL arguments  Same thing happens inside of urlparse      new url   ParseResult          parsed url scheme  parsed url netloc  parsed url path          parsed url params  encoded get args  parsed url fragment       geturl        return new url   Please be aware that there may be some issues  if you ll find one please let me know and we will make this thing better

User · Answer

If you are using the requests lib   import requests     params     tag    python   requests get url  params params

User · Answer

I liked Lukasz version  but since urllib and urllparse functions are somewhat awkward to use in this case  I think it s more straightforward to do something like this   params   urllib urlencode params   if urlparse urlparse url  4       print url     amp     params else      print url         params

User · Answer

python3  self explanatory I guess from urllib parse import urlparse  urlencode  parse qsl  url    https   www linkedin com jobs search keywords engineer   parsed   urlparse url  current params   dict parse qsl parsed query   new params     location    United States   merged params   urlencode    current params    new params   parsed   parsed  replace query merged params   print parsed geturl      https   www linkedin com jobs search keywords engineer amp location United States

User · Answer

Yet another answer   def addGetParameters url  newParams        scheme  netloc  path  params  query  fragment    urlparse urlparse url      queryList   urlparse parse qsl query  keep blank values True      for key in newParams          queryList append  key  newParams key        return urlparse urlunparse  scheme  netloc  path  params  urllib urlencode queryList   fragment

User · Answer

I find this more elegant than the two top answers   from urllib parse import urlencode  urlparse  parse qs  def merge url query params url  str  additional params  dict  - gt  str      url components   urlparse url      original params   parse qs url components query        Before Python 3 5 you could update original params with        additional params  but here all the variables are immutable      merged params      original params    additional params      updated query   urlencode merged params  doseq True         replace   is how you can create a new NamedTuple with a changed field     return url components  replace query updated query  geturl    assert merge url query params       http   example com search q question         lang   en   tag   python          http   example com search q question amp lang en amp tag python    The most important things I dislike in the top answers  they are nevertheless good     Lukasz  having to remember the index at which the query is in the URL components Sapphire64  the very verbose way of creating the updated ParseResult   What s bad about my response is the magically looking dict merge using unpacking  but I prefer that to updating an already existing dictionary because of my prejudice against mutability

[python] Add params to given URL in Python

Examples related to python

Examples related to url