[python] List of tables, db schema, dump etc using the Python sqlite3 API

For some reason I can't find a way to get the equivalents of sqlite's interactive shell commands:

.tables
.dump

using the Python sqlite3 API.

Is there anything like that?

#!/usr/bin/env python
# -*- coding: utf-8 -*-

if __name__ == "__main__":

   import sqlite3

   dbname = './db/database.db'
   try:
      print "INITILIZATION..."
      con = sqlite3.connect(dbname)
      cursor = con.cursor()
      cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
      tables = cursor.fetchall()
      for tbl in tables:
         print "\n########  "+tbl[0]+"  ########"
         cursor.execute("SELECT * FROM "+tbl[0]+";")
         rows = cursor.fetchall()
         for row in rows:
            print row
      print(cursor.fetchall())
   except KeyboardInterrupt:
      print "\nClean Exit By user"
   finally:
      print "\nFinally"

If someone wants to do the same thing with Pandas

import pandas as pd
import sqlite3
conn = sqlite3.connect("db.sqlite3")
table = pd.read_sql_query("SELECT name FROM sqlite_master WHERE type='table'", conn)
print(table)

The FASTEST way of doing this in python is using Pandas (version 0.16 and up).

Dump one table:

db = sqlite3.connect('database.db')
table = pd.read_sql_query("SELECT * from table_name", db)
table.to_csv(table_name + '.csv', index_label='index')

Dump all tables:

import sqlite3
import pandas as pd


def to_csv():
    db = sqlite3.connect('database.db')
    cursor = db.cursor()
    cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
    tables = cursor.fetchall()
    for table_name in tables:
        table_name = table_name[0]
        table = pd.read_sql_query("SELECT * from %s" % table_name, db)
        table.to_csv(table_name + '.csv', index_label='index')
    cursor.close()
    db.close()

I'm not familiar with the Python API but you can always use

SELECT * FROM sqlite_master;

The FASTEST way of doing this in python is using Pandas (version 0.16 and up).

Dump one table:

db = sqlite3.connect('database.db')
table = pd.read_sql_query("SELECT * from table_name", db)
table.to_csv(table_name + '.csv', index_label='index')

Dump all tables:

import sqlite3
import pandas as pd


def to_csv():
    db = sqlite3.connect('database.db')
    cursor = db.cursor()
    cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
    tables = cursor.fetchall()
    for table_name in tables:
        table_name = table_name[0]
        table = pd.read_sql_query("SELECT * from %s" % table_name, db)
        table.to_csv(table_name + '.csv', index_label='index')
    cursor.close()
    db.close()

In Python:

con = sqlite3.connect('database.db')
cursor = con.cursor()
cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
print(cursor.fetchall())

Watch out for my other answer. There is a much faster way using pandas.

I've implemented a sqlite table schema parser in PHP, you may check here: https://github.com/c9s/LazyRecord/blob/master/src/LazyRecord/TableParser/SqliteTableDefinitionParser.php

You can use this definition parser to parse the definitions like the code below:

$parser = new SqliteTableDefinitionParser;
$parser->parseColumnDefinitions('x INTEGER PRIMARY KEY, y DOUBLE, z DATETIME default \'2011-11-10\', name VARCHAR(100)');

Apparently the version of sqlite3 included in Python 2.6 has this ability: http://docs.python.org/dev/library/sqlite3.html

# Convert file existing_db.db to SQL dump file dump.sql
import sqlite3, os

con = sqlite3.connect('existing_db.db')
with open('dump.sql', 'w') as f:
    for line in con.iterdump():
        f.write('%s\n' % line)

Here's a short and simple python program to print out the table names and the column names for those tables (python 2. python 3 follows).

import sqlite3

db_filename = 'database.sqlite'
newline_indent = '\n   '

db=sqlite3.connect(db_filename)
db.text_factory = str
cur = db.cursor()

result = cur.execute("SELECT name FROM sqlite_master WHERE type='table';").fetchall()
table_names = sorted(zip(*result)[0])
print "\ntables are:"+newline_indent+newline_indent.join(table_names)

for table_name in table_names:
    result = cur.execute("PRAGMA table_info('%s')" % table_name).fetchall()
    column_names = zip(*result)[1]
    print ("\ncolumn names for %s:" % table_name)+newline_indent+(newline_indent.join(column_names))

db.close()
print "\nexiting."

(EDIT: I have been getting periodic vote-ups on this, so here is the python3 version for people who are finding this answer)

import sqlite3

db_filename = 'database.sqlite'
newline_indent = '\n   '

db=sqlite3.connect(db_filename)
db.text_factory = str
cur = db.cursor()

result = cur.execute("SELECT name FROM sqlite_master WHERE type='table';").fetchall()
table_names = sorted(list(zip(*result))[0])
print ("\ntables are:"+newline_indent+newline_indent.join(table_names))

for table_name in table_names:
    result = cur.execute("PRAGMA table_info('%s')" % table_name).fetchall()
    column_names = list(zip(*result))[1]
    print (("\ncolumn names for %s:" % table_name)
           +newline_indent
           +(newline_indent.join(column_names)))

db.close()
print ("\nexiting.")

After a lot of fiddling I found a better answer at sqlite docs for listing the metadata for the table, even attached databases.

meta = cursor.execute("PRAGMA table_info('Job')")
for r in meta:
    print r

The key information is to prefix table_info, not my_table with the attachment handle name.

In Python:

con = sqlite3.connect('database.db')
cursor = con.cursor()
cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
print(cursor.fetchall())

Watch out for my other answer. There is a much faster way using pandas.

Some might find my function useful if you just want to print out all of the tables and columns in your db.

In the loop, I query each TABLE with LIMIT 0 so it just returns the header info without all the data. You make an empty df out of it, and use the iterable df.columns to print each column name out.

conn = sqlite3.connect('example.db')
c = conn.cursor()

def table_info(c, conn):
    '''
    prints out all of the columns of every table in db
    c : cursor object
    conn : database connection object
    '''
    tables = c.execute("SELECT name FROM sqlite_master WHERE type='table';").fetchall()
    for table_name in tables:
        table_name = table_name[0] # tables is a list of single item tuples
        table = pd.read_sql_query("SELECT * from {} LIMIT 0".format(table_name), conn)
        print(table_name)
        for col in table.columns:
            print('\t-' + col)
        print()

table_info(c, conn)

Results will be:

table1
    -column1
    -column2

table2
    -column1
    -column2
    -column3 

etc.

Check out here for dump. It seems there is a dump function in the library sqlite3.

I'm not familiar with the Python API but you can always use

SELECT * FROM sqlite_master;

#!/usr/bin/env python
# -*- coding: utf-8 -*-

if __name__ == "__main__":

   import sqlite3

   dbname = './db/database.db'
   try:
      print "INITILIZATION..."
      con = sqlite3.connect(dbname)
      cursor = con.cursor()
      cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
      tables = cursor.fetchall()
      for tbl in tables:
         print "\n########  "+tbl[0]+"  ########"
         cursor.execute("SELECT * FROM "+tbl[0]+";")
         rows = cursor.fetchall()
         for row in rows:
            print row
      print(cursor.fetchall())
   except KeyboardInterrupt:
      print "\nClean Exit By user"
   finally:
      print "\nFinally"

I'm not familiar with the Python API but you can always use

SELECT * FROM sqlite_master;

After a lot of fiddling I found a better answer at sqlite docs for listing the metadata for the table, even attached databases.

meta = cursor.execute("PRAGMA table_info('Job')")
for r in meta:
    print r

The key information is to prefix table_info, not my_table with the attachment handle name.

Here's a short and simple python program to print out the table names and the column names for those tables (python 2. python 3 follows).

import sqlite3

db_filename = 'database.sqlite'
newline_indent = '\n   '

db=sqlite3.connect(db_filename)
db.text_factory = str
cur = db.cursor()

result = cur.execute("SELECT name FROM sqlite_master WHERE type='table';").fetchall()
table_names = sorted(zip(*result)[0])
print "\ntables are:"+newline_indent+newline_indent.join(table_names)

for table_name in table_names:
    result = cur.execute("PRAGMA table_info('%s')" % table_name).fetchall()
    column_names = zip(*result)[1]
    print ("\ncolumn names for %s:" % table_name)+newline_indent+(newline_indent.join(column_names))

db.close()
print "\nexiting."

(EDIT: I have been getting periodic vote-ups on this, so here is the python3 version for people who are finding this answer)

import sqlite3

db_filename = 'database.sqlite'
newline_indent = '\n   '

db=sqlite3.connect(db_filename)
db.text_factory = str
cur = db.cursor()

result = cur.execute("SELECT name FROM sqlite_master WHERE type='table';").fetchall()
table_names = sorted(list(zip(*result))[0])
print ("\ntables are:"+newline_indent+newline_indent.join(table_names))

for table_name in table_names:
    result = cur.execute("PRAGMA table_info('%s')" % table_name).fetchall()
    column_names = list(zip(*result))[1]
    print (("\ncolumn names for %s:" % table_name)
           +newline_indent
           +(newline_indent.join(column_names)))

db.close()
print ("\nexiting.")

If someone wants to do the same thing with Pandas

import pandas as pd
import sqlite3
conn = sqlite3.connect("db.sqlite3")
table = pd.read_sql_query("SELECT name FROM sqlite_master WHERE type='table'", conn)
print(table)

Check out here for dump. It seems there is a dump function in the library sqlite3.

I'm not familiar with the Python API but you can always use

SELECT * FROM sqlite_master;

Some might find my function useful if you just want to print out all of the tables and columns in your db.

In the loop, I query each TABLE with LIMIT 0 so it just returns the header info without all the data. You make an empty df out of it, and use the iterable df.columns to print each column name out.

conn = sqlite3.connect('example.db')
c = conn.cursor()

def table_info(c, conn):
    '''
    prints out all of the columns of every table in db
    c : cursor object
    conn : database connection object
    '''
    tables = c.execute("SELECT name FROM sqlite_master WHERE type='table';").fetchall()
    for table_name in tables:
        table_name = table_name[0] # tables is a list of single item tuples
        table = pd.read_sql_query("SELECT * from {} LIMIT 0".format(table_name), conn)
        print(table_name)
        for col in table.columns:
            print('\t-' + col)
        print()

table_info(c, conn)

Results will be:

table1
    -column1
    -column2

table2
    -column1
    -column2
    -column3 

etc.

I've implemented a sqlite table schema parser in PHP, you may check here: https://github.com/c9s/LazyRecord/blob/master/src/LazyRecord/TableParser/SqliteTableDefinitionParser.php

You can use this definition parser to parse the definitions like the code below:

$parser = new SqliteTableDefinitionParser;
$parser->parseColumnDefinitions('x INTEGER PRIMARY KEY, y DOUBLE, z DATETIME default \'2011-11-10\', name VARCHAR(100)');

Apparently the version of sqlite3 included in Python 2.6 has this ability: http://docs.python.org/dev/library/sqlite3.html

# Convert file existing_db.db to SQL dump file dump.sql
import sqlite3, os

con = sqlite3.connect('existing_db.db')
with open('dump.sql', 'w') as f:
    for line in con.iterdump():
        f.write('%s\n' % line)