Social News in 1000 Steps – Step 13

CSV files are great, but of course even a basic database such as SQLite has many added benefits. It’s efficient to start with the most appropriate setup, however even on a developer workstation we usually want to have a lightweight database with minimal data for practical reasons. I made a script (hopefully one off) to migrate CSV file content to the database:

import os
import core
import csv
db = core.connect()
tables = set(db.tables)
for f in os.listdir('.'):
    if f.endswith('.csv'):
        name = f.replace('.csv', '')
        if name not in tables:
            with open(f, encoding='cp1252') as csv_file:
                reader = csv.DictReader(csv_file)
                for row in reader:
                    db[name].insert({k: v for k, v in row.items()
                                     if k is not None and len(k) > 0})

The views from step 12 needed tweaks, for instance default sorting and HTML rendering:

class CseSearchAdmin(sqla.ModelView):
    def _html_formatter(view, context, model, name):
        return Markup('<a href={0}>{0}</a>'.format(
    def _snippet_formatter(view, context, model, name):
        return Markup(model.html_snippet)
    column_formatters = {'link': _html_formatter,
                         'html_snippet': _snippet_formatter}
    column_default_sort = ('search_date', True)
admin.add_view(CseSearchAdmin(CseSearch, db.session))
Posted in programming | Tagged | Comments Off on Social News in 1000 Steps – Step 13