Doris Lam

From UrbanWiki

Jump to: navigation, search

Contents

[edit] Contact

dlamoris@ucla.edu

[edit] WSGI

# wsgiref is built-in
from wsgiref import simple_server

# a callable (function in this case) to handle http requests
# environ is a dictionary that contains details about the request
# start_response is called to set http response status and headers
# the function returns the actual response as an iterable (list)
# the echo_app assumes some text has been posted to the server and simply
#     echo them back
def echo_app(environ, start_response):
    content_length = environ["CONTENT_LENGTH"]
    # environ['wsgi.input'] is input stream to the raw post data
    # MUST supply the amount to read indicated by content_length
    content = environ['wsgi.input'].read(int(content_length))
    headers = [('content-type', 'text/plain'),
               ('content-length', str(len(content))]
    start_response('200 OK', headers)
    return [content]

# server will listen on port 8000, using the echo_app callback to handle incoming requests
# each request spawns a new thread, done by the wsgi implementation
server = simple_server.make_server('', 8000, echo_app)
server.serve_forever()
  • Instead of functions, the callback can be any callable object in python
  • sample echo_app using an echo class: 
from wsgiref import simple_server

class Echo(object):

    def __init__(self, prepend):
        self.prepend = prepend

    # implement __call__ to make this class callable
    def __call__(self, environ, start_response):
        content_length = environ["CONTENT_LENGTH"]
        content = environ['wsgi.input'].read(int(content_length))
        body = self.prepend + ' ' + content
        headers = [('content-type', 'text/plain'),
                   ('content-length', str(len(body))]
        start_response('200 OK', headers)
        return [body]    

# instantiate an Echo object and pass as the callable 
# now requests to the server will receive back their posted 
#     message with "Echoing back" prepended
server = simple_server.make_server('', 8000, Echo("Echoing back"))
server.serve_forever()
  • You can create chains of wsgi middleware to process each request
  • example http basic auth middleware: 
import base64
from wsgiref import simple_server

class HTTPBasicAuth(object):
 
    # this class is a wsgi middleware that does http basic authentication
    # using a supplied auth function
    # calls the actual app to handle requests if user has been authenticated
    def __init__(self, app, auth_func):
        self.app = app
        self.check_password = auth_func

    # if not authorized, send back a 401 response
    # else pass the request along
    def __call__(self, environ, start_response):
        if not self.authorized(environ):
            return self.auth_required(environ, start_response)
        return self.app(environ, start_response)

    # checks if user is authorized
    # environ['PATH_INFO'] contains the url after domain name
    #     e.g. www.yahoo.com/check --> environ['PATH_INFO'] = '/check'
    def authorized(self, environ):
        if not self.path_needs_auth(environ['PATH_INFO']):
            return True
        auth_header = environ.get('HTTP_AUTHORIZATION')
        if not auth_header:
            return False
        auth_type, encoded_info = auth_header.split(None, 1)
        assert auth_type.lower() == 'basic'
        decoded_info = base64.b64decode(encoded_info)
        username, password = decoded_info.split(':', 1)
        if not self.check_password(username, password):
            return False
        # middleware can modify environ before passing requests on
        environ['REMOTE_USER'] = username
        del environ['HTTP_AUTHORIZATION']
        return True

    def path_needs_auth(self, path):
        #do url checks here
        if path == "/restricted":
            return True
        return False

    def auth_required(self, environ, start_response):
        headers = [('Content-type', 'text/html'),
                   ('WWW-Authenticate', 'Basic realm="cens"')]
        start_response('401 Authentication Required', headers)
        return ["""
        <html>
         <head><title>Authentication Required</title></head>
         <body>
          <h1>Authentication Required</h1>
          Bad username or password
         </body>
        </html>"""]

# the actual request handler sends back a greeting and the url path
def app(environ, start_response):
    user = environ.get('REMOTE_USER')
    if not user:
        user = "random person"
    content = "Welcome %s! Path is %s." % (user, environ['PATH_INFO'])
    start_response('200 OK', [('content-type', 'text/plain'),
                               ('content-length', str(len(content)))])
    return [content]

# this naive password checker only accepts an user with username and password of 'cens'
# a more realistic checker will perhaps open a database connection and check user info
def check_password(username, password):
    if username != 'cens' or password != 'cens':
        return False
    return True

server = simple_server.make_server('', 8080, HTTPBasicAuth(app, check_password))
server.serve_forever()
  • there can be any number of middleware chains like this, each processing a different part of a request
  • notable/useful environ variables:
    • REQUEST_METHOD - GET or POST
    • PATH_INFO - The remainder of the request URL's "path", designating the virtual "location" of the request's target within the application. This may be an empty string, if the request URL targets the application root and does not have a trailing slash
    • QUERY_STRING - The portion of the request URL that follows the "?", if any. May be empty or absent.
    • CONTENT_LENGTH - The contents of any Content-Length fields in the HTTP request. May be empty or absent
    • CONTENT_TYPE - The contents of any Content-Type fields in the HTTP request. May be empty or absent.
    • HTTP_ variables - Variables corresponding to the client-supplied HTTP request headers (i.e., variables whose names begin with "HTTP_"). The presence or absence of these variables should correspond with the presence or absence of the appropriate HTTP header in the request
  • url reconstruction method - reconstruct full original url requested (contributed by Ian Bicking) 
from urllib import quote
url = environ['wsgi.url_scheme']+'://'

if environ.get('HTTP_HOST'):
    url += environ['HTTP_HOST']
else:
    url += environ['SERVER_NAME']

    if environ['wsgi.url_scheme'] == 'https':
        if environ['SERVER_PORT'] != '443':
           url += ':' + environ['SERVER_PORT']
    else:
        if environ['SERVER_PORT'] != '80':
           url += ':' + environ['SERVER_PORT']

url += quote(environ.get('SCRIPT_NAME',''))
url += quote(environ.get('PATH_INFO',''))
if environ.get('QUERY_STRING'):
    url += '?' + environ['QUERY_STRING']

[edit] JSON

  • Javascript Object Notation - a lightweight data interchange format
  • http://www.json.org/ this site covers it pretty well and provides the full grammar
  • PERFF uses this format extensibly
  • JSON strings can be mapped directly to python dictionaries and lists
  • http://pypi.python.org/pypi/simplejson use simplejson library to convert between python objects and JSON strings 
import simplejson

jsonString = """ {"a":1, "b":2, "c":"Cat", "list":[1, 23, "blah", [34, "z"], {"dict":"nested"}, 70] } """
pyDict = simplejson.loads(jsonString)

# pyDict is now the dictionary that corresponds to jsonString, nested lists and all
# note that keys and string elements will be converted to python unicode strings
pyDict['c']
# this gives u'Cat'
print pyDict['c']
# this prints Cat

print simplejson.dumps(pyDict)
# this prints '{"a":1, "b":2, "c":"Cat", "list":[1, 23, "blah", [34, "z"], {"dict":"nested"}, 70] }'
  • sample client to post json data to a server: 
import httplib, simplejson

json = {"startProb":None,"trace":[{"updated": "2007-11-21 19:29:51", "created":"2007-11-21 19:29:51", "sbid": "1142",
 "phone_time": "2007-10-09 16:30:11", "lon":"-118.246063333", "collection":"6b84ce7c-9867-11dc-b11c-001c42ec2666", 
"gpsSpeed": "0", "freeway": 0, "havSpeed": "0", "lat": "34.1483683333", "id":"20e37962-9868-11dc-b11c-001c42ec2666", 
"mapActivityClass": "still"},{"updated":"2007-11-21 19:29:51", "created":"2007-11-21 19:29:51", "sbid":"1143","phone_time":
 "2007-10-09 16:30:16", "lon": "-118.246063333", "collection":"6b84ce7c-9867-11dc-b11c-001c42ec2666", "gpsSpeed": "0", 
"freeway": 0,"havSpeed": "0", "lat": "34.1483683333", "id": "2125907c-9868-11dc-b11c-001c42ec2666", "mapActivityClass": 
"still"}]}
jsonString = simplejson.dumps(json)

conn = httplib.HTTPConnection("localhost:8000")

headers = {"Content-type": "text/plain", "Content-length":str(len(jsonString)) }
conn.request("POST", "/activity", jsonString, headers)
response = conn.getresponse()
print response.status, response.reason
print response.read()
conn.close()

[edit] Connecting to PostgreSQL database from Python

import psycopg2
   
try:
    conn = psycopg2.connect("dbname='template1' user='dbuser' host='localhost' password='dbpass'");
except:
    print "I am unable to connect to the database"

# use cursor objects to execute queries and get back results
# psycopg2 is thread safe level 2, which means threads can share the module and connection, but not cursors
cur = conn.cursor()

# execute a sql statement
cur.execute("""SELECT a, b from table""")

# cur.rowcount specifies the number of rows that the last executeXXX() produced
if cur.rowcount > 0:
    # fetch one row, or next row as a sequence (e.g. tuple or list)
    row = cur.fetchone()
    a1 = row[0]
    b1 = row[1]
 
    # fetch all remaining rows as list of lists
    rows = cur.fetchall()
    a2 = rows[0][0]
    b2 = rows[0][1]

cur.close()
conn.close()

[edit] PostGIS sample queries

CREATE TABLE sample(
    sbid INTEGER PRIMARY KEY,
    imei_phone VARCHAR(20), 
    lat NUMERIC, 
    lon NUMERIC, 
    phone_timestamp TIMESTAMP,
    exposure FLOAT, 
);

-- SELECT AddGeometryColumn([schema_name], [table_name], [column_name_to_add], [SRID], [geometry_type], [dimension])
SELECT AddGeometryColumn('test', 'sample', 'the_geom', 4326, 'POINT', 2)

CREATE TABLE exposures(
    exposure FLOAT
);
SELECT AddGeometryColumn('test', 'exposures', 'the_geom', 4326, 'POLYGON', 2)

-- AddGeometryColumn ensures that metadata info about the geometry column added (e.g. srid, dimension, type, table) will be
   inserted into the postgis 'geometry_columns' table (if using current schema, schema_name can be omitted)
-- likewise, use DropGeometryColumn or DropGeometryTable when deleting
-- srid of 4326 is usually right for gps data
  • Creating index on geometry columns: 
-- CREATE INDEX [indexname] ON [tablename] USING GIST ( [geometryfield] );
CREATE INDEX polygon_index ON exposures USING GIST (the_geom );

-- Collect statistics after making index:
VACUUM ANALYZE [table_name] [column_name]
  • Sample SQL queries:
    • distance between two points: 
SELECT distance_sphere(GeomFromText('POINT(lon1 lat1)', 4326), GeomFromText('POINT(lon2 lat2)', 4326));

-- lon1, lat1, lon2, lat2 are longitude latitude values, return value is in meters
-- GeomFromText (or PointFromText) takes a Well Known Text representation of a geometry and returns the geometry specified
    • polygon that contains a point: 
SELECT AsText(the_geom), attributes 
FROM polygon_table 
WHERE the_geom && GeomFromText('POINT(lon lat)', 4326) AND Contains(the_geom, GeomFromText('POINT(lon lat)', 4326));

-- AsText function returns Well Known Text representation of geometry (here the geometry is from the geomtry column named  
   'the_geom' in table 'polygon_table'
-- the && operator makes use of geometry index if one was created on the_geom, uses bounding box tests to filter out
    • updating points with exposure info: 
UPDATE TABLE sample 
SET exposure = (SELECT exposure 
                FROM exposures 
                WHERE exposures.the_geom && sample.the_geom AND
                      Contains(exposures.the_geom, sample.the_geom))
WHERE sample.exposure IS NULL;

-- here we're setting the exposure info in the sample table according to which polygon in the exposures table contains the 
   lat lon point
    • closest roads/points from a point (e.g. within 100 meters): 
SELECT s.road, s.distance FROM (SELECT road, distance(transform(GeomFromText('POINT(lon lat)', 4326), 26911),
                                                      transform(the_geom, 26911)) AS distance
     FROM a_streets_table
     WHERE expand(transform(GeomFromText('POINT(lon lat)', 4326), 26911), 100) && 
           transform(the_geom, 26911)) AS s
WHERE s.distance < 100 ORDER BY s.distance;

-- the transform function transforms a geometry from one projection to another so we can compare distance in meters using 
   srid 26911 
-- use function ST_DWithin if using PostGIS 1.3.1+
 SELECT road, distance(transform(GeomFromText('POINT(lon lat)', 4326), 26911),
                        transform(the_geom, 26911)) as distance
 FROM streets_table
 WHERE ST_DWithin(transform(GeomFromText('POINT(lon lat)', 4326), 26911), transform(the_geom, 26911), 100)
  • other potentially useful postgis functions:
    • line_locate_point and line_interpolate_point - can be used to map a point to the closest point to a nearby road
    • centroid - returns the center of a polygon or multipolygon
    • intersects - tests if two geometries spatially intersect
  • see postgis documentation chapter 6 for full list of functions

[edit] Python for Symbian S60


[edit] From Summer

[edit] Using Geoserver web admin tool to set up stuff

  1. create table with geometry in postgis
  2. login to geoserver web admin site (http://localhost:8080/geoserver)
  3. Config->Data
  4. Namespace: new namespace (can be anything, eg. cens, uri = http://www.cens.ucla.edu
  5. Datastore: new PostGIS datastore, fill in required stuff, default port for database is 5432
  6. Featuretype: add new tables in database, make sure tables' geometry columns have been added with SELECT AddGeometryColumn(...) default Style can be 'poi', SRS should already been filled in (4326), either click generate bounding box or use -180 min long, 180 max long, -90 min lat, 90 max lat if table doesn't have anything yet.
  7. submit, apply, save (in that order, apply and save are on left column)
  8. Styles: can put output styles and filters in here, see included styles for examples or see tutorial here
  9. for more stuff see geoserver user's guide
  • code for HTTP POST to automatically add feature types, see here for original (scroll to bottom for php), modify datastore, layername, and post destination in last two lines from geov2 to geoserver
  • mock up code of user submitting table, creating table and geometry in postgis, register featuretype with geoserver, use WFS GetCapabilities to see list of featuretypes in Geoserver - drop down menu for google maps
  • code of receiving data from campaignr

[edit] Getting stuff onto Google Earth

  1. set up featuretypes in Geoserver
  2. in Google Earth, add a network link, put http://server_name:8080/geoserver/wms/kml_reflect?layers=namespace:table_name
  3. see here for bells and whistles

[edit] Google Maps

  1. get google maps api key
  2. get wms-gs.js from here
  3. look at example on that page
  • code for peir pilot 2
  • code for peir pilot 1

[edit] PostGIS Stuff

  • PostGIS cheatsheet here
  • to add a geometry column to a table: SELECT AddGeometryColumn([schema_name], [table_name], [column_name_to_add], [SRID], [geometry_type], [dimension])
    • eg. SELECT AddGeometryColumn('public','table', 'the_geom', 4326, 'POINT', 2);
  • drop geometry column: DropGeometryColumn(<schema_name>, <table_name>, <column_name>)
  • see view geometry_columns for a list of tables with geometry type columns, this should be reflective of what's actually in the database
  • building index on geometry columns: CREATE INDEX [indexname] ON [tablename] USING GIST ( [geometryfield] GIST_GEOMETRY_OPS );
  • get geometry from well known text: eg. GeomFromText('POINT(lon lat)', 4326) where lon and lat are longitude latitude values
  • updating table with geometries from non geometric fields: UPDATE table SET the_geom = GeomFromText('POINT(' || longitude || ' ' || latitude || ')',4326) assuming longitude and latitude are separate columns in the table
  • converting ESRI shapefile into PostGIS (for zip code shapefiles here or boundary files here: use shp2pgsql (type shp2pgsql to see all options)
  • converting tiger shapefiles: use ogr2ogr from here, ogr2ogr -t_srs EPSG:4326 -f "PostgreSQL" PG:dbname=test <tiger_shapefile> or see tutorial here (a bit of overkill)
  • ogr2ogr cheatsheet here
  • find zip code that contains a point: SELECT zcta FROM calizip WHERE Contains(the_geom, GeomFromText('POINT(lon lat)', 4326))
  • to use index in query: do a bounding box check first: SELECT zcta FROM calizip WHERE the_geom && GeomFromText('POINT(lon lat)', 4326) AND Contains(the_geom, GeomFromText('POINT(lon lat)', 4326)) assuming there's a GiST index on the_geom
  • updating table with zip codes: UPDATE table SET zip_code = (SELECT zcta FROM calizip WHERE Contains(calizip.the_geom, table.the_geom)) WHERE table.the_geom IS NOT NULL
  • querying: use AsText(the_geom) to display geometry data as readable text

[edit] twisted stuff

[edit] mostly useful links

Retrieved from "http://wiki.urban.cens.ucla.edu/index.php/Doris_Lam"